Shulou(Shulou.com)06/02 Report--

Curl:

Curl is a powerful URL transfer tool, and the most common function of this command is to send http requests and download files from the command line, such as simulating refer (jumping from that page), cookie, agent (what browser to use), etc., as well as simulating form data.

Curl command options:

-I: display header information

-I: displays only the header information, not the body

-X: specify the request method, such as GET\ POST, etc.

-- from: simulation form. With this parameter, you can upload files, simulate clicking buttons, etc.

-A: specify a user agent, such as Mozilla/4.0

-b: set cookie

-c: specify the cookie file

-e: specify referer. Some URLs must jump from a page.

-- header: sets request header information

-- user: some pages require http authentication and pass name:passwd authentication

Curl-X POST-d "*"

Use the redirect ">" symbol to redirect output to a local file

Curl http://www.jd.com > index.html

You can also save the content to a file through the-o _ curl option that comes with it.

-o (lowercase o): the result is saved to the file name provided on the command line

Curl-o index.html http://www.jd.com

-O (uppercase O): the file name in URL will be used to save the file locally

Curl https://cache.yisu.com/upload/information/20200309/28/2481-O

Resume: when the process ends before the file is downloaded, continue to download the file through the Japan-C option, and the downloaded file will not be downloaded repeatedly.

-- limit-rate: speed limit for curl

Curl-- limit-rate 1024B http://www.jd.com

To judge the last modification time of the file, if the file has been modified within the specified date, it will be downloaded, otherwise it will not be downloaded, this function is achieved through-z: for example, if the a.html file has been updated after 2019-12-28.

Curl-z 21-Dec-18 http://www.a.com/a.html

When a link is opened through a browser, it automatically jumps to http://www.jd.com. At this point, we want curl to follow the link like a browser to get the final web content. You can add the-L option to the command to follow the cohesion redirection.

Curl-L http://www.jd.com

Wget command

Wget is a file download tool that is used on the command line and supports breakpoint downloads as well as ftp and http downloads. Support for proxy servers and easy to set up.

Wget-c: breakpoint continuation use

Wget-b: download in the background

Wget-- spider URL: test whether the download link is valid

Wget ftp-url: use wget to download the ftp link. Anonymous ftp download is similar to http download.

Wget-- ftp-user=name-- ftp-password=passwd url: ftp downloads authenticated with wget username and password

Wget-Q: limit the total number of downloaded files

The download file exceeds 10MB and exits.

Wget-Q10m-I iso_limit.txt

Download a single file:

Wget http://www.jd.com/a.iso

When the downloaded file name is too long for operators to operate, you can customize the file name and use "- O" to specify a file name:

Wget-O Centos-7.iso http://www.jd.com/a.iso

Save the download information to the log: wget-o

Wget-o a.log URL

Use-- limit-rate speed limit to download:

Wget-O Centos-7.iso-- limit-rate=600k http://www.jd.com/a.iso

Use wget to mirror the website:

Wget-- mirror-p-- convert-links-p. / local url

-- mirror: set this parameter to build a local image

-p: download all the elements suitable for display in the html file

-- convert-links: after the download is complete, convert the document links to local ones.

-P. / local: save all files and directories to the specified folder

When downloading the site, you are not going to download gif animated pictures

Wget-reject=gif url

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.