How to solve the problem that Go language does crawler status code return 418?
This article mainly introduces how to solve the problem of how to do crawler status code return 418 in Go language. It is very detailed and has a certain reference value. Interested friends must finish reading it!
Background
When uses the go language as a crawler, it uses http.Get (url) to get the web page content. The status code returns 404, and the Body body is empty.
Cause analysis
http.Get (url) is a http request that does not need to set the header attribute, which is relatively simple and quick, but the status code returns 418, indicating that we need to set its header property, so we can use http.NewRequest to set its header attribute.
The code section func main7 () {client: = & http.Client {} url: = "https://movie.douban.com/top250?start=0&filter=" reqest, err: = http.NewRequest (" GET ", url, nil) / / sets the header property reqest.Header.Add (" User-Agent "," Mozilla/5.0 (Windows NT 10.0; Win64) " X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36 OPR/66.0.3515.115 ") if erratic response {fmt.Println (err) return} response, _: = client.Do (reqest) defer response.Body.Close () buf: = make ([] byte 4096) var result string for {n Err:=response.Body.Read (buf) if n = 0 {fmt.Println ("read the web page completed") break} if erratic readers nil & & errors read io.EOF {fmt.Println ("resp body err") Err) return} result + = string (buf [: n]) / / print the read web page fmt.Println (result)}} above are all the contents of this article entitled "how to solve the problem of how to use Go language to return 418 crawler status codes" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!
