Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to implement concurrent crawlers in Go language. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

1. Single thread crawler

Define a user

Var Client http.Client

Principal function

Func main () {url: = "http://localhost:3000/api/v1/products" start: = time.Now () for I: = 0; I

< 10; i++ { Spider(url, i) } elapsed := time.Since(start) fmt.Printf("Time %s", elapsed)} 爬取函数 func Spider(url string, i int) { reqSpider, err := http.NewRequest("GET", url, nil) if err != nil { log.Fatal(err) } reqSpider.Header.Set("content-length", "0") reqSpider.Header.Set("accept", "*/*") reqSpider.Header.Set("x-requested-with", "XMLHttpRequest") respSpider, err := Client.Do(reqSpider) if err != nil { log.Fatal(err) } bodyText, _ := ioutil.ReadAll(respSpider.Body) var result Result _ = json.Unmarshal(bodyText, &result) fmt.Println(i,result.Data)} 运行时间为：651.8207ms

two。 Multithreaded crawler 2.1 channel main function

We construct an unbuffered channel to block the main process and wait for the execution of the child process.

Func main () {url: = "http://localhost:3000/api/v1/products" ch: = make (chan bool) start: = time.Now () for I: = 0; I < 10; iTunes + {go Spider (url, ch, I)} for I: = 0; I < 10; iTunes + {

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.