How to implement concurrent crawler in Go language
This article will explain in detail how to implement concurrent crawlers in Go language. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.
1. Single thread crawler
Define a user
Var Client http.Client
Principal function
Func main () {url: = "http://localhost:3000/api/v1/products" start: = time.Now () for I: = 0; I
< 10; i++ { Spider(url, i) } elapsed := time.Since(start) fmt.Printf("Time %s", elapsed)} 爬取函数 func Spider(url string, i int) { reqSpider, err := http.NewRequest("GET", url, nil) if err != nil { log.Fatal(err) } reqSpider.Header.Set("content-length", "0") reqSpider.Header.Set("accept", "*/*") reqSpider.Header.Set("x-requested-with", "XMLHttpRequest") respSpider, err := Client.Do(reqSpider) if err != nil { log.Fatal(err) } bodyText, _ := ioutil.ReadAll(respSpider.Body) var result Result _ = json.Unmarshal(bodyText, &result) fmt.Println(i,result.Data)} 运行时间为：651.8207ms
two。 Multithreaded crawler 2.1 channel main function
We construct an unbuffered channel to block the main process and wait for the execution of the child process.
Func main () {url: = "http://localhost:3000/api/v1/products" ch: = make (chan bool) start: = time.Now () for I: = 0; I < 10; iTunes + {go Spider (url, ch, I)} for I: = 0; I < 10; iTunes + {
