What are the methods of implementing multithreaded downloaders with go
This article mainly explains "what are the methods of using go to achieve multithreaded downloaders", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what are the methods of using go to achieve a multithreaded downloader?"
Catalogue
1. Principle of multithreaded download
two。 Construct a downloader
2.1 provide initialization methods for downloaders
3. Implementation of download integrated scheduling logic
3.1 download file segments
3.2 subthreads download functions
4. Save download file function
5. Complete code
In this article, we use Go to implement a simple multithreaded downloader.
1. Principle of multithreaded download
By judging the Accept-Ranges field in the information returned by the download file link, if it is bytes, the breakpoint continuation is supported.
Then set the Range field to bytes= [start]-[end] in the request header to request that the segmented parts of the file be downloaded, and then merge all the segments into one complete file.
two。 Construct a downloader type HttpDownloader struct {url string filename string contentLength int acceptRanges bool / / whether to support breakpoint continuation of numThreads int / / number of simultaneous download threads} 2.2.1.It provides initialization methods for downloaders func New (url string, numThreads int) * HttpDownloader {var urlSplits [] string = strings.Split (url, "/") var filename string = urlSplits [len (urlSplits)-1] res Err: = http.Head (url) check (err) httpDownload: = new (HttpDownloader) httpDownload.url = url httpDownload.contentLength = int (res.ContentLength) httpDownload.numThreads = numThreads httpDownload.filename = filename if len (res.Header ["Accept-Ranges"])! = 0 & & res.Header ["Accept-Ranges"] [0] = "bytes" {httpDownload.acceptRanges = true} else {httpDownload.acceptRanges = false} return httpDownload} 3. Implementation of download integrated scheduling logic
If multithreaded downloads are not supported, use single-threaded downloads.
Func (h * HttpDownloader) Download () {f, err: = os.Create (h.filename) check (err) defer f.Close () if h.acceptRanges = = false {fmt.Println ("this file does not support multithreaded download In single-threaded download: ") resp, err: = http.Get (h.url) check (err) save2file (h.filename, 0, resp)} else {var wg sync.WaitGroup for _, ranges: = range h.Split () {fmt.Printf (" multithreaded download:% dmurt% d\ n ", ranges [0] Ranges [1]) wg.Add (1) go func (start, end int) {defer wg.Done () h.download (start, end)} (ranges [0] Ranges [1])} wg.Wait ()} 3.1 download file segments func (h * HttpDownloader) Split () [] [] int {ranges: = [] [] int {} blockSize: = h.contentLength / h.numThreads for iVRO I
