Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to crawl web pages in VB.NET. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

To give a small example is as follows. The following is the function LobDotCn for VB.Net to crawl web pages. Note: whether the IsGb2312 of the target page crawled by url_Link is a Gb2312 character or not.

VB.NET practical tutorial code:

Public Function LobDotCn (ByVal url_Link As String, ByVal IsGb2312 As Boolean) On Error Resume Next Dim XmlHttp As Object XmlHttp = CreateObject ("Microsoft.XMLHttp") XmlHttp.Open ("POST", url_Link, False) XmlHttp.Send () Dim WebContent As Object Dim Str_WebContent As String If IsGb2312 Then WebContent = XmlHttp.ResponseBody Str_WebContent = System.Text.Encoding.Default.GetString (WebContent) Else WebContent = XmlHttp.ResponseText Str_WebContent = WebContent.ToString End If XmlHttp = Nothing LobDotCn = Str_WebContent End Function

Call method:

Variable = LobDotCn ("http://www.lob.cn", True) 'crawl Gb2312 pages

Variable = LobDotCn ("fill in the URL here", False) 'crawl the utf-8 page

A description of the Microsoft.XMLHttp component is attached:

Open (bstrMethod, bstrUrl, varAsync, bstrUser, bstrPassword)

◆ bstrMethod: data transfer method, that is, GET or POST.

◆ bstrUrl: the URL of the service web page.

◆ varAsync: whether to execute synchronously. The default is True, which means synchronous execution, but synchronous execution can only be implemented in DOM. In use, it is generally set to False, that is, asynchronous execution.

◆ bstrUser: user name, which can be omitted

◆ bstrPassword: user password, which can be omitted.

Send (varBody)

◆ varBody: instruction set. It can be data in XML format, a string, a stream, or an array of unsigned integers. You can also omit it and let the instruction be substituted through the URL parameter of the Open method.

◆ setRequestHeader (bstrHeader, bstrvalue)

◆ bstrHeader:HTTP header (header)

◆ bstrvalue: the value of the HTTP header (header)

If the Open method is defined as POST, you can define the form to upload:

Xmlhttp.setRequestHeader ("Content-Type", "application/x-www-form-urlencoded")

XMLHTTP attribute:

◆ onreadystatechange: gets the event handle that returns the result in synchronous execution mode. Can only be called in DOM.

◆ responseBody: the result is returned as an array of unsigned integers.

◆ responseStream: the result is returned as an IStream stream.

◆ responseText: the result is returned as a string.

◆ responseXML: the result is returned as data in XML format.

This is the end of this article on "how to crawl web pages in VB.NET". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.