How to crawl the data of second-hand houses by python
This article is about how python crawls data from second-hand houses. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.
First, find the location of the data:
Open Lianjia's official website, enter the second-hand housing page, select a city, you can see the total number of houses in the city and housing list data.
Second, determine the location of data storage:
The data of some websites is stored in html, while some are api, and some are even encrypted in js. Fortunately, Lianjia's housing data is stored in html:
3. Obtain html data:
Request a page through requests to get the html data of each page
# crawled url, default crawled real estate information of Nanjing Lianjia url = 'https://nj.***.com/ershoufang/pg{}/'.format(page)# request urlresp = requests.get (url, headers=headers, timeout=10)
The website in the code is not a real URL, can not be run directly!
Fourth, parse html to extract useful data:
Parse html through BeautifulSoup and extract corresponding useful data
Soup = BeautifulSoup (resp.content, 'lxml') # filter all li tags sellListContent = soup.select (' .sellListContent li.LOGCLICKDATA') # Loop through for sell in sellListContent: # title title = sell.select ('div.title a') [0] .string # grab all div information first Then extract houseInfo = list (sell.select ('div.houseInfo') [0] .stripped _ strings) # Real Estate name loupan = houseInfo [0] # split (' |') # House Type house_type = info [1]. Strip () # area size area = info [2] .strip ( ) # rooms facing toward = info [3] .strip () # Decoration type renovation = info [4] .strip () # House address positionInfo = '.join (list (sell.select (' div.positionInfo') [0] .stripped _ strings)) # Total house price totalPrice =''.join (list (sell.select (' div.totalPrice') [0] .stripped _ strings)) Housing unit price unitPrice = list (sell.select ('div.unitPrice') [0] .stripped _ strings) [0] Thank you for reading! This is the end of the article on "how python crawls the data of second-hand housing". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!
