Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to use Python to collect Beijing second-hand housing data". In the daily operation, I believe many people have doubts about how to use Python to collect Beijing second-hand housing data. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to use Python to collect Beijing second-hand housing data". Next, please follow the editor to study!

1. Tool description

The Python libraries used in this article are

Plus Baidu map API

2. Data acquisition

All the data comes from Lianjia's second-hand housing trading platform, which arranges 30 pieces of second-hand housing data on each page. The first 100 pages of data are collected in this paper. The title, unit price, price, address, year, room style and other fields are extracted from each second-hand housing transaction data as the data source for visual analysis.

The website does not have a strong anti-crawling mechanism. You can use the combination of requests and Cookies+ PyQuery when crawling. It is best to add a time.sleep () command when crawling, and sleep every few seconds. Part of the code is as follows:

Finally, a total of 3000 pieces of data were collected.

3. Address latitude and longitude coordinate conversion

The data obtained is that the address is in the form of a string (for example, Van Gogh Water County * Jiuxianqiao). When marking the location of the map, you need latitude and longitude data, and you need to convert all addresses into latitude and longitude coordinates. Here, with the help of Baidu Map API, this operation is completed.

1. Baidu Map AK application

To use API, you need to apply for an AK verification code on Baidu Map Open platform, log in to your own Baidu account, and create an application on the console.

Console panel-> my application-> create application

Other parameters are default. Just apply the custom name. Enter 0.0.0.0Univer 0 in the IP whitelist, and click submit.

After the above steps are completed, there will be one more application in the console panel, which is the one we just established.

2. Address latitude and longitude coordinate conversion

After obtaining your AK parameters, pass AK and address (Chinese address) as parameters into the following function to obtain the corresponding latitude and longitude coordinates.

Def get_location (address,AK): # convert to latitude and longitude try according to address: url = 'http://api.map.baidu.com/geocoding/v3/?address={}&output=json&ak={}&callback=showLocation'.format(address,AK) res_test = requests.get (url). Text res = re.search ('\ ((. *?)\)' Res_test) .group (0). Strip ('('). Strip (') lng = json.loads (res) ['result'] [' location'] ['lng'] lat = json.loads (res) [' result'] ['location'] [' lat'] print (lng, lat) return lng, lat except Exception as e: print ('error -\ n') return None

The longitude and latitude conversion function of Baidu map API is not only limited to provinces and cities, but also can locate the house number. as long as the more detailed the address you provide, the more accurate the longitude and latitude information is obtained.

4. Visual analysis

This part enters the visual analysis content, after all, the city is Beijing, so the first thing is to pay attention to the housing price problem. In the captured data, the highest is nearly 190000 + / per square, and the lowest is only 15000 / per square.

I am not calm when I see 15000 per square meter, and there must be something wrong with this figure (Beijing housing prices cannot be so low in common sense). In order to verify the idea, I first made a real estate location mark to look at it first.

Above is the location of the highest Top10 of the house price, which seems to be in a good geographical position, all within the third Ring Road and the fourth Ring Road, and distributed in the neighborhood of the city center. If you can't get useful information if you only look at the map of this map, you can compare the lowest Top10 of the house price.

Above, the prices of second-hand houses in Beijing are distributed in the bottom 10 areas, with prices ranging from 15000 to 30,000. before seeing this distribution map, I only thought that the above price was mismarked by the seller. After reading this picture, I found that the seller was selling dog meat with sheep's head and the name of Beijing. selling houses outside Beijing (some houses are already located in Hebei), this may be one of the routines of selling houses.

When things are abnormal, there must be demons, whether buying a house or other goods, the seller will not give us a discount for no reason, when the price is much lower than the market price before trading need to be cautious, cautious, and then cautious!

The house price is divided into five ranges, which are 0-30, 120000, 30-80, 8-120000, 12-150000, 150000 + 5 respectively.

Among them, the largest proportion of house prices is between 30,000 and 80,000, accounting for 76%, followed by 8-120000 and 12-150000, accounting for 16.38% and 3.46%, respectively, while housing prices at 150000 + account for the least, only about 1.12%.

Next, let's take a look at the distribution of second-hand houses at different prices in Beijing.

150000 +

8-120000

30-80,000

Judging from the location distribution, the house price in Beijing is 80, 000 as the watershed, and the property distribution within 80, 000 + and 80, 000 changes obviously. The higher the price, the more close to the city center. From the map information, if you want to buy a house with a good location in Beijing, the lowest house price is expected to be 80, 000 +, what is the concept of 80, 000, that is to say, if you want to buy a 50-square-meter house in Beijing, the lowest is 4 million!

About 150000 + real estate, roughly distributed in two areas, one is the university district (surrounded by the National people's Congress, Beihang, Tsinghua and other universities), and the other is located in the lower right corner, that is, Chaoyang District, Beijing.

The second-hand houses sold in Beijing are not necessarily decorated, but they may also be rough (no decoration has been carried out just after construction and delivery). Referring to the figure below, the blank accounts for about 1.39% of the total 41 second-hand houses in Beijing. the householder buys these houses for asset appreciation rather than rigid demand.

With the help of the Treemap map, let's take a look at the time distribution of second-hand housing construction in Beijing. Most of them are concentrated in the nine years from 2002 to 2010. Beijing real estate also developed rapidly during this period, and slowed down significantly after 2015, due to the fact that the construction site has reached saturation and pressure from relevant national policies.

At this point, the study on "how to use Python to collect second-hand housing data in Beijing" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.