How java crawler crawls Cat's Eye Movie TOP list data
This article is about how java crawlers crawl data from the TOP list of cat's eye movies. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.
How the crawler crawled the TOP list of cat's eye movies. The main contents are ranking, picture, movie name, leading role, release time and rating information. Before crawling, let's open the cat's eye movie TOP100 page, study and analyze the page, find the location of the information we need, and then grab it.
The code is as follows:
Import json
Import requests
From requests.exceptions import RequestException
Import re
Import time
Def get_one_page (url):
Try: headers= {'User-Agent':' agent Information'} response = requests.get (url, headers=headers) if response.status_code = = 200: return response.text return Noneexcept RequestException: return None
Def parse_one_page (html):
Pattern = re.compile ('. *? board-index.*? > (\ d +). *? data-src= "(. *?)". *? name "> (. *?). *? star" > (. *?)
. *? releasetime "> (. *?)
'+'. *? integer "> (. *?). Fraction" > (. *?). *?', re.S) items = re.findall (pattern, html) for item in items: yield {'index': item [0],' image': item [1], 'title': item [2],' actor': item [3] .strip () [3:] 'time': item [4] .strip () [5:], 'score': item [5] + item [6]}
Def write_to_file (content):
With open ('result.txt',' asides, encoding='utf-8') as f: f.write (json.dumps (content, ensure_ascii=False) +'\ n')
Def main (offset):
Url = 'http://maoyan.com/board/4?offset=' + str (offset) html = get_one_page (url) for item in parse_one_page (html): print (item) write_to_file (item)
If name = 'main':
For i in range (10): main (offset=i * 10) time.sleep (1)
Through the above code, we can get the data information of the TOP list of cat's eye movies.
Thank you for reading! This is the end of this article on "how java crawlers crawl TOP list data of cat's eye movies". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!
