Shulou(Shulou.com)06/03 Report--

A relatively simple one, python3 multithreading crawls well using the requests library, and uses regular extraction download links Save it to MongoDB databases python# python#-*-coding:utf-8-*-"" @ author:Aiker Zhao@file:doutinghao.py@time: 8:18 pm "import requestsimport reimport pymongofrom multiprocessing import PoolMONGO_URL = 'localhost:27017'MONGO_DB =' doutinghao'MONGO_TABLE = 'doutinghao'client = pymongo.MongoClient (MONGO_URL)" Connect=False) db = client [Mongo _ DB] def get_result (url): response = requests.get (url). Text # print (reponse.text) pattern = re.compile (', re.S) result = re.findall (pattern, response) if result: for i in result: url, name = I yield {"name": name 'url': url} def save_to_mongo (result): if DB [Mongo _ TABLE] .insert (result): print (' stored to MongoDB successful' Result) return True return Falsedef main (result): # result = get_result (url) save_to_mongo (result) if _ _ name__ = ='_ main__': pool = Pool () url = "https://www.xl720.com/thunder/34283.html" item = [item for item in get_result (url)] # print (item) pool.map (main, item) pool.close () pool.join ()

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.