Shulou(Shulou.com)06/02 Report--

Last time I said that if you create a docker image through dockerfile, then we will create a good image, create a container, and map the code to the container, and finally complete the multi-task app grab system. Source code: github.com/limingios/dockerpython.git (source code/"docker combat chapter" python docker-docker system management-basic concepts (27))

Detailed mission requirements

You need to capture the chatter of three applications, fast hand, today's headlines, and the specific content you need to capture.

1. Capture the author data of the current video

2. Capture the author data of the current video quickly

3. Grab Today's Headline Recommendation Plate News

Code part-python-appium-docker source code inside has

>Python Executes Python Part of Code

#!/ usr/bin/env python# -*- coding: utf-8 -*-# @Time : 2019/3/18 20:31# @Author : Aries# @Site : # @File : handle_appium_docker.py# @Software: PyCharmimport multiprocessingimport timefrom appium import webdriverfrom selenium.webdriver.support.wait import WebDriverWaitdef get_size(driver): x = driver.get_window_size()['width'] y = driver.get_window_size()['height'] return (x, y)def handle_appium(info): cap = { "platformName": "Android", "platformVersion": "4.4.2", "deviceName":info['device'], "udid": info['device'], "appPackage": info['appPackage'], "appActivity": info['appActivity'], "noReset": True, "unicodeKeyboard": True, "resetkeyboard": True } driver = webdriver.Remote("http://192.168.199.140:" + str(info["port"]) + "/wd/hub", cap) l = get_size(driver) x1 = int(l[0] * 0.5) y1 = int(l[1] * 0.15) y2 = int(l[1] * 0.9) #Twittering if info["appPackage"] == "com.ss.andrpid.ugc.aweme": #According to actual I write directly here//android, via if WebDriverWait(driver,60).until(lambda x:x.find_element_by_xpath("//android")): while True: #Initial mouse position, where to start, where to end mouse position, where to end driver.swipe(x1,y1,x1,y2) time.sleep(3) #Fast if info["appPackage"] == "com.smile.gifmaker": #According to the actual I write directly here//android if WebDriverWait(driver, 60).until(lambda x: x.find_element_by_xpath("//android")): while True: #Initial mouse position, where to start, where to end mouse position, where to end driver.swipe(x1, y1, x1, y2) time.sleep(3) #Fast if info["appPackage"] == "com.ss.android.article.news": #According to the actual I write directly here//android if WebDriverWait(driver, 60).until(lambda x: x.find_element_by_xpath("//android")): while True: #Initial mouse position, where to start, where to end mouse position, where to end driver.swipe(x1, y1, x1, y2) time.sleep(3)if __name__ =='__main__': m_list = [] devices_list = [ { "device": "192.168.199.133:5555", "appPackage": "com.ss.android.ugc.aweme", "appActivity": "com.ss.android.ugc.aweme.main.MainActivity", "port": 4723, "key": 'chatter' }, { "device": "192.168.199.133:5555", "appPackage": "com.smile.gifmaker", "appActivity": "com.yxcorp.gifshow.HomeActivity", "port": 4725, "key": 'Quick hand' }, { "device": "192.168.199.133:5555", "appPackage": "com.ss.android.article.news", "appActivity": "com.ss.android.article.news.activity.SplashBadgeActivity", "port": 4727, "key": 'Today's headline' } ]for device in (devices_list): m_list.append(multiprocessing.Process(target=handle_appium,args=(device,)))for m1 in m_list: m1.start()

Code part-decode_data source code inside there are

>python The code that executes the python part is stored in mongodb via mitmdump.

#!/ usr/bin/env python# -*- coding: utf-8 -*-# @Time : 2019/3/18 19:57# @Author : Aries# @Site : # @File : decode_data.py# @Software: PyCharmimport jsonfrom handle_mongo import mongo_infodef response(flow): #Twittering if 'aweme.snssdk.com/aweme/v1/feed' in flow.request.url: douyin_data_dict = json.loads(flow.response.text) for douyin_item in douyin_data_dict['aweme_list']: mongo_info.insert_item(douyin_item) #Fast elif 'api.gifshow.com/rest/n/feed/hot' in flow.request.url: kuaishou_data_dict = json.loads(flow.response.text) for kuaishou_item in kuaishou_data_dict['feeds']: mongo_info.insert_item(kuaishou_item) #Today's headline elif 'is.snssdk.com/api/news/feed' in flow.request.url: jrtt_data_dict = json.loads(flow.response.text) for kuaishou_item in jrtt_data_dict['feeds']: mongo_info.insert_item(kuaishou_item)

deployment

Python crawler access to information is not difficult, the most difficult is the deployment environment.

Virtual machine directly in the form of vagrant

Source code, including vagrant files, want to know how to use can view my intermediate articles explained in great detail

image download

Download mirror of mongodb, mirror of Appium, mirror of zhugeaming/python3-appium

1. Mirror image of mongodb

mkdir bitnamicd bitnamimkdir mongodbdocker run -d -v /path/to/mongodb-persistence:/root/bitnami -p 27017:27017 bitnami/mongodb:latest

2. Mirror image of Appium

docker search appium#is larger than 1 g, because the accelerator has been set up before, according to their own network speed to download. docker pull appium/appium

3.zhugeaming/python3-appium mirror

docker pull zhugeaming/python3-appium

network connection

> Before in the "docker combat chapter" python docker-create appium container and set appium container to connect Android simulator (30)] said a more abnormal method, let Android simulator connect to the dockerapium inside the virtual machine, but this is suitable for vagrant. I suggest we stick with this. It's been tested.

The virtual machines created by Vagrant are all created through virtual boxes.

2. Set up shared folders

This is the Windows environment.

3. In the virtual machine mount, mount the shared folder into the virtual machine

Remember that the name of this shared folder is handle_docker

mkdir dockercd dockersudo yum update && sudo yum -y install kernel-headers kernel-develsudo mount -t vboxsf handle_docker /root/docker/

PS: The basic files are all mounted, it is already very late, next time we continue to run the environment.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.