Shulou(Shulou.com)06/01 Report--

This article is about how Python implements subprocess execution externally. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. Python executes external commands

1. Brief introduction of subprocess module

The subprocess module allows us to start a new process and connect to their input / output / error pipes to get the return value.

This module is used to create and manage child processes. It provides a high-level interface to replace modules and functions such as os.system* (), os.spawn* (), os.popen* (), os,popen2.* (), and commands.*.

Subprocess provides parameters for a class called Popen to start and set child processes. Because of the complexity of this class, subprocess also provides several convenient functions that encapsulate the Popen class.

2. Traversal function of subprocess module

Linux install ipython

Pip3 install ipython

(1) call function

The call function is defined as follows:

Subprocess.ca11 (args, *, stdin=None, stdout=None, stderr=None, she11=False) # runs the command provided by the args parameter, waits for the command execution to finish and returns the return code. When the args parameter is provided as a string and there are multiple command parameters, you need to provide the shell=True parameter

Args: represents the command to be executed. Must be a string, a list of string parameters.

Stdin, stdout, and stderr: standard input, output, and errors for child processes. Its value can be subprocess.PIPE, subprocess.DEVNULL, an existing file descriptor, an open file object, or None. Subprocess.PIPE means to create a new pipe for the child process. Subprocess.DEVNULL means to use os.devnull. None is used by default, which means to do nothing. In addition, stderr can be merged into stdout and output together.

Shell: if this parameter is True, the specified command will be executed through the operating system's shell.

Sample code:

[root@python ~] # ipython # launches ipythonPython 3.8.1 (default, Mar 9 2020, 12:35:12) Type 'copyright',' credits' or 'license' for more informationIPython 7.13.0-An enhanced Interactive Python. Type'?' For help.In [1]: import subprocess # calls the function In [2]: subprocess.call (['ls','-l']) drwxr-xr-x. 2 root root 6 October 31 23:04 Public drwxr-xr-x. 2 root root 6 October 31 23:04 template drwxr-xr-x. 2 root root 6 October 31 23:04 video drwxr-xr-x. 2 root root 4096 October 31 22:40 picture drwxr-xr-x. 2 root root 6 October 31 23:04 document drwxr-xr-x. 2 root root 6 October 31 23:04 download drwxr-xr-x. 2 root root 6 October 31 23:04 Music drwxr-xr-x. 2 root root 6 October 31 15:27 Desktop Out [2]: 0In [3]: subprocess.call ('exit 1 October 31) Out [3]: 1

(2) check_call function

The function of the check_call function is similar to that of the call function, except that it is returned in a different form in the case of an exception.

For the call function, the engineer determines whether the command is executed successfully by capturing the return value of the call command. If it succeeds, it returns 0, otherwise it returns non-0. For the check_call function, it returns 0 if it succeeds, and throws a subrocess.CalledProcessError exception if it fails. As follows:

In [5]: subprocess.check_call (['ls','-l']) drwxr-xr-x. 2 root root 6 October 31 23:04 Public drwxr-xr-x. 2 root root 6 October 31 23:04 template drwxr-xr-x. 2 root root 6 October 31 23:04 video drwxr-xr-x. 2 root root 4096 October 31 22:40 picture drwxr-xr-x. 2 root root 6 October 31 23:04 document drwxr-xr-x. 2 root root 6 October 31 23:04 download drwxr-xr-x. 2 root root 6 October 31 23:04 Music drwxr-xr-x. 2 root root 6 October 31 15:27 Desktop Out [5]: 0In [6]: subprocess.check_call ('exit 1' Shell=True)-CalledProcessError Traceback (most recent call last) in-> 1 subprocess.check_call ('exit 1 writing writing shellfish true) / usr/local/python381/lib/python3.8/subprocess.py in check_call (* popenargs) * * kwargs) 362 if cmd is None: 363 cmd = popenargs [0]-- > 364 raise CalledProcessError (retcode, cmd) 365 return 0 366CalledProcessError: Command 'exit 1' returned non-zero exit status 1.

(3) check_output

The subprocess.check_output function in Python3 can execute a sh command and return the output of the command, as follows:

In [10]: output = subprocess.check_output (['df'] '- h']) In [11]: print (output.decode ()) File system capacity available available mount point / dev/mapper/cl-root 17G 5.2g 12G 31% / devtmpfs 473M 0473m 0% / devtmpfs 489m 92K 489m 1% / dev/ Shmtmpfs 489M 7.1m 482m 2% / runtmpfs 489M 0 489m 0% / sys/fs/cgroup/dev/sda1 1014M 173M 842M 18% / boottmpfs 98M 16K 98m 1% / run/user/42tmpfs 98m 098m 0% / run/user/0In [12]: lines = output.decode (). Split ( '') In [13]: lines Out [13]: ['File system capacity used, available, used mount point' '/ dev/mapper/cl-root 17G 5.2G 12G 31%,' devtmpfs 473M 0473M 0% / dev', 'tmpfs 489M 92K 489m 1% / dev/shm',' tmpfs 489m 7.1m 482m 2% / run', 'tmpfs 489M 0489m 0% / sys/fs/cgroup' '/ dev/sda1 1014M 173M 842M 18% / boot',' tmpfs 98M 16K 98m 1% / run/user/42', 'tmpfs 98M 0 98m 0% / run/user/0' ''] In [14]: for line in Lines [1:-1]:...: if line:: print (line.split () [- 2])...: # intercept mount point data 31% 0% 0% 1% 2% 0% 1% 0%

Executes the command in the child process and returns the output of the execution result as a string. If the child process exit code is not 0, a subprocess.CalledProcessError exception is thrown, and the output field of the exception contains the error output:

In [19]: try:

...: output = subprocess.check_output (['df','-h']). Decode () # correct

...: except subprocess.CalledProcessError as e:

...: output = e.output

...: code = e.returncode

/ / correct without any output

In [23]: try:

: output = subprocess.check_output (['wsd','-h'], stderr=subprocess.STDOUT)

.: .decode () # incorrect

...: except subprocess.CalledProcessError as e:

...: output = e.output

...: code = e.returncode

...:

/ / the previous error code is omitted

FileNotFoundError: [Errno 2] No such file or directory: 'wsd'

3. Popen class (PyCharm) of subprocess module

In fact, all three of our above functions are based on the wrapper of Popen (). The purpose of these wrappers is to make it easy for us to use child processes. When we want to more personalize our requirements, we turn to the Popen class, which generates objects that represent child processes.

The basic process creation and management in the subprocess module is handled by the Popen class

Subprocess.popen is used to replace os.popen.

Popen is the core of subprocess, and it handles the creation and management of child processes.

Constructor:

Class subprocess.Popen (args, bufsize=-1, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0,restore_signals=True, start_new_session=False, pass_fds= (), *, encoding=None, errors=None)

(1) Common parameters:

Args:shell command, which can be a string or sequence type (e.g. list, tuple)

Bufsize: buffer size. Used when creating pipe objects for standard streams, the default is-1.

0: do not use buffer

1: indicates line buffering, available only when universal_newlines=True, that is, text mode

Positive number: indicates buffer size

Negative number: indicates that the system default buffer size is used.

Stdin, stdout, stderr: standard input, output, error handle of the program, respectively

Preexec_fn: valid only on Unix platforms and used to specify an executable object (callable object) that will be called before the child process runs

Shell: if this parameter is True, the specified command will be executed through the operating system's shell.

Cwd: used to set the current directory of the child process.

Env: the environment variable used to specify the child process. If env = None, the environment variables of the child process are inherited from the parent process.

Create a child process and execute a simple command:

> import subprocess > p = subprocess.Popen ('ls-linger, shell=True) > total 164Murray rwkashi-1 root root 133Murray 4 16:25 admin-openrc.sh-rw-r--r-- 1 root root 268 Jul 10 15:55 admin-openrc-v3.sh... > > p.returncode > p.wait () 0 > > p.returncode0

You can also use p = subprocess.Popen (['ls','-cl']) to create child processes here.

(2) attributes of Popen object

P.pid:

The PID of the child process.

P.returncode:

This attribute represents the return status of the child process, and returncode may have multiple situations:

None-the child process has not been completed

= = 0-the child process exits normally

> 0Mel-the child process exited abnormally. Returncode corresponds to the error code.

< 0-- 子进程被信号杀掉了。 p.stdin, p.stdout, p.stderr： 子进程对应的一些初始文件，如果调用Popen()的时候对应的参数是subprocess.PIPE，则这里对应的属性是一个包裹了这个管道的 file 对象。 （3）Popen 对象方法 poll(): 检查进程是否终止，如果终止返回 returncode，否则返回 None。 wait(timeout): 等待子进程终止。 communicate(input,timeout): 和子进程交互，发送和读取数据。 send_signal(singnal): 发送信号到子进程 。 terminate(): 停止子进程,也就是发送SIGTERM信号到子进程。 kill(): 杀死子进程。发送 SIGKILL 信号到子进程。 子进程的PID存储在child.pid import timeimport subprocessdef cmd(command): subp = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,encoding="utf-8") subp.wait(2) if subp.poll() == 0: print(subp.communicate()[1]) else: print("失败")cmd("java -version")cmd("exit 1") 输出结果如下： java version "1.8.0_31" Java(TM) SE Runtime Environment (build 1.8.0_31-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode) 失败 （4）子进程的文本流控制 (沿用child子进程) 子进程的标准输入，标准输出和标准错误也可以通过如下属性表示: child.stdin child.stdout child.stderr 我们可以在Popen()建立子进程的时候改变标准输入、标准输出和标准错误，并可以利用subprocess.PIPE将多个子进程的输入和输出连接在一起，构成管道(pipe): import subprocesschild1 = subprocess.Popen(["ls","-l"], stdout=subprocess.PIPE)child2 = subprocess.Popen(["wc"], stdin=child1.stdout,stdout=subprocess.PIPE)out = child2.communicate()print(out) 执行结果如下： (b' 2 11 60 ', None) subprocess.PIPE实际上为文本流提供一个缓存区。child1的stdout将文本输出到缓存区，随后child2的stdin从该PIPE中将文本读取走。child2的输出文本也被存放在PIPE中，直到communicate()方法从PIPE中读取出PIPE中的文本。 要注意的是，communicate()是Popen对象的一个方法，该方法会阻塞父进程，直到子进程完成。 我们还可以利用communicate()方法来使用PIPE给子进程输入: import subprocesschild = subprocess.Popen(["cat"], stdin=subprocess.PIPE)child.communicate("vamei".encode()) 我们启动子进程之后，cat会等待输入，直到我们用communicate()输入"vamei"。 通过使用subprocess包，我们可以运行外部程序。这极大的拓展了Python的功能。如果你已经了解了操作系统的某些应用，你可以从Python中直接调用该应用(而不是完全依赖Python)，并将应用的结果输出给Python，并让Python继续处理。shell的功能(比如利用文本流连接各个应用)，就可以在Python中实现。 4、使用python自动安i装并启动mongodb PyCharm记得连接linux 简易流程 Python自动化运维 -->

Encapsulation based on shell command

Write automated scripts-- > encapsulate the execution of shell commands with Python syntax

Python executes shell commands-- > python external commands

The python function executes the shell command

Os.system (cmd): executes cmd instructions

Subprocess module

Subprocess.call (['ls','-l']) subprocess.call (' ll', shell=True)

Run successfully: return 0

Failed to run: return non-0

Subprocess. Check_call (['ls','-l']) subprocess. Check_call ('ll', shell=True)

Run successfully: return 0

Failed to run: return CalledProcessError

Subprocess. Check_ output (['cat',' apache.log'], stderr= subprocess.STDOUT)

Run successfully: return the output of the command

Failed to run: custom error output stderr

Popen class of the subprocess module

(1) PyCharm creation file

# coding=utf-8import subprocessimport osimport shutilimport tarfile# function def execute_cmd (cmd) to execute external commands:''execute shell commands' p = subprocess.Popen (cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr= p.communicate () if p.returncode! = 0: return p.returncode Stderr return p.returncode, stdout# decompress def unpackage_mongo (package, package_dir): # get the master file name of the MongoDB package That is, the extracted directory name # mongodb-linux-x86_64-rhe170-4.2.3 unpackage_dir = os.path.splitext (package) [0] if os.path.exists (unpackage_dir): shutil.rmtree (unpackage_dir) if os.path.exists (package_dir): shutil.rmtree (package_dir) # decompressed try: t = tarfile.open (package 'RRRV GZ') t.extractall ('.') Print ('tar is ok.') Except Exception as e: print (e) # rename shutil.move (unpackage_dir, 'mongo') # create mongodatadef create_datadir (data_dir): if os.path.exists (data_dir): shutil.rmtree (data_dir) os.mkdir (data_dir) # splice launch MongoDBdef format_mongod_commamd (package_dir, data_dir, logfile): # mongo/bin/mongod mongod = os.path.join (package_dir 'bin',' mongod') # mongo/bin/mongod-- fork-- logpath mongodata/mongod.log-- dbpath mongodata mongod_format = "" {0}-- fork-- dbpath {1}-- logpath {2} "" return mongod_format.format (mongod, data_dir, logfile) # launch MongoDBdef start_mongod (cmd): returncode Out = execute_cmd (cmd) if returncode! = 0: raise SystemExit ('execute {0} error: {1}' format (cmd, out)) else: print ('execute {0} successfuly.'.format (cmd)) # entry function def main (): package =' mongodb-linux-x86_64-rhel70-4.2.3.tgz' cur_dir = os.path.abspath ('.') Package_dir = os.path.join (cur_dir, 'mongo') data_dir = os.path.join (cur_dir,' mongodata') logfile = os.path.join (data_dir, 'mongod.log') # determine whether if not os.path.exists (package) exists in MongoDB package: raise SystemExit (' {0} not found.'.format (package)) # decompress unpackage_mongo (package) Package_dir) create_datadir (data_dir) # launch mongodb start_mongod (format_mongod_commamd (package_dir, data_dir, logfile)) # configure environment variable os.system ('echo "export PATH=./mongo/bin:$PATH" > ~ / .bash_profile') os.system (' source ~ / .bash_profile') os.system ('. / mongo/bin/mongo') main ()

In this program, we first define several variables in the main function, including the path of the current directory, the path of the MongoDB binaries, the path of the MongoDB data directory, and the log file of MongoDB.

We then determine whether the installation package for MongoDB exists, and if not, end the program by throwing a SystemExit exception.

In the unpackage_mongo function, we get the directory after the MongoDB installation package has been unzipped through the Python program. If the directory already exists, delete it. Then, we use tarfile to extract the MongoDB database, and when the decompression is complete, rename the command to the mongo directory.

In the create_datadir directory, we first determine whether the MongoDB database directory exists, and if so, delete it, and then create the MongoDB database directory.

In the start_mongod function, we execute the startup command of the MongoDB database to start the MongoDB database. To execute the shell command in the Python code, we used the subprocess library. We encapsulate the logic of the subprocess library to execute the she11 command into an execute_cmd function, which can be called directly when the shell command is executed.

(2) upload the files in PyCharm to Linux

If the file is available in Linux directly:

If it is created locally:

(3) Linux executes the script and tests

Remember to enter the directory where PyCharm is connected to linux (currently / opt)

[root@python opt] # python auto_install_mongodb.py # execute the pre-written script tar is ok.execute / opt/mongo/bin/mongod-- fork-- dbpath / opt/mongodata-- logpath / opt/mongodata/mongod.log successfuly. [root@python opt] # netstat-anpt | grep mongo # check whether mongo starts tcp 0 0127.0.1 tar is ok.execute 27017 0.0.0.0 * LISTEN 4616mongod [root@python opt] # ls # check whether to generate the mongo directory 01find_cmd.py bb.bmp mongodb-linux-x86_64-rhel70-4.2.3.tgzaaa.jpg cc.png rhadc.txt mongo subprocess_demoauto_install_mongodb.py mongodata [root@python Opt] # cd mongo [root@python mongo] # cd bin/ [root@python bin] #. / mongo # go to mongoMongoDB shell version v4.2.3connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodbImplicit session: session {"id": UUID ("c302ff50-7e27-40b7-8046-8441af8cb965")} MongoDB server version: 4.2.3 > show databases # check the database admin 0.000GBconfig 0.000GBlocal 0.000GB. Thank you for reading! This is the end of the article on "how Python implements the external implementation of subprocess". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.