Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Python automation tool Fabric". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to use Python automation tool Fabric".

Fabric is mainly used in the automation of application deployment and system management tasks, simple and lightweight, and provides rich SSH extension interfaces. In Fabric 1.x, it mixes local and remote functions, but since Fabric 2.x, it has separated out a separate Invoke library to handle local automation tasks, while Fabric focuses on remote and network-level tasks.

In order to do this, Fabric mainly relies on another core component, Paramiko, which is a remote control module based on SSH protocol. Fabric encapsulates a more friendly interface based on it, which can remotely execute Shell commands, transfer files, batch operation servers, identity authentication, multiple configuration and setting agents, and so on.

I. the distinction between the versions of Fabric

The Python 2 version has been announced by the official that New Year's Day has "retired" this year, and the future will only be the stage of Python 3. In order to accommodate the incompatible migration of the Python version, many projects must also release their own new versions (compatible with or only support Python 3), including Fabric, the protagonist of this article.

There are two large versions of Fabric itself: Fabric 1 and Fabric2, and on top of this library, there are two related libraries that are easy to confuse: Fabric2 and Fabric3 (note that the numbers here are part of the library name).

The distinction between them is as follows:

Fabric 1.x: supports Python 2.5-2.7, but not Python 3

Fabric 2.x: fabfile that supports Python 2.7x and 3.4mm but is not compatible with Fabric 1.x

Fabric2: equivalent to Fabric 2.x, in order for different versions to coexist (install an old version of 1.x and install it as a new version)

Fabric3: a Fabric1.x-based fork (unofficial), compatible with Python 2: 3, compatible with Fabric1.x 's fabfile

To sum up, we recommend the official Fabric 2.x series, but note that some outdated tutorials may be based on earlier versions (or unofficial Fabric3, also based on Fabric 1.x) and need to be identified.

For example, in the Fabric 1.x series, import: from fabric.api import run; will report an error in the new version: "ImportError: No module named api" (PS: you can determine the version of Fabric based on the availability of fabric.api, just as you can determine the version based on print statements or print functions in Python). At the same time, because the new version does not support the old version of fabfile, an error may be reported when using it: "No idea what 'xxx' is!"

Fabric 2 is an incompatible version, and its main improvements over the previous version are as follows:

Support for Python 2.7and 3.4 +

Thread safety, eliminating the concurrent implementation of multi-processes

API reorganized around fabric.connection.Connection

The command line parser has been completely modified to allow regular GNU/POSIX-style flags and options to be used on a per-task basis (fab mytask:weird = custom,arg = format is no longer required)

Can declare pre-task and post-task

…… (official list of more than 10 items [1], not one by one in this article)

The invoke introduced earlier was separated during the development of Fabric 2. For specific reasons, please see this answer [2]. In a nutshell, you should pay attention to version differences when using Fabric.

II. The basic usage of Fabric

1. Installation

First, install: pip intall fabric. After installation, you can view the version information in the command line window:

> fab-V Fabric 2.5.0 Paramiko 2.7.1 Invoke 1.4.0

Execute "fab-V". The above results show that I installed version 2.5.0 of Fabric, as well as the version information of its two core dependent libraries, Paramiko and Invoke.

2. A simple example

Fabric is mainly used for remote tasks, that is, to operate on a remote server. Here is a simple example:

# you can use any file name from fabric import Connection host_ip = '47.xx.xx.xx' # server address user_name =' root' # server username password ='*'# server password cmd = 'date' # shell command to query the time on the server con = Connection (host_ip, user_name, connect_kwargs= {' password': password}) result = con.run (cmd, hide=True) print (result)

With the above code, log in to the remote server with account and password, and then execute the date command to view the time of the server and execute the result:

Command exited with status 0. = = stdout = = Fri Feb 14 15:33:05 CST 2020 (no stderr)

Now in the printed results, in addition to the server time, there is some irrelevant information. This is because the "result" it prints is a "fabric.runners.Result" class, in which we can parse the information:

Print (result.stdout) # Fri Feb 14 15:33:05 CST 2020 print (result.exited) # 0 print (result.ok) # True print (result.failed) # False print (result.command) # date print (result.connection.host) # 47.xx.xx.xx

The above code uses the Connection class and its run () method to run the shell command on the connected server. If administrator privileges are required, replace them with the sudo () method. If you want to execute the shell command locally, replace it with the local () method.

In addition, there are get (), put () and other methods, as described below.

3. Command line usage

The above code can be written in any .py script and then run, or slightly encapsulated and then imported into other scripts for use.

In addition, Fabric is a command-line tool that allows you to perform tasks through the fab command. Let's modify the code in the previous example slightly:

# File name: fabfile.py from fabric import Connection from fabric import task host_ip = '47.xx.xx.xx' # server address user_name =' root' # server username password ='*'# server password cmd = 'date' # shell command, query the time on the server @ task def test (c): "Get date from remote host." Con = Connection (host_ip, user_name, connect_kwargs= {'password': password}) result = con.run (cmd, hide=True) print (result.stdout) # print time only

To explain, the main changes are:

Fabfile.py file name: the script name of the entry code must be this name.

@ task decorator: this decorator needs to be introduced from fabric. It encapsulates the @ task decorator of invoke and is used in the same way as invoke. (note: it also needs to have a context parameter "c", but it is not actually used in the code block, but uses an instance of the Connection class).

Then, in the command line window of the script's sibling directory, you can view and perform the appropriate tasks:

Fab-l Available tasks: test Get date from remote host. > fab test Fri Feb 14 16:10:24 CST 2020

Fab is an extended implementation of Invoke, inheriting many of the original functions, so execute "fab-help". Compared with the "inv-help" introduced earlier, you will find that many of their parameters and interpretations are exactly the same.

Fab has added several command line options (marked blue) for remote service scenarios, where:

-- prompt-for-login-password: causes the program to enter the SSH login password on the command line (the above example specifies the connect_kwargs.password parameter in the code. If you use this option, you can require the password to be entered manually during execution)

-- prompt-for-passphrase: causes the program to enter the path to the encrypted file with SSH private key on the command line

-H or-- hosts: specify the host name to connect to

-I or-- identity: specify the private key file used for the SSH connection

-S or-- ssh-config: specifies the SSH configuration file to load at run time

For more information about the command line interface of Fabric, see the document [3].

4. Interactive operation

If there is an interactive prompt on the remote server, asking for information such as a password or "yes", this requires the Fabric to be able to listen and respond.

Here is a simple example. The Responder of invoke is introduced. The initialization content is a regular string and response information, and finally the value is assigned to the watchers parameter:

From invoke import Responder from fabric import Connection c = Connection ('host') sudopass = Responder (pattern=r'\ [sudo\] password:', response='mypassword\ n') c.run (' sudo whoami', pty=True, watchers= [sudopass])

5. Transfer files

File transfer between local and server is a common use. Fabric is well encapsulated in this respect, and the following two methods are available in the Connection class:

Get (* args, * * kwargs): pull a remote file to a local file system or class file (file-like) object

Put (* args, * * kwargs): push local file or class file objects to the remote file system

In the case where a connection is established, the example:

# (short) con.get ('/ opt/123.txt', '123.txt') con.put (' test.txt','/ opt/test.txt')

The first parameter refers to the source file to be transferred, and the second parameter is the destination to be transferred, which can be specified as a file name or folder (if empty or None, the default path is used):

# (abbreviated) con.get ('/ opt/123.txt',') # if empty, use the default path con.put ('test.txt',' / opt/') # to specify the path / opt/

The default storage path for the get () method is os.getcwd, while the default storage path for the put () method is the home directory.

6. Batch operation of server

For batch operations on a server cluster, the easiest way to do this is to use a for loop, and then set up the connection and perform operations one by one, like this:

For host in ('web1',' web2', 'mac1'): result = Connection (host) .run (' uname-s')

But sometimes there are problems with such a scheme:

If there are multiple sets of different server clusters and need to perform different operations, then you need to write a lot of for loops

If you want to aggregate the results of each set of operations (for example, dictionary form, key- host, value- results), you have to add additional actions to the for loop

The for loop is executed sequentially and synchronously, which is too inefficient and lacks exception handling mechanism (if an exception occurs in the middle, it will cause subsequent operations to jump out)

For these problems, Fabric put forward the concept of Group, which can define a group of hosts as a Group, and its API method is the same as Connection, that is, a Group can be simplified as a Connection.

Then, the developer simply manipulates the Group and ends up with a result set, reducing his work on exception handling and execution order.

Fabric provides a fabric.group.Group base class and derives two subclasses from it, the difference is:

SerialGroup (* hosts, * * kwargs): perform operations in a serial manner

ThreadingGroup (* hosts, * * kwargs): perform operations concurrently

The type of Group determines how the host cluster operates, and we only need to make a choice. Then, the result of their execution is a fabric.group.GroupResult class, which is a subclass of dict, which stores the correspondence between each host connection and its execution result.

> from fabric import SerialGroup > results = SerialGroup ('web1',' web2', 'mac1'). Run (' uname-s') > print (results)

In addition, GroupResult also provides two attributes, failed and succeeded, to fetch a subset of failures / successes. As a result, the secondary operation can be carried out in batches conveniently.

III. Advanced usage of Fabric

1. Identity authentication

Fabric uses the SSH protocol to establish remote sessions, which is a relatively secure encrypted transport protocol based on the application layer.

Basically, it has two levels of security authentication:

Password-based authentication: use accounts and passwords to log in to remote hosts with low security and vulnerability to man-in-the-middle attacks

Key-based authentication: the use of key pairs (public key to the server, private key to the client) will not be attacked by "man in the middle", but login takes a long time

In the previous example, we used the first way, that is, to log in with a password by specifying the connect_kwargs.password parameter.

Of course, Fabric also supports the second method. There are three ways to specify the path of the private key file. The priority is as follows:

First look up the connect_kwargs.key_filename parameter, and then use it as the private key

Second, look for the command line usage-- identify option

Finally, the default value of IdentityFile in the ssh_config file of the operating system is used.

If the private key file itself is encrypted, you need to use the connect_kwargs.passphrase parameter.

2. Configuration file

Fabric supports separating some parameter items from the business code, that is, managing them through configuration files, such as the password and private key files mentioned earlier, which can be written in the configuration file to avoid coupling with the code.

Fabric basically follows the configuration file system of Invoke (nine layers are listed in the official document), while adding some configuration items related to SSH. The supported file formats are .yaml, .yml, .json and .py (according to this priority). It is recommended to use yaml format (the suffix can be simplified to yml).

Among them, the more common configuration files are:

System-level configuration file: / etc/fabric.yml

User-level configuration file: ~ / .fabric.yml (Windows under C:\ Users\ xxx)

Project-level configuration file: / myproject/fabric.yml

The priority of the above files is decreasing, because my native machine is Windows, for convenience, I create a ".fabric.yml" file in the user directory, which is as follows:

# filename:.fabric.yml user: root connect_kwargs: password: xxxx # if the key is used, it is as follows # key_filename: #-your_key_file

We've extracted the username and password, so we can delete them from fabfile:

# File name: fabfile.py from fabric import Connection from fabric import task host_ip = '47.xx.xx.xx' # server address cmd =' date' # shell command, query server time @ task def test (c): "Get date from remote host." Con = Connection (host_ip) result = con.run (cmd, hide=True) print (result.stdout)

Then, execute on the command line:

> fab test Tue Feb 18 10:33:38 CST 2020

Many parameters can also be set in the configuration file

3. Network gateway

If the remote service is network isolated and cannot be accessed directly (on different Lans), a gateway / agent / tunnel is required, and this middle-tier machine is often referred to as a springboard or fortress machine.

There are two gateway solutions in Fabric that correspond to two options for OpenSSH clients:

ProxyJump: simple, low overhead, nesting

ProxyCommand: high cost, non-nesting, more flexible

When you create a Connection object for Fabric, you can apply these two schemes by specifying the gateway parameter:

The ProxyJump method is to nest a Connection in a Connection as the gateway of the former, the latter uses the direct-tcpip of the SSH protocol to open a connection with the actual remote host for the former, and the latter can continue to nest and use its own gateway.

From fabric import Connection c = Connection ('internalhost', gateway=Connection (' gatewayhost'))

The ProxyCommand method is that the client uses the ssh command locally (similar to "ssh-W% HGV% p gatewayhost") to create a child process that communicates with the server and can read standard input and output.

The implementation details of this section are in paramiko.channel.Channel and paramiko.proxy.ProxyCommand, respectively, and can also be defined in the configuration files supported by Fabric, in addition to being specified in the parameters.

Thank you for your reading, the above is the content of "how to use Python automation tool Fabric". After the study of this article, I believe you have a deeper understanding of how to use Python automation tool Fabric, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.