Shulou(Shulou.com)06/03 Report--

This article mainly introduces the skills of Python unit testing, the article is very detailed, has a certain reference value, interested friends must read it!

1. Unit test status of requests project

The unit test code for requests is all in the tests directory and is configured using pytest.ini. In addition to pytest, the test requires installation:

The library name describes httpbin, a http service implemented using flask, which can define http responses on the client side. It is mainly used to test plug-ins for http protocol pytest-httpbinpytest, encapsulate plug-ins for httpbin to implement pytest-mockpytest, provide plug-ins for mockpytest-covpytest, and provide coverage.

The above dependent master version is defined in the requirement-dev file; version 2.24.0 is defined in pipenv.

The test case uses the make command, and the subcommands are defined in Makefile. Running all unit tests using make ci results are as follows:

$make ci pytest tests-- junitxml=report.xml = test session starts = platform linux-- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 rootdir: / home/work6/project/requests, inifile: pytest.ini plugins: mock-2.0.0, httpbin-1.0.0 Cov-2.9.0 collected 552 items tests/test_help. Py. [0%] tests/test_hooks.py. 1% tests/test_lowlevel.py. [3] tests/test_packages.py... [4] tests/test _ requests.py. .... [39%] 127.0.0.1-- [10/Aug/2021 08:41:53] "GET / stream/4 HTTP/1.1" 200 756.127.0.0.1-- [10/Aug/2021 08:41:53] "GET / stream/4 HTTP/1.1" 500 59-- -Exception happened during processing of request from ('127.0.0.1' 46048) Traceback (most recent call last): File "/ usr/lib64/python3.6/wsgiref/handlers.py", line In run self.finish_response () x. [56%] tests/test_structures.py. [59%] tests/test_testserver.py .s. [61] tests/test_utils.py.. s. .ssss [98%] ssssss. [100%]-generated xml file: / home/work6/project/requests/report.xml- -= 539 passed 12 skipped, 1 xfailed in 64.16 seconds =

You can see that requests passed a total of 539 test cases in 1 minute, and the results were good. Use make coverage to view unit test coverage:

$make coverage-coverage: platform linux Python 3.6.8-final-0-Name Stmts Miss Cover-requests/__init__.py 71 0% requests/__version__.py 10 100% requests/_internal_utils.py 16 5 69% requests/adapters.py 222 67 70% requests/api.py 20 13 35% requests/auth.py 17454 69% requests/certs.py 4 40% requests/compat.py 47 47% requests/cookies.py 238 115 52% requests/exceptions.py 35 29 17% requests/help.py 63 19 70% requests/hooks.py 15 4 73% requests/models.py 455 119 74% requests/packages.py 160% requests/sessions.py 283 67 76% requests/status_codes.py 15 150% requests/structures.py 40 19 52% requests/utils.py 465 170 63%-TOTAL 2189 844 61% Coverage XML written to file coverage.xml

The results show that the overall coverage of the requests project is 61%, and the coverage of each module is also clearly visible.

Unit test coverage is determined by the number of lines of code, Stmts displays the number of valid lines of the module, and Miss shows the lines that have not been executed. If you generate a report for html, you can also navigate to specific rows that are not covered; pycharm's coverage has a similar feature.

The files and test classes under tests are as follows:

File describes compatpython2 and python3 compatible conftestpytest configuration test_help,test_packages,test_hooks,test_structures simple test class utils.py tool function test_utils test tool function test_requests test requeststestserver\ server simulation service test_testserver simulation service test test_lowlevel use simulation service test simulation network test 2, how to test a simple tool class

2.1Analysis of test_help implementation

Start with the simplest test_help, where the names of the test class and the object being tested correspond. First take a look at the module under test, help.py. This module mainly consists of two functions, info and _ implementation:

Import idna def _ implementation ():... Def info ():... System_ssl = ssl.OPENSSL_VERSION_NUMBER system_ssl_info = {'version':'% x'% system_ssl if system_ssl is not None else''} idna_info = {'version': getattr (idna,' _ version__',''),}. Return {'platform': platform_info,' implementation': implementation_info, 'system_ssl': system_ssl_info,' using_pyopenssl': pyopenssl is not None, 'pyOpenSSL': pyopenssl_info,' urllib3': urllib3_info, 'chardet': chardet_info,' cryptography': cryptography_info, 'idna': idna_info 'requests': {'version': requests_version,},}

Info provides information about the system environment, and _ implementation is its internal implementation, beginning with an underscore * _ *. Then take a look at the test class test_help:

From requests.help import info def test_system_ssl (): "Verify we're actually setting system_ssl when it should be available." Assert info () ['system_ssl'] [' version']! =''class VersionedPackage (object): def _ _ init__ (self, version): self.__version__ = version def test_idna_without_version_attribute (mocker): "Older versions of IDNA don't provide a _ version__ attribute, verify that if we have such a package, we don't blow up." Mocker.patch ('requests.help.idna', new=None) assert info () [' idna'] = {'version':'} def test_idna_with_version_attribute (mocker): "Verify we're actually setting idna version when it should be available." Mocker.patch ('requests.help.idna', new=VersionedPackage (' 2.6')) assert info () ['idna'] = {' version': '2.6'}

First of all, you can see from the import information in the header that only the info function is tested, which is easy to understand. The info test passes and naturally overrides the internal function _ implementation. Here's the first tip for unit testing: test only the interface of public

Both test_idna_without_version_attribute and test_idna_with_version_attribute have a mocker parameter, which is provided by pytest-mock and automatically injects a mock implementation. Use this mock to simulate the idna module

# Simulation empty implementation mocker.patch ('requests.help.idna', new=None) # Simulation version 2.6 mocker.patch (' requests.help.idna', new=VersionedPackage ('2.6'))

You may be surprised that here patch simulates requests.help.idna, while what we import into help is the inda module. This is because the module name is redirected to inda in requests.packages:

For package in ('urllib3',' idna', 'chardet'): locals () [package] = _ _ import__ (package) # This traversal is apparently necessary such that the identities are # preserved (requests.packages.urllib3.* is urllib3.*) for mod in list (sys.modules): if mod = = package or mod.startswith (package +'.): sys.modules ['requests.packages.' + mod] = sys.modules [mod]

With mocker, the _ _ version__ information of the idna can be controlled so that the idna results in the info can be expected. Then you can get the second tip: use mock to assist unit testing

2.2 Analysis of test_hooks implementation

Let's move on to see how hooks is tested:

From requests import hooks def hook (value): return value [1:] @ pytest.mark.parametrize ('hooks_list, result', ((hook,' ata'), ([hook, lambda x: None, hook], 'ta'),) def test_hooks (hooks_list, result): assert hooks.dispatch_hook (' response', {'response': hooks_list}) 'Data') = = result def test_default_hooks (): assert hooks.default_hooks () = = {' response': []}

The two interfaces of the hooks module, default_hooks and dispatch_hook, are tested. Among them, default_hooks is a pure function, and there is a return value with no arguments. This kind of function is the easiest to test, just check whether the return value is as expected. Dispatch_hook is more complicated and involves calls to callback functions (hook functions):

Def dispatch_hook (key, hooks, hook_data, * * kwargs): "Dispatches a hook dictionary ona given piece of data." Hooks = hooks or {} hooks = hooks.get (key) if hooks: # determine the hook function if hasattr (hooks,'_ call__'): hooks = [hooks] for hook in hooks: _ hook_data = hook (hook_data * * kwargs) if _ hook_data is not None: hook_data = _ hook_data return hook_data

Pytest.mark.parametrize provides two sets of parameters for testing. The first set of parameters hook and ata is very simple, hook is a function, the parameter will be cropped, the first bit will be removed, and ata is the expected return value. The parameter to response for test_hooks is Data, so the result should be ata. The first parameter in the second set of parameters will be more complicated and become an array, starting with the hook function, with an anonymous function in the middle, which has no return value, thus overriding the bypass branch of if _ hook_data is not None:. The execution process is as follows:

The hook function cuts the first place of Data and the rest of ata

Anonymous function does not modify the result, remaining ata

The hook function continues to tailor the first place of ata, leaving the ta

After testing, we can find that the design of dispatch_hook is very clever, using the pipeline pattern to string all the hooks together, which is different from the event mechanism. If we are careful, we can find that if hooks: there is no bypass test, which is not rigorous enough and violates our third skill:

The test covers all branches of the objective function as much as possible

2.3 Analysis of test_structures implementation

The test cases for LookupDict are as follows:

Class TestLookupDict: @ pytest.fixture (autouse=True) def setup (self): "LookupDict instance with" bad_gateway "attribute." Self.lookup_dict = LookupDict ('test') self.lookup_dict.bad_gateway = 502def test_repr (self): assert repr (self.lookup_dict) = "" get_item_parameters = pytest.mark.parametrize (' key, value', (('bad_gateway', 502), (' not_a_key') None) @ get_item_parameters def test_getitem (self, key, value): assert self.lookup_ keys [key] = = value @ get_item_parameters def test_get (self, key, value): assert self.lookup_dict.get (key) = = value

You can see that using the setup method with @ pytest.fixture initializes a lookup_dict object for all test cases; at the same time, pytest.mark.parametrize can be reused between different test cases, we can get the fourth tip:

Reuse test objects using pytest.fixture and test parameters using pytest.mark.parametriz

Through TestLookupDict's test_getitem and test_get, you can more intuitively understand the role of LookupDict's get and _ _ getitem__ methods:

Class LookupDict (dict):... Def _ getitem__ (self, key): # We allow fall-through here, so values default to None return self.__dict__.get (key, None) def get (self, key, default=None): return self.__dict__.get (key, default)

Get custom dictionary so that it can get values using the get method

_ _ getitem__ custom dictionary so that it can use [] to get values

CaseInsensitiveDict's test cases are tested in both test_structures and test_requests. The former is mainly basic testing, while the latter is biased towards the business usage level. We can see these two differences:

Class TestCaseInsensitiveDict:# class test def test_repr (self): assert repr (self.case_insensitive_dict) = = "{'Accept':' application/json'}" def test_copy (self): copy = self.case_insensitive_dict.copy () assert copy is not self.case_insensitive_dictassert copy = = self.case_insensitive_dictclass TestCaseInsensitiveDict:# use method test def test_delitem (self): cid = CaseInsensitiveDict () cid ['Spam'] =' someval'del cid [' SPam'] assert 'spam' not in cidassert len (cid) = = 0def test_contains (self): cid = CaseInsensitiveDict () cid [' Spam'] = 'someval'assert' Spam' in cidassert 'spam' in cidassert' SPAM' in cidassert 'sPam' in cidassert' notspam' not in cid

Using the above test method for reference, it is not difficult to come up with the fifth skill:

You can unit test the same object from different levels

Later test_lowlevel and test_requests also applied this technique.

2.4 utils.py

A generator (provided by the yield keyword) that can be written to env is built in utils and can be used as a context decorator:

Import contextlibimport os@contextlib.contextmanagerdef override_environ (* * kwargs): save_env = dict (os.environ) for key, value in kwargs.items (): if value is None:del os.keys [key] else: os.keys [key] = valuetry:yieldfinally:os.environ.clear () os.environ.update (save_env)

Here is an example of how to use it:

# test_requests.pykwargs = {var: proxy} # simulated control proxy environment variable with override_environ (* * kwargs): proxies = session.rebuild_proxies (prep, {}) def rebuild_proxies (self, prepared_request, proxies): bypass_proxy = should_bypass_proxies (url, no_proxy=no_proxy) def should_bypass_proxies (url, no_proxy):... get_proxy = lambda k: os.environ.get (k) or os.environ.get (k.upper ()).

Get the sixth tip: where environment variables are involved, you can use the context decorator to simulate multiple environment variables

2.5 utils test cases

There are many test cases in utils, so we choose some of them for analysis. Let's first look at the to_key_val_list function:

# object transfer list def to_key_val_list (value): if value is None:return Noneif isinstance (value, (str, bytes, bool, int)): raise ValueError ('cannot encode objects that are not 2 if isinstance') if isinstance (value, Mapping): value = value.items () return list (value)

The corresponding test case TestToKeyValList:

Class TestToKeyValList:@pytest.mark.parametrize ('value, expected', ([(' key', 'val')], [(' key', 'val')]), (' key', 'val'),), [(' key', 'val')]), ({' key': 'val'}, [(' key', 'val')], (None, None)) def test_valid (self, value) Expected): assert to_key_val_list (value) = = expecteddef test_invalid (self): with pytest.raises (ValueError): to_key_val_list ('string')

The focus is on the handling of exceptions using pytest.raise in test_invalid:

Tip 7: use pytest.raises to catch exceptions

TestSuperLen introduces several methods for IO simulation testing:

Class TestSuperLen:@pytest.mark.parametrize ('stream, value', ((StringIO.StringIO,' Test'), (BytesIO, baked Test'), pytest.param (cStringIO, 'Test',marks=pytest.mark.skipif (' cStringIO is None'),) def test_io_streams (self, stream) Value): "Ensures that we properly deal with different kinds of IO streams." assert super_len (stream ()) = 0assert super_len (stream (value)) = = 4def test_super_len_correctly_calculates_len_of_partially_read_file (self): "Ensure that we handle partially consumed file like objects." Ensure that we handle partially consumed file like objects. "" s = StringIO.StringIO () s.write ('foobarbogus') assert super_len (s) = 0@pytest.mark.parametrize (' mode, warnings_num', (('ritual, 1)) ('rb', 0),) def test_file (self, tmpdir, mode, warnings_num Recwarn): file_obj = tmpdir.join ('test.txt') file_obj.write (' Test') with file_obj.open (mode) as fd:assert super_len (fd) = = 4assert len (recwarn) = = warnings_numdef test_super_len_with_tell (self): foo = StringIO.StringIO ('12345') assert super_len (foo) = 5foo.read (2) assert super_len (foo) = 3def test_super_len_with_fileno (self): with open (_ _ file__) 'rb') as f:length = super_len (f) file_data = f.read () assert length = = len (file_data)

Using StringIO to simulate IO operations, you can configure tests for various IO. Of course, you can also use BytesIO/cStringIO, but unit test cases generally do not focus on performance, and StringIO is simple enough.

Pytest provides fixture of tmpdir to test file read and write operations.

You can use _ _ file__ for a read-only test of a file, and _ _ file__ represents the current file without side effects.

Tip 8: unit testing using IO simulation cooperation

2.6 how to test request-api

The testing of requests requires httpbin, which starts a local service, and pytest-httpbin, which installs a pytest plug-in, and the test case gives you the fixture of httpbin, which is used to operate the URL of the service.

Class functional TestRequestsrequests business test TestCaseInsensitiveDict case-insensitive dictionary test TestMorselToCookieExpirescookie expiration test TestMorselToCookieMaxAgecookie size TestTimeout response timeout test TestPreparingURLsURL preprocessing. Some piecemeal test cases

Frankly speaking: this test case is huge, reaching 2500 lines. It seems to be a fragmented case for a variety of businesses, and I haven't completely straightened out its organizational logic. I choose some interesting businesses to introduce. Let's take a look at the test of TimeOut:

TARPIT = 'http://10.255.255.1'class TestTimeout:def test_stream_timeout (self, httpbin): try:requests.get (httpbin (' delay/10'), timeout=2.0) except requests.exceptions.Timeout as e:assert 'Read timed out' in e.args [0] .args [0] @ pytest.mark.parametrize (' timeout', (0.1, None), Urllib3Timeout (connect=0.1, read=None)) def test_connect_timeout (self, timeout): try:requests.get (TARPIT Timeout=timeout) pytest.fail ('The connect () request should time out.') except ConnectTimeout as e:assert isinstance (e, ConnectionError) assert isinstance (e, Timeout)

Test_stream_timeout uses httpbin to create an interface that delays the response by 10 seconds, and then sets the request itself to 2 seconds, so that you can receive an error from the local timeout. Test_connect_timeout accesses a service that does not exist and catches a connection timeout error.

TestRequests tests the business processes of requests, and you can see at least two:

Class TestRequests:def test_basic_building (self): req = requests.Request () req.url = 'http://kennethreitz.org/'req.data = {' life': '42'} pr = req.prepare () assert pr.url = = req.urlassert pr.body =' life=42'def test_path_is_not_double_encoded (self): request = requests.Request ('GET' "http://0.0.0.0/get/test case") .prepare () assert request.path_url ='/ get/test%20case...def test_HTTP_200_OK_GET_ALTERNATIVE (self, httpbin): r = requests.Request ('GET', httpbin (' get')) s = requests.Session () s.proxies = getproxies () r = s.send (r.prepare ()) assert r.status_code = 200ef test_set_cookie_on_301 (self) Httpbin): s = requests.session () url = httpbin ('cookies/set?foo=bar') s.get (url) assert s.cookies [' foo'] = = 'bar'

To verify the url, you only need to prepare the request. In this case, the request is not sent, and without network transmission, the test case will be faster.

If you need response data, you need to use httbin to build real request-response data.

3. Underlying API test

Testserver builds a simple thread-based tcp service with _ _ enter__ and _ _ exit__ methods, and can be used as a context.

Class TestTestServer:def test_basic (self): "" messages are sent and received properly "question = b" success? "answer = b" yeah, success "def handler (sock): text = sock.recv (1000) assert text = = questionsock.sendall (answer) with Server (handler) as (host, port): sock = socket.socket () sock.connect ((host) Port) sock.sendall (question) text = sock.recv (1000) assert text = = answersock.close () def test_text_response (self): "the text_response_server sends the given text" server = Server.text_response_server ("HTTP/1.1 200OK\ r\ n" + "Content-Length: 6\ r\ n" + "\ r\ nroflol") with server as (host, port): r = requests.get ('http://{}:{}'.format(host, Port) assert r.status_code = = 200assert r.text = = u'roflol'assert r.headers ['Content-Length'] = =' 6'

The test_basic method carries on the basic check to the Server to ensure that both the sender and the sender can send and receive data correctly. First, the sock of the client sends the question, and then the server determines in the handler that the received data is question, then returns answer after confirmation, and finally the client confirms that it can receive the answer response correctly. The test_text_response method tests the http protocol incompletely. The http request is sent according to the specification of the http protocol, and the Server.text_response_server echoes the request. The following is the testcase that simulates that the anchor point of the browser will not be transmitted over the network:

Def test_fragment_not_sent_with_request (): "" Verify that the fragment portion of a URI isn't sent to the server. "" def response_handler (sock): req = consume_socket_content (sock, timeout=0.5) sock.send (b'HTTP/1.1 200OK\ r\ n'b'Content-Length:'+ bytes (len (req)) + b'\ r\ nroomb'\ r\ n'+req) close_server = threading.Event () server = Server (response_handler) Wait_to_close_event=close_server) with server as (host, port): url = 'http://{}:{}/path/to/thing/#view=edit&token=hunter2'.format(host, port) r = requests.get (url) raw_request = r.contentassert r.status_code = = 200headers, body = raw_request.split (b'\ r\ n\ r\ n\ n, 1) status_line, headers = headers.split (b'\ r\ n') 1) assert status_line = = b'GET / path/to/thing/ HTTP/1.1'for frag in (bounded viewpoints, baked edits, baked tokens, bounded hunters 2'): assert frag not in headersassert frag not in bodyclose_server.set ()

You can see that the requested path is / path/to/thing/#view=edit&token=hunter2, where the part after # is the local anchor and should not be transmitted over the network. In the above test case, the received response is judged and the response header and response body do not contain these keywords.

Combining the two levels of requests testing, we can come up with the ninth tip:

Construction simulation service coordination test

The above is all the content of the article "what are the skills of Python unit testing?" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.