How to handle duplicate values in configuration files

This article mainly explains "how to deal with the duplicate values in the configuration file". The content in the article is simple and clear, and it is easy to learn and understand. let's study and learn how to deal with the duplicate values in the configuration file.

When we create a configuration file, there may be problems with some values that need to be filled in repeatedly. For example, I recently wrote a configurable crawler that only needs to configure a few XPath to automatically generate a Scrapy crawler. So as to quickly complete the crawling of simple websites.

The configuration file looks like this:

Name: a website crawler host: 'https://www.kingname.info' headers: user-agent: xxx host: yyyy referer: zzz rule: start_url:' https://www.kingname.info' detail_url: / / div [@ class= "xxx"] / a/@href next_page_xpath: / / div [@ class= "next"] / @ href... Other configuration parameters.

This configuration file is created using the YAML format. We can see that the value of the outermost host is https://www.kingname.info, and in rule, the value of start_url is also this URL. This repeats itself. There are many such repetitions, for example, the XPath of the flip link of the list page is the same as the XPath of the flip link of the body page, multiple types can have the same execution rules, and so on.

If you use JSON as the format of the configuration file, then you do have to repeat it. But if you use YAML as the configuration file format, you can fill in once and use it many times by adding anchor and alias.

Let's start with a simple example:

Import yaml config = 'name: & name Qingnan salary: 99999 other_name: * name' info = yaml.safe_load (config) print (info)

The running effect is shown in the following figure:

As you can see, the value that originally defines a key-value type should be in the form of key: value, but here I write it as the key: & anchor name value, so the anchor name is equivalent to a variable name that can be referenced elsewhere. When quoting, write * anchor names. This is a bit like getting the memory address of a variable (&) in C and then displaying the value of the pointer (*). The anchor name can be the same key or different.

Of course, in addition to simple key-value, it can also be used in complex scenarios, such as:

Import yaml config = 'article_xpath: & article title: / / div [@ class= "title"] / text () detail: / / div [@ class= "content"] / text () image:-/ / div/img [@ class= "xx"] / @ href-/ / p/img [@ class= "yy"] / @ href about_xpath: summary: / / div [@ class= "summary"] / text () book_xpath: * article''info = yaml.safe_load (config)

The running effect is shown in the following figure:

In this way, if a configuration that will occur multiple times is changed, we only need to change one place to automatically take effect in multiple places at the same time, without having to modify it manually. It saves time and is not easy to make mistakes.

Thank you for your reading. the above is the content of "how to deal with the duplicate values in the configuration file". After the study of this article, I believe you have a deeper understanding of how to deal with the duplicate values in the configuration file. the specific use also needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!