Shulou(Shulou.com)06/02 Report--

This article mainly explains the "which difficult to understand Python library", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's train of thought slowly in depth, together to study and learn "what is difficult to understand Python library" bar!

1. Scrapy

Every data scientist's project starts with data processing, and the Internet is the largest, richest and most accessible database. Unfortunately, data scientists have no idea when it comes to grabbing data from websites with complex data structures, except through the pd.read_html function.

Web crawlers are often used to analyze the structure of Web sites and store extracted information, but Scrapy makes this process easier than rebuilding web crawlers.

The Scrapy user interface is very simple and easy to use, but its biggest advantage is its efficiency. Scrapy can send, schedule, and process Web site requests asynchronously, that is, while it takes time to process and complete one request, it can also send another request. Scrapy iterates over site content in the most efficient way by sending multiple requests to a site at the same time, using very fast crawling.

In addition to the above advantages, Scrapy also allows data scientists to export archived data in different formats (such as JSON,CSV or XML) and different backends (such as FTP,S3 or local).

Image source: unsplash

2. Statsmodels

What kind of statistical modeling method should be adopted? Every data scientist has been hesitant about this, but Statsmodels is an option that must be understood. It can implement important algorithms (such as ANOVA and ARIMA) that are not found in standard machine learning libraries such as Sci-kit Learn, and its most valuable lies in its detailed processing and information applications.

For example, when a data scientist wants to use Statsmodels to calculate an ordinary least square method, Statsmodels can provide all the information he needs, whether it's a useful metric or detailed information about coefficients. The same is true of all other models implemented in the library, which are not available in Sci-kit learn.

OLSRegressionResults = Dep. Variable: Lottery R-squared: 0.348 Model: OLS Adj. R-squared: 0.333 Method: LeastSquares F-statistic: 22.20 Date: Fri, 21Feb2020 Prob (F-statistic): 1.90e-08 Time: 13:59:15 Log-Likelihood:-379.82 No. Observations: 86 AIC: 765.6 DfResiduals: 83 BIC: 773.0 DfModel: 2 CovarianceType: nonrobust = coef std err t P > | t | [0.025 0.975]- Intercept 246.4341 35.233 6.995 0.000 176.358 316.510 Literacy-0.4889 0.128-3.832 0.000-0.743-0.235 np.log (Pop1831)-31.3114 5.977-5.239 0.000-43.199-19.424 = Omnibus: 3.713 Durbin-Watson: 2.019 Prob (Omnibus): 0.156 Jarque-Bera (JB): 3.394 Skew:-0.487 Prob (JB): 0.183 Kurtosis: 3.003 Cond. No. 702. =

It's important for data scientists to have this information, but their problem is that they often trust a model they don't really understand. Because high-dimensional data is not intuitive enough, it is necessary for data scientists to have an in-depth understanding of data and models before deploying them. If you blindly pursue performance indicators such as accuracy or mean square error, it may have a serious negative impact.

Statsmodels not only has extremely detailed statistical modeling, but also provides a variety of useful data characteristics and metrics. For example, data scientists often perform temporal decomposition, which can help them better understand the data and analyze which transformations and algorithms are more appropriate, or pinguoin can be used for a less complex but very accurate statistical function.

Image source: Statsmodels

3. Pattern

Some mature websites may have more specific methods for retrieving data, in which case writing Web crawlers in Scrapy is a bit "overkill", and Pattern is the more advanced Web data mining and natural language processing module in Python.

Pattern not only seamlessly integrates data from Google, Twitter and Wikipedia, but also provides a less personalized Web crawler and HTML DOM parser. It uses part of speech tagging, n-grams search, emotion analysis and WordNet. Whether it is clustering analysis, classification processing, or network analysis visualization, the text data preprocessed by Pattern can be used in a variety of machine learning algorithms.

From data retrieval to preprocessing to modeling and visualization, Pattern can deal with all the problems in the process of data science, and it can also quickly transfer data in different libraries.

Image source: unsplash

4. Mlxtend

Mlxtend is a library that can be applied to any data science project. It can be said to be an extension of the Sci-kit learn library to automatically optimize common data science tasks:

Automatic feature extraction and selection.

Extend the existing data converters of the Sci-kit learn library, such as centralization and transaction encoders.

A large number of evaluation indicators, including deviation variance decomposition (that is, deviation and variance in the measurement model), feature point detection, McNemar test, F test and so on.

Model visualization, including feature boundary, learning curve, PCA interaction circle and enrichment drawing.

Contains many built-in datasets that are not available in Sci-kit Learn libraries.

Image and text preprocessing functions, such as name generalizers, can recognize and transform text with different naming systems (e.g., it can recognize "Deer,John", "J.Deer", "J.D." Is the same as "John Deer").

Mlxtend also has very useful image processing features, such as the ability to extract facial signs:

Image source: Mlxtend

Let's take a look at its decision boundary drawing function:

Image source: Mlxtend

5. REP

Like Mlxtend, REP can also be seen as an extension of Sci-kit learning library, but more in the field of machine learning. First, it is a unified Python wrapper for different machine learning libraries extended from Sci-kit-learn. It can integrate Sci-kit learn with more professional machine learning libraries such as XGBoost, Pybrain, Neurolab and so on.

For example, when data scientists want to convert a XGBoost classifier to a Bagging classifier through a simple wrapper, and then convert it into a Sci-kit-learn model, only REP can do that, because an algorithm like this that is easy to wrap and transform cannot be found in other libraries.

From sklearn.ensemble importBaggingClassifier from rep.estimators importXGBoostClassifier, SklearnClassifier clf = BaggingClassifier (base_estimator=XGBoostClassifier (), n_estimators=10) clf = SklearnClassifier (clf)

In addition, REP can transform models from any library to cross-validation (folding) and stacking models. It has an extremely fast grid search function and a model factory, which can help data scientists to effectively use multiple machine learning classifiers in the same data set. Using REP and Sci-kit learn at the same time can help us build models more easily.

Thank you for your reading, the above is the content of "what is difficult to understand Python library", after the study of this article, I believe you have a deeper understanding of what is difficult to understand Python library, the specific use of the situation also needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.