Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "method steps of data cleaning and conversion". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations. I hope you can read carefully and learn something!

01 Understanding Data Sets

The critical and iterative phase of data preparation is data exploration. A set of data that is too large to be manually read, checked, and edited for each value still needs to be verified for quality and suitability before it can be delegated to a model that is worth the time and calculations.

As simple as dumping a sample of a large dataset into a spreadsheet program, simply look at the type or range of values appearing in each column to identify errors such as irresponsible defaults (for example, using zeros instead of NULL when there are no measurements) or impossible ranges or incompatible merges (data appears to come from multiple sources, each with different units used). For example, degrees Fahrenheit and degrees Celsius).

Data analysis tools are very rich. Python scripts or applications like RStudio have powerful capabilities for visualizing, summarizing, or reporting data when the dataset is too large to open in a spreadsheet program. Using any method you are familiar with, determine at least the format and general distribution of the values of the different attributes.

02 Data processing tools

There are many tools available to clean, manipulate, and understand a dataset before it can be used. Python is the de facto standard for this, and it has many tools to understand and manipulate data.

Packages such as Matplotlib often make it easy to generate data graphs for visual inspection.

Pillow offers a variety of functions for processing, converting and manipulating images.

Python has a built-in package for performing statistics, as does NumPy if more functionality is needed.

Python also has extensive built-in and third-party support to handle almost any file format you'll encounter, including CSV, JSON, YAML, XML, and HTML, as well as more esoteric formats such as TOML or INI files.

If none of this works, there's a package indexer worth searching for to see if there's a solution to your problem. Or, just search for "I want to do things with Python" and most of the time you'll find someone who has the same problem and has a solution for it, or at least some guidelines to look at.

If you don't like Python, almost all programming languages of choice have similar tools and features. What we like about Python is that it's already done for you, and there are plenty of examples to start with. Python is nothing magical in this respect, but it is the most popular choice, so we advocate sticking with mainstream tools.

Another good option is a spreadsheet program such as Excel, Numbers or Google Sheets. They are often criticized because data preparation in these programs can be cumbersome, but you can use them to gain a lot of useful insight and preparation very quickly before you need to use Python(or another tool of your choice). As a bonus tool to the system, you're almost certain to have one installed and ready to run on your machine.

Finally, don't be afraid to think outside the box-something as simple as compressing a dataset can give you a rough idea of the entropy of a dataset without even looking inside it. If one dataset compresses very well and another dataset from the same source compresses less well, the entropy of the data in the second dataset may be greater than that in the first dataset.

Image datasets are not easy to observe, but it's definitely worth taking the time to look at the overall quality of the images and what cropping methods were used. Visualizations like Turi Create are useful for understanding data. Figure 3-1 shows an example.

Figure 3-1: Understanding Your Data with Turi Create

03 Cleaning data

In the process of understanding the dataset, you may encounter some errors. Recording data may produce errors. There are several types of errors that need to be checked:

consensus error

single-valued error

missing values

Consistent value errors include situations that may cause an entire column or set of values to be inaccurate, such as using an instrument to record data that is incorrectly calibrated for a uniform quantity, measuring temperature from an object that generates extra heat, weighing using a balance that is not zeroed in advance, and so on. This also includes cases where data from different sources were improperly combined without conversion: simply compressing one set of data from the US and one from the UK, and now the system thinks 100 degrees Celsius is perfectly reasonable.

Single-valued errors are used to describe outliers or inconsistent miscalibrations that result in inaccurate or completely illogical values only in a few cases. Possible scenarios, such as overloading the sensor for one day, produce values 1000% higher than theoretically possible (which should be quite noticeable).

Missing values can occur when there is a problem with the method used to record the data, or when the dataset undergoes some kind of misshapen transformation at some point in its lifecycle. These could be simple nil or NULL values, or something less useful, such as the string "NONE" or the default value of 0. Some may even be meaningless characters, anything is possible.

If a consistent error can be identified, this can usually be corrected by scaling or transforming the entire value set by the consistent error value. Single-valued errors and missing values require you to either guess the value that needs to be replaced in some feasible way, or delete rows entirely or observe values to prevent errors.

You can guess this value by taking the average of all the other values in the column, using the observation in the column that is closest to the missing value, or using some application-specific method that uses knowledge of other attributes.

04 Conversion of data

There are two main reasons to transform data before using it: to meet the format requirements of the algorithm you want to use; and to improve or extend current data with new inferred attributes. For these two purposes, there are usually three data transformations:

1. Normalization

A method for numerical data that binds upper and lower bounds to a numerical range, making them easier to handle.

An example of this is when observations of numerical data need to be compared to different metrics. If you try to assess the health of different fish based on their length, weight, age, and number of eyes lost, presumably everyone will agree to compare two fish by different criteria (e.g., one eye versus a year-old fish, or one centimeter in length). If the same criteria are used for comparison, different results will be obtained.

Normalization to positive is simple:

2. Generalization

A method of replacing specific values with higher-level concepts for better crowd observation.

This usually happens when the method of recording certain attributes is more accurate than needed. For example, if you have GPS statistics for someone's movements, you can reduce latitude and longitude to one address, preventing the system from seeing every small movement as a change in position. Alternatively, numerical measurements are converted to human populations, meaning that the correlation factor may not be measuring an individual's height in millimeters, but classifying it as below, near, or above average height.

3. aggregation

A way to summarize certain complex attributes to make analysis more efficient.

For example, instead of analyzing paragraphs of text (Attribute: Text, Classification: Class), keywords (or even word frequencies) can be extracted from the text to show only the aspects that are most relevant or unique to the given classification.

Before, between, or after these steps, different types of data transformations may occur, and data may be altered, expanded, or reduced:

Feature construction

A method of creating new attributes, usually by reasoning or combining other values that already exist.

An example of this is generalization or aggregation, where the original value is also preserved, or more commonly, when two or more values exist (or a third value is allowed to be found). For example, if you have the name of a company and the country in which it operates, you can look up its business registration number; if you have someone's height and weight, you can construct their BMI.

Data reduction

A way to delete certain attributes that may or may not be related to another attribute or to the problem you are trying to solve.

For example, if you have someone's address, zip code, and area code, at least one of these pieces of information is redundant. Maybe--as in the case of feature construction--you want to analyze both for algorithmic reasons, but that's impossible. A high correlation between two or more attributes indicates that they may cause errors in the analysis and may be deleted.

05 Verify the suitability of the dataset

At this point, you should spend more time looking closely at the problem you are trying to solve and the data set you intend to use for the task. In the pre-AI world of data analytics, there may not be as strict rules as you want, but you usually know if a solution works and if a dataset tells the story you want.

Trust that little voice, because if you look back, wasted work is worth it.

Explore your data again. Browse through and visualize it, test your solution with a small subset of data-do whatever you need to do. If it still feels right, move on.

"Data cleaning and conversion method steps" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.