Shulou(Shulou.com)06/01 Report--

Database data is too big memory how to do, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

In the experience of studying and applying machine learning algorithms, it is believed that people often encounter situations where the data set is too large and there is not enough memory.

This leads to a series of questions:

How to load tens or dozens of GB data files?

What if the algorithm crashes while running the dataset?

How to deal with errors caused by insufficient memory?

Seven ideas for dealing with large ML data Files

1. Allocate more memory

Some machine learning tools / libraries have default memory settings, such as Weka. This is a limiting factor.

You need to check whether you can reset the tool / library to allocate more memory.

For Weka, you can adjust the memory as a parameter when you open the application.

two。 With a smaller sample

Do you really need to use all the data?

You can take a random sample of data, such as the first 1000 or 100000 rows. Before training the final model on all the data (using progressive data loading techniques), try to solve the problem with this small sample.

Generally speaking, it is a good habit in the field of machine learning to do a quick spot check on the algorithm and see the changes of the results before and after.

You can also consider doing a sensitivity analysis of data size relative to modeling skills. Perhaps, for your random small sample, there is a natural marginal effect decreasing watershed. Beyond this threshold, the benefits of the continued increase in the size of the data are minimal.

3. More memory

Do you have to use PC?

You can consider computing devices with memory and an order of magnitude higher performance. For example, rent a cloud service like AWS. Machines with dozens of GB of memory in the cloud are rented for less than $1 an hour. Personally, I think this is a very practical choice.

4. Convert data format

Do you save the data as raw ASCII text, such as CSV files?

Perhaps, using other formats can speed up data loading and reduce memory footprint. Good choices include binary formats such as GRIB, NetCDF, and HDF.

There are many command-line tools that can help you convert data formats without having to load the entire dataset into memory.

Changing the format may help you store data in a more compact form, saving memory space, such as 2-byte integers, or 4-byte floating points.

5. Streaming data, or progressive data loading

Does all your data need to be in memory at the same time?

Perhaps, you can use code or library to stream or gradually load the data you need at any time and import it into memory to train the model.

This may require iterative learning using optimization techniques, such as random gradient descent. Algorithms that require all the data in memory for matrix operations, such as some implementations of linear and logical regression, are not applicable.

For example, Keras Deep Learning API provides the ability to load image files progressively, called flow_from_directory

Another example Pandas library, which can bulk load large CSV files.

6. Using a relational database (Relational database)

Relational database provides a standardized method for storing and accessing large data sets.

Internally, the data is stored in the hard disk and can be loaded in batches by in batch gradually, and retrieved using the standard retrieval language SQL.

Open source database tools like MySQL and Postgres support the vast majority of (all?) programing language. Many machine learning tools can be directly connected to relational databases. You can also use a lighter method like SQLite.

I find this approach very efficient for large tabular datasets.

Lei Feng reminds you that you need an algorithm that can learn iteratively.

7. Use big data platform

In some cases, you may have to use the big data platform, which is developed to deal with very large datasets. They allow you to convert data and develop machine learning algorithms on it.

Two good examples are Hadoop and the machine learning library Mahout, and the Spark and MLLib libraries.

In my opinion, this is a means that needs to be adopted only when the above-mentioned methods cannot be solved. Just the extra hardware and software complexity that this brings to your machine learning project will consume a lot of energy.

Even so, some tasks do have too much data to work with.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.