Shulou(Shulou.com)06/03 Report--

Users often ask this question, how much data can your product handle? It seems that the higher the value, the better the product.

In fact, this question does not make much sense.

How much data can be handled, another key factor is the expected response time. When we talk about the data processing capacity of big data's products without this factor, I don't know how to answer it.

Consider a simple case where there is only a single machine. If you want an OLAP-style summary of seconds of response, then GB-level data is quite large, and it is almost impossible for any product to handle TB-level data (unless there is a lot of memory). And if the ETL operation is completed in a few hours, then a single machine to deal with the TB level is not much of a problem. Therefore, this phenomenon occurs: the same product can handle TB-level data in some situations, but not GB-level data in other situations.

However, it makes sense to discuss the size of a cluster that big data technology can support.

The implementation technology of large cluster and small cluster is very different, large cluster needs strong fault tolerance and unified management mechanism, while small cluster does not; and using large cluster technology in small cluster will cause a lot of waste of resources. How large a technology is for a cluster is generally determined at the beginning of the design and is not easy to change at will.

After determining the expected response time and the size of the cluster that may be supported, it is a meaningful question to ask how much data a big data technology can handle.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.