Shulou(Shulou.com)06/03 Report--

Big data's four V statements are already well known in the industry, which refers to the characteristics of big data itself. Now let's take a look at the characteristics of the technology used to deal with big data. In order to facilitate memory, similar to 4 V, we summarize these features into 4 E, which can be used as a reference when users choose big data technical solution.

1. Easy big data technology should be simple and easy to use.

This E is easy to understand.

There are many scenes to be dealt with by big data, and there are all kinds of staff involved. If the technology is too difficult, it will lead to only a few people can apply it, and the implementation complexity is high, so big data's application will be greatly reduced.

There are many examples in big data's field. When Hadoop first came out, there was only MapReduce. Compared to hard writing in Java, MapReduce is much easier, so it will accumulate a number of fans. However, the difficulty of MapReduce is still not small, so it is gradually replaced by the later packaged HIVE SQL. Scala on Spark has been popular for a while, but it is still a lot of difficulty, and now it has gradually calmed down, and more people are still willing to use the simpler Spark SQL.

2. Elastic big data technology should have the ability of flexible expansion.

This E is also easy to understand.

In many cases, big data is not very big all at once, but becomes bigger gradually. Even if the data is already large, it will become even bigger. Therefore, it is natural to require big data's processing technology to have a certain flexibility to expand, which is generally not ignored by big data technology providers.

Of course, any technology has its limitations, and there is a big difference between general scale and very large-scale technology. it is unlikely that a technology can effectively adapt to all stages of the data scale from 0 to infinity (the so-called effective adaptation is that the technology can achieve excellent performance at all stages, not just can be handled). Users should also have an estimate of the range of changes in their data scale when choosing technologies.

3. Embeddable big data technology should be embedded and integrated.

This E needs to be specifically pointed out and is often ignored.

Big data processing is often not an independent thing, it needs to work with specific applications to give full play to its business value, and these processes often need to be carried out when the application is executed to a certain link. this requires that the corresponding technology can be easily embedded into the application and called by the main program anytime and anywhere.

In particular, most applications are based on the J2EE architecture, so the integrability of Java applications is a particularly important indicator. Generally, big data technology based on Java or SQL system has no big problem in integration, while other technology systems are difficult to say. Moreover, most big data technologies often need to be deployed independently, even if their computing power can be integrated, but must rely on external independent processes, can not be completely controlled by the application, and sometimes seem very cumbersome.

4. Environment-friendly big data technology requires low data environment as much as possible.

This E is a lot of big data technology does not have, but it is very important.

The current big data technology, such as Hadoop and MPP, requires that the data be put into some kind of storage system stipulated by this technology first. This certainly makes sense, and the data will achieve higher performance after being organized in advance. However, often, the big data we need to deal with is not in these storage systems in advance, and moving external data into these storage systems is also a kind of big data processing. These big data technologies can no longer be used in these scenarios.

Better big data technology should not be able to pick a data source, data from any source can be processed, but the performance may be degraded to a certain extent because of the limitations of the data source, but it does not require a good ETL before it can be processed.

In fact, the last feature is not very appropriate to use E, but in order to put together 4 E to deal with. The word originally means environmental protection. The open big data technology can copy less data, deploy less hardware and save electricity.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.