Shulou(Shulou.com)06/03 Report--

With the rapid development and progress of IT Internet information technology. At present, the big data industry is getting more and more popular, resulting in an extreme shortage of big data talents in China. Here are some tips on managing big data storage in Hadoop environment.

1. Distributed storage

Traditional centralized storage has been around for some time. But big data is not really suitable for centralized storage architectures. Hadoop is designed to bring computation closer to the data node while taking advantage of the massive scale-out capabilities of HDFS file systems.

Although, the usual solution to Hadoop's own data inefficiency is to store Hadoop data on a SAN. But it also creates bottlenecks in its own performance and scale. Now, if you process all your data through centralized SAN processors, it goes against the distributed and parallelized nature of Hadoop. You can either manage multiple SANs for different data nodes or centralize all data nodes into one SAN.

But Hadoop is a distributed application and should run on distributed storage, which retains the same flexibility as Hadoop itself, but it also requires embracing a software-defined storage solution and running on commercial servers, which is naturally more efficient than bottleneck Hadoop.

 2. Hyper-convergence VS distributed

Be careful not to confuse hyperconvergence with distribution. Some hyperconverged scenarios are distributed storage, but often the term means that your applications and storage are kept on the same compute node. This is an attempt to solve the problem of data localization, but it causes too much contention for resources. This Hadoop application and storage platform compete for the same memory and CPU. Hadoop runs on a proprietary application layer and distributed storage runs on a proprietary storage layer, which is better. Caching and layering are then used to address data localization and compensate for network performance losses.

3. Avoid Controller Choke Point

An important aspect of achieving this goal is avoiding processing data through a single point, such as a traditional controller. Conversely, to ensure storage platform parallelism, performance can be significantly improved.

In addition, the scheme provides incremental scalability. Adding functionality to a data lake is as easy as throwing x86 servers into it. A distributed storage platform automatically adds functionality and realigns data as needed.

4. Deletion and compression

The key to mastering big data is deletion and compression technology. Usually 70 to 90 percent of data reduction occurs within large data sets. Tens of thousands of dollars in disk cost savings in petabytes. Modern platforms offer inline (vs. post-processing) erasure and compression, greatly reducing the capacity required to store data.

 5. Merge Hadoop distributions

Many large enterprises have multiple Hadoop distributions. It could be developer needs or enterprise departments that have adapted to different versions. In any case, it is often necessary to maintain and operate these clusters. Once massive amounts of data really start to impact an enterprise, multiple Hadoop distributions store inefficiencies. We can achieve data efficiency by creating a single, deletable and compressed data lake

 Virtualizing Hadoop

Virtualization has swept the enterprise market. More than 80% of physical servers in many regions are now virtualized. However, there are still many enterprises that avoid virtualized Hadoop because of performance and data localization issues.

7. Create an elastic data lake

Creating a data lake isn't easy, but there may be demand for big data storage. There are many ways to do this, but which one is right? The right architecture should be a dynamic, resilient data lake that can store data for all resources in multiple formats (architected, unstructured, semi-structured). More importantly, it must support application execution not on remote resources but on local data resources.

Unfortunately, traditional architectures and applications (i.e., non-distributed) are not ideal. As data sets get larger, migrating applications to data is inevitable, and the latency is too long to reverse.

An ideal data lake infrastructure would enable storage of a single copy of data and have applications executing on a single data resource without migrating data or making copies.

8. Integrated analysis

Analytics is not a new feature; it has been around for years in traditional RDBMS environments. The difference is based on the emergence of open-source applications and the ability to integrate database forms and social media, unstructured data resources (e.g. Wikipedia). The key lies in the ability to consolidate multiple data types and formats into a single standard, making visualization and reporting easier and more consistent. The right tools are also critical to the success of analytics/business intelligence projects.

conclusion

Thank you for watching. If there are any shortcomings, please comment and correct them.

In order to help you make learning easy and efficient, I will share a large amount of information for free to help you overcome difficulties on the road to becoming a big data engineer and even an architect. Here I recommend a big data learning exchange circle: 658558542 Welcome to ×× flow discussion, learning exchange and common progress.

When you really start learning, you inevitably don't know where to start, which leads to inefficiency and affects confidence in continuing learning.

However, the most important thing is that I don't know which technologies need to be mastered. I often step on pits when learning, and eventually waste a lot of time. Therefore, it is necessary to have effective resources.

Finally, I wish all big data programmers who encounter bottle diseases and don't know what to do, and wish everyone all the best in their future work and interviews.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.