Shulou(Shulou.com)06/03 Report--

The past two days have really been scanned by the commercial "what is Page". Paige is obviously a comedy character, but he makes everyone cry!

In the middle of the plot, the little grandson said, "I want Page." as a result, Grandpa began to search for Page all over the village, and finally found Page, who the editor thought was the most beautiful.

I don't know how everyone feels after watching it. Anyway, I feel very moved after reading it. But after a few days of fermentation, the word "Page" seems to have more meaning! All kinds of "Page" emerge endlessly, what is a woman's "Page" like? What is the programmer's "Page" like?

Here I still want to recommend the big data Learning Exchange Group I built myself: 529867072, all of them are developed by big data. If you are studying big data, the editor welcomes you to join us. Everyone is a software development party. Irregularly share practical information (only related to big data software development), including the latest big data advanced materials and advanced development tutorials sorted out by myself. Welcome to join us if you want to go deep into big data.

Today, I would like to share with you what engineer big data's "Page" is like!

"Page" skill

1. Programming ability

Whether it's Java or Python, to learn a programming language, you must first settle down to specialize in a particular subject, especially open source tools, which are widely used in any company.

For example, learning the basic syntax of Java language, OOP programming, multithreading and network programming, MySQL database, Maven project management and other development tools can train yourself to master the basic coding skills necessary for big data, and also lay a solid foundation for the follow-up study of advanced content such as big data's analysis or recommendation system.

2.Hadoop

Hadoop plays an important role in big data's technology system. Hadoop is the foundation of big data's technology. The solid degree of mastering the basic knowledge of Hadoop will determine how far to go on big data's technological road. Hadoop includes several components HDFS, MapReduce and YARN,HDFS are the places where data is stored, just like our computer's hard disk, files are stored on this, MapReduce is the data processing calculation, it has a characteristic is that no matter how big the data is, as long as you give it time, it can run the data, but the time may not be very fast, so it is called data batch processing.

YARN is an important component that embodies the concept of Hadoop platform. With its big data ecosystem, other software can run on hadoop, so that we can make better use of the advantages of HDFS large storage and save more resources. For example, we no longer have to build a separate spark cluster, just let it run on the existing hadoop yarn. The following is a common module architecture diagram of Hadoop:

3.Spark

It is used to make up for the shortcomings of data processing speed based on MapReduce, which is characterized by loading data into memory for computing rather than reading slow, slow-evolving hard drives. It is especially suitable for iterative operations, so algorithm streams are particularly fond of it. It is written in scala. Either the Java language or Scala can operate on it because they all use JVM.

4.Storm

Storm is a free and open source distributed real-time computing system. Using Storm can easily and reliably handle unlimited data streams, just like Hadoop batch processing big data, Storm can process data in real time. Storm is simple and can be used in any programming language.

5.Kafka

Kafka is a distributed,partitioned,replicated commit logservice . It provides features similar to JMS, but is completely different in design and implementation, and it is not an implementation of the JMS specification. Kafka classifies messages according to Topic when they are saved, the sender becomes Producer and the message receiver becomes Consumer. In addition, the kafka cluster is composed of multiple kafka instances, and each instance (server) becomes broker. Both kafka clusters, producer and consumer rely on zookeeper to ensure system availability that the cluster holds some meta information.

6.Flink

Flink is a distributed computing engine, which can be used to do batch processing, that is, to deal with static and historical data sets; it can also be used to do stream processing, that is, to process some real-time data streams in real time and produce data results in real time; it can also be used to do some event-based applications, such as Didi to monitor the behavior flow of users and drivers in real time through Flink CEP to judge whether the behavior of users or drivers is legitimate. Big data Learning Exchange Group: 529867072

7.Hive

Hive is implemented by Facebook and open source

Is a data warehouse tool based on Hadoop

You can map structured data to a database table

And provide HQL (Hive SQL) query function.

The underlying data is stored on HDFS

The essence of Hive is to convert SQL statements into MapReduce tasks to run.

It is convenient for users who are not familiar with MapReduce to use HQL to process and calculate structured data on HDFS, which is suitable for offline batch data calculation.

8.ElacsticSearch

ES is a Lucene-based distributed full-text search server, similar to SQL Server's full-text index (Fulltext Index). It is a full-text search engine based on word segmentation and segmentation, with the functions of word segmentation, synonym and stem query, but ES is inherently distributed and real-time. This essay demonstrates installing ElasticSearch in the Windows environment and the Head plug-in for managing ElasticSearch.

Summary

In the technology industry, something new appears every day, and we need to pay attention to the latest technological developments and keep learning. Any general technology is a process of learning the theory first, and then constantly perfecting the theory in practice.

If you think you are too slow to read, you can collect some courses online.

The ability to learn quickly, problem-solving and communication skills are really important indicators in this industry.

Be good at using StackOverFlow and Google to help you with the problems encountered in your learning process.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.