In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/01 Report--
Xiaobian to share with you what the role of hadoop services in the framework of big data systems, I believe most people do not know much, so share this article for your reference, I hope you have a lot of gains after reading this article, let's find out together!
Hadoop service roles:
1. Zookeeper role: Zookeeper service refers to a cluster consisting of one or more nodes providing a service framework for cluster management. For clusters, Zookeeper services provide functions such as maintaining configuration information, naming, and providing distributed synchronization of HyperBase. It is recommended to have at least 3 nodes in Zookeeper clusters.
2. JDK role: JDK is the Java language software development kit, JDK is the core of the entire Java development, it contains Java runtime environment, Java tools and Java basic library.
3. Apache-Flume role: Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system provided by Cloudera. Flume supports customization of various data senders in the log system for collecting data; at the same time, Flume provides the ability to simply process data and write it to various data receivers (customizable).
4. Apache-Hive Role: Hive is a Hadoop-based data warehouse tool that maps structured data files to a database table and provides simple SQL query capabilities that convert SQL statements into MapReduce tasks for execution.
5. Apache-Storm role: Storm is a memory-level computation where data is imported directly into memory over the network. Reading and writing memory is n orders of magnitude faster than reading and writing disk. When the computational model is more suitable for streaming, Storm's streaming process saves the time of collecting data in batch processing.
6. Elasticsearch role: Developed in Java and distributed as open source under the Apache license, Elasticsearch is the current popular enterprise search engine. Designed for cloud computing, it can achieve real-time search, stability, reliability, speed, easy installation and use.
7. NameNode role: A node in an HDFS system that maintains the directory structure of all files in the file system and keeps track of which data nodes file data is stored in. When a client needs to retrieve a file from the HDFS file system, it communicates with NameNode to know which data node has the file the client needs. There can only be one NameNode in a Hadoop cluster. NameNode cannot be assigned other roles.
8. DataNode role: In HDFS, a DataNode is a node used to store data blocks.
9. Secondary NameNode role: A node that creates periodic checkpoints for data on NameNode. The node will periodically download the current NameNode image and log files, merge the log and image files into a new image file and upload it to the NameNode. Machines assigned the NameNode role should no longer be assigned the Secondary NameNode role.
10. Standby NameNode role: Standby mode NameNode metadata (Namescae information and Block are synchronized with Active NameNode metadata, but once switched to Active mode, NameNode service can be provided immediately.
11. JournalNode roles: Standby NameName and Active NameNode communicate via JournalNode to keep information synchronized.
12. HBase role: HBase is a distributed, column-oriented open source database. HBase provides BigTable-like capabilities on top of Hadoop. HBase is a subproject of Apache's Hadoop project. HBase differs from relational databases in that it is a database suitable for unstructured data storage. Another difference is HBase's column-based rather than row-based schema.
13. Kafka Role: Kafka is a high-throughput distributed publish-subscribe messaging system that handles all action flow data in consumer-scale websites. This action (web browsing, searching, and other user actions) is a key factor in many social functions on the modern web. This data is usually addressed by log processing and log aggregation due to throughput requirements. For log data and offline analytics systems like Hadoop, but with real-time processing constraints, this is a viable solution. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through clustering.
14. Redis Role: Redis is an open source, web-enabled, memory-based and persistent log, Key-Value database written in C and available in multiple languages.
15. Scala Role: Scala is a multi-paradigm programming language, a Java-like programming language designed to be scalable and integrate features of object-oriented and functional programming.
16. Sqoop role: Sqoop is a tool used to transfer data between Hadoop and relational databases. It can import data from a relational database (such as MySQL ,Oracle ,Postgres, etc.) into HDFS of Hadoop, and can also import HDFS data into relational databases.
17. Impala role: Impala is a new query system led by Cloudera. It provides SQL semantics and can query petabytes of big data stored in HDFS and HBase of Hadoop. Although the existing Hive system also provides SQL semantics, because Hive uses MapReduce engine at the bottom, it is still a batch processing process, which is difficult to satisfy the interactivity of queries. In contrast, Impala's biggest feature and biggest selling point is its speed.
18. Crawler role: Crawler is a proprietary component of DKHadoop, crawler system, crawling dynamic static data.
19. Spark Role: Spark is an open source clustered computing environment similar to Hadoop, but there are some useful differences that make Spark superior for certain workloads, in other words, Spark enables in-memory distributed datasets that optimize iterative workloads in addition to providing interactive queries. Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala can be tightly integrated, with Scala making it easy to manipulate distributed data sets as easily as native collection objects.
20. HUE role: HUE is a set of network applications that can interact with your Hadoop cluster. The HUE app lets you browse HDFS and jobs, manage Hive metastores, run Hive, browse HBase Sqoop export data, submit MapReduce programs, build custom search engines and schedule repetitive workflows with Solr.
The above is all the contents of this article "What are the service roles of hadoop in the framework of big data system?" Thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 259
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.