Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge about "Hadoop or Spark which is better". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

The main modules of the Hadoop framework include the following:

Hadoop Common

Hadoop Distributed File System (HDFS)

Hadoop YARN

Hadoop MapReduce

While these four modules form the core of Hadoop, there are several others. These modules include Ambari, Avro, Cassandra, Hive, Pig, Oozie, Flume, and Sqoop, which further enhance and extend Hadoop's capabilities.

Spark is indeed fast (up to 100 times faster than Hadoop MapReduce). Spark can also perform batch processing, but what it really excels at is handling streaming workloads, interactive queries, and machine learning.

Compared to MapReduce's disk-based batch processing engine, Spark is known for its real-time data processing capabilities. Spark is compatible with Hadoop and its modules. In fact, on the Hadoop project page, Spark is listed as a module.

Spark has its own page because while it can run in a Hadoop cluster via YARN(another resource coordinator), it also has a standalone mode. It can be run as a Hadoop module or as a standalone solution.

The main difference between MapReduce and Spark is that MapReduce uses persistent storage, while Spark uses Resilient Distributed Datasets (RDDS).

performance

The reason Spark is so fast is that it processes everything in memory. Yes, it can also use disk to process data that is not fully loaded into memory.

Spark's in-memory processing provides near-real-time analytics for data from multiple sources: marketing campaigns, machine learning, IoT sensors, log monitoring, security analytics, and social media sites. MapReduce uses batch processing and was never designed for incredible speeds. It is intended to collect information from websites on an ongoing basis, without requiring that the data be real-time or near-real-time.

ease of use

Supports Scala(native language), Java, Python and Spark SQL. Spark SQL is very similar to SQL 92, so there's almost no learning required to get started right away.

Spark also has an interactive mode where both developers and users get instant feedback on queries and other actions. MapReduce has no interactive mode, but with additional modules such as Hive and Pig, it is easier for adopters to use MapReduce.

cost

"Spark has proven to be comfortable with petabytes of data. It was used to sort 100 terabytes of data three times faster than Hadoop MapReduce on one-tenth the number of machines. "This achievement makes Spark the benchmark for Daytona GraySort 2014.

compatibility

MapReduce and Spark are compatible with each other;MapReduce is compatible with many data sources, file formats, and business intelligence tools through JDBC and ODC, and Spark has the same compatibility as MapReduce.

data processing

MapReduce is a batch processing engine. MapReduce operates in sequential steps, reading data from the cluster, performing operations on the data, writing the results back to the cluster, reading updated data from the cluster, performing the next data operation, writing those results back to the results, and so on. Spark performs a similar operation, but in one step in memory. It reads data from the cluster, performs operations on the data, and then writes back to the cluster.

Spark also includes its own graphing library GraphX. GraphX allows users to view the same data as graphs and collections. Users can also use Resilient Distributed Data Sets (RDD), change and union graphs, and fault tolerance sections are discussed.

fault-tolerant

As for fault tolerance, MapReduce and Spark approach the problem from two different directions. MapReduce uses TaskTracker nodes, which provide heartbeats for JobTracker nodes. If there is no heartbeat, the JobTracker node reschall pending and ongoing operations to another TaskTracker node. This approach is effective at providing fault tolerance, but can significantly increase the completion time of certain operations, even if there is only one fault.

Spark uses Resilient Distributed Datasets (RDD), which are fault-tolerant collections of data elements that can perform parallel operations. An RDD can reference a dataset from an external storage system, such as a shared file system, HDFS, HBase, or any data source that provides a Hadoop Input Format. Spark can create an RDD from any storage source supported by Hadoop, including a local file system or one of the file systems listed above.

RDD has five main attributes:

partition list

Compute the function for each slice

List of projects dependent on other RDD

Partitioner for key-oriented RDD (say RDD is hash partition), optional attribute

Computes a list of preferred locations for each shard (e.g. block locations for HDFS files), which is optional

An RDD may be persistent so that the dataset is cached in memory. In this way, the subsequent operation is greatly accelerated, up to 10 times. Spark's cache is fault-tolerant because if any partition of the RDD is lost, it is automatically recalculated using the original transformation.

scalability of

By definition, both MapReduce and Spark can be extended using HDFS. So how big can a Hadoop cluster get?

Yahoo is said to have a 42000-node Hadoop cluster, which can be said to scale without limit. The largest known Spark cluster is 8000 nodes, but as big data grows, expect the cluster size to grow to continue meeting throughput expectations.

security

Hadoop supports Kerberos authentication, which is cumbersome to manage. However, third-party vendors enable Tencent Cloud Organization to leverage Active Directory Kerberos and LDAP for authentication. Those third parties also provide data encryption for data in transit and data at rest.

The Hadoop distributed file system supports access control lists (ACL) and traditional file permission patterns. Hadoop provides Service Level Authorization for user control in task submission, which ensures that customers have the correct permissions.

Spark is a bit less secure and currently only supports authentication via shared keys (password authentication). The security benefit of Spark is that if you run Spark on HDFS, it can use HDFS ACL and file-level permissions. In addition, Spark can run on YARN and thus can use Kerberos authentication.

"Hadoop or Spark which is better" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.