Shulou(Shulou.com)06/02 Report--

This article is about what apache spark refers to. The editor thought it was very practical, so I shared it with you as a reference. Let's follow the editor and have a look.

Apache Spark is an open source cluster computing framework originally developed by AMPLab at the University of California, Berkeley. MapReduce relative to Hadoop will store intermediary data to disk after running work. Spark uses in-memory computing technology, which can analyze the operation in memory before the data is written to the hard disk.

The operation speed of Spark running program in memory is 100 times faster than that of Hadoop MapReduce, and Spark can be 10 times faster even when running program on hard disk. Spark allows users to load data into cluster memory and query it multiple times, which is very suitable for machine learning algorithms.

Using Spark requires a combination of cluster administrators and distributed storage systems. Spark supports cluster management in standalone mode (local Spark cluster), Hadoop YARN, or Apache Mesos.

In terms of distributed storage, Spark can be loaded with HDFS, Cassandra, OpenStack Swift and Amazon S3 interfaces. Spark also supports pseudo-distributed (pseudo-distributed) native mode, but it is usually only used for development or testing to replace distributed storage systems with native file systems. In this case, Spark runs the program using each CPU core on only one machine.

In 2014, more than 465 contributors participated in Spark development, making it the most active project among the Apache Software Foundation and big data's many open source projects.

Thank you for reading! So much for sharing what apache spark refers to. I hope the above content can help you to some extent, so that you can learn more knowledge. If you think the article is good, you can share it and let more people see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.