Shulou(Shulou.com)06/01 Report--

This article will explain in detail what the auxiliary tools for big data data development are, and the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

As a programmer's development tools are like human hands and feet, only by making good use of these development tools can we meet the requirements of a product. Most of them use SQL database to store / retrieve data, and in many cases, it no longer meets our needs. The following editor introduces some commonly used auxiliary tools for big data data development.

Open source enterprise search platform: Solr

Written in Java, from the Apache Lucene project. Solr is an independent enterprise search application server that provides an API interface similar to Web-service. Users can submit a certain format of XML file to the search engine server through http request to generate an index, or they can make a search request through Http Get operation and get the return result in XML format.

Here I still want to recommend the big data Learning Exchange Group I built myself: 529867072, all of them are developed by big data. If you are studying big data, the editor welcomes you to join us. Everyone is a software development party. Irregularly share practical information (only related to big data software development), including the latest big data advanced materials and advanced development tutorials sorted out by myself. Welcome to join us if you want to go deep into big data.

Like ElasticSearch, it is also based on Lucene, but it extends it to provide a richer query language than Lucene, while achieving configurable, extensible and optimized query performance.

Cloud-built distributed RESTful search engine: Elasticsearch

ElasticSearch is a Lucene-based search server. It provides a full-text search engine with distributed multi-user capability, based on RESTful web interface. Elasticsearch is developed in Java and released as open source under the Apache license terms. It is a popular enterprise search engine.

ElasticSearch is not only a full-text search engine, but also a distributed real-time document storage, in which each field is indexed data and can be searched; it is also a distributed search engine with real-time analysis function, and can be extended to hundreds of servers to store and process PB-level data. ElasticSearch uses Lucene to accomplish its indexing function at the bottom, so many of its basic concepts are derived from Lucene.

Open source distributed database management system: Cassandra-

Originally developed by Facebook, it is designed to handle large amounts of data on many commodity servers, providing high availability without a single point of failure.

Open source distributed NoSQL database system: Apache Cassandra

Integrates the data model of Google BigTable and the fully distributed architecture of Amazon Dynamo. It was open source in 2008. Since then, because of its good scalability, Cassandra has been adopted by Web 2.0 sites such as Digg and Twitter, and has become a popular distributed structured data storage scheme.

Because Cassandra is written in Java, it can theoretically run on machines with JDK6 or above, and JDK is officially tested as well as OpenJDK and Sun JDK. Cassandra operation command, similar to the relational database we usually operate, for friends who are familiar with MySQL, the operation will be very easy to use.

Cross-platform, document-oriented database: MongoDB

MongoDB is a database based on distributed file storage, written in C++ language. Designed to provide scalable high-performance data storage solutions for Web applications. Application performance depends on database performance, while MongoDB is the most functional and most like relational database in non-relational databases. With the release of MongDB 3.4, its application scenario adaptability has been further expanded.

The core advantages of MongoDB are flexible document model, highly available replication sets, and scalable shard clusters. You can try to learn about MongoDB from several aspects, such as real-time monitoring MongoDB tools, memory usage and page errors, connections, database operations, replication sets, etc.

Open source (BSD licensed) in-memory data structure storage: Redis is used as a database, cache and message broker.

Redis is an open source log-based, Key-Value database written in ANSI C, network-enabled, memory-based and persistent, and provides API in multiple languages. Redis has three main features that distinguish it from many of its competitors: Redis is a database that stores data entirely in memory and uses disk only for persistence purposes; Redis has a relatively rich range of data types compared to many key data storage systems; and Redis can copy data to any number of slave servers.

Java-based Open Source in-memory data Grid: Hazelcast

Hazelcast is an in-memory data grid in-memory data grid that provides mission-critical transactions and trillions of memory applications for Java programmers. Although Hazelcast does not have the so-called 'Master', it still has a Leader node (the oldest member), which is similar to the concept of Leader in ZooKeeper, but the implementation principle is completely different. At the same time, the data in Hazelcast is distributed, and each member holds part of the data and the corresponding backup data, which is also different from ZooKeeper.

Hazelcast's application convenience is loved by developers, but it needs to be carefully considered if it is to be put into use.

Widely used open source Java distributed cache: EHCache is mainly aimed at general-purpose caching, Java EE, and lightweight containers.

EhCache is a pure Java in-process cache framework, which is fast and lean, and is the default CacheProvider in Hibernate. The main features are: fast and simple, with multiple cache strategies; cached data has two levels, memory and disk, so there is no need to worry about capacity; cached data will be written to disk during virtual machine restart; distributed caching can be carried out through RMI, pluggable API, etc.; listening interface with cache and cache manager; support for multiple cache manager instances and multiple cache areas of one instance Provides a caching implementation of Hibernate.

An open source software framework written in Java for distributed storage and distributed processing of very large data sets: Hadoop

Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the cluster for high-speed operation and storage. Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. The core design of Hadoop's framework is: HDFS and MapReduce. HDFS provides storage for huge amounts of data, while MapReduce provides computing for huge amounts of data.

On big data data development tools which share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.