Shulou(Shulou.com)06/03 Report--

Big data is a broad term that refers to data sets that are so large and complex that they need specially designed hardware and software tools to deal with them. The dataset is usually the size of a trillion or EB. These data sets are collected from a variety of sources: sensors, climate information, public information, such as magazines, newspapers, articles. Other examples generated by big data include purchase transaction records, web logs, medical records, military surveillance, video and image files, and large e-commerce.

In the analysis of big data and big data, they have an upsurge of interest in the impact of the enterprise. Big data analysis is to look for patterns, relevance and other useful information in the process of studying a large amount of data, which can help enterprises better adapt to changes and make more informed decisions.

1. Hadoop

Hadoop is a software framework capable of distributed processing of a large amount of data. But Hadoop is handled in a reliable, efficient, and scalable way. Hadoop is reliable because it assumes that computing elements and storage will fail, so it maintains multiple copies of working data to ensure that processing can be redistributed against failed nodes. Hadoop is efficient because it works in parallel and speeds up processing through parallel processing. Hadoop is also scalable and can handle PB-level data. In addition, Hadoop relies on community servers, so it is cheap and can be used by anyone.

Hadoop is a distributed computing platform that makes it easy for users to structure and use. Users can easily develop and run applications that deal with huge amounts of data on Hadoop. It mainly has the following advantages:

⒈ has high reliability. The ability of Hadoop to store and process data bit by bit is trustworthy.

⒉ is highly scalable. Hadoop distributes data and completes computing tasks among available computer clusters, which can be easily extended to thousands of nodes.

High efficiency of ⒊. Hadoop can move data dynamically between nodes and ensure the dynamic balance of each node, so the processing speed is very fast.

⒋ has high fault tolerance. Hadoop can automatically save multiple copies of data and automatically reassign failed tasks.

Hadoop comes with a framework written in the Java language, so it is ideal to run on the Linux production platform. Applications on Hadoop can also be written in other languages, such as C++.

II. HPCC

Abbreviation for HPCC,High Performance Computing and Communications (High performance Computing and Communications). In 1993, the Federal Coordinating Council for Science, Engineering and Technology submitted a report on "Major Challenge projects: high performance Computing and Communications" to Congress, that is, the report known as the HPCC Program, that is, the President's Science Strategy Project, which aims to solve a number of important scientific and technological challenges through strengthening research and development. HPCC is a program implemented in the United States to implement the information superhighway, which will cost 10 billion US dollars. Its main goals are to develop scalable computing systems and related software to support terabit network transmission performance, develop gigabit network technology, and expand the capacity of research and educational institutions and network connections.

The project is mainly composed of five parts:

1. High performance computer system (HPCS), including the research of future generations of computer systems, system design tools, advanced typical systems and the evaluation of original systems, etc.

2. Advanced Software Technology and algorithms (ASTA), including software support for great challenges, new algorithm design, software branches and tools, computing and high performance computing research centers, etc.

3. National Scientific Research and Education Grid (NREN), which includes the research and development of pick-up stations and 1 billion-bit transmission.

Basic Research and Human Resources (BRHR), which includes basic research, training, education, and curriculum materials, is designed to be initiated by rewarding investigators, long-term surveys to increase the innovative stream of consciousness in scalable high-performance computing, and to increase the pool of skilled and trained personnel through improved education and high-performance computing training and communications. And to provide the necessary infrastructure to support these research and research activities

5. Information Infrastructure Technology and Application (IITA), which aims to ensure the leading position of the United States in the development of advanced information technology.

III. Storm

Storm is free open source software, a distributed, fault-tolerant real-time computing system. Storm can reliably handle large data streams, which can be used to deal with batch data of Hadoop. Storm is simple, supports many programming languages, and is fun to use. Storm is open source from Twitter, and other well-known application companies include Groupon, Taobao, Alipay, Alibaba, Le element, Admaster and so on.

Storm has many applications: real-time analysis, online machine learning, non-stop computing, distributed RPC (remote procedure call protocol, a way to request services from remote computer programs over the network), ETL (abbreviation for Extraction-Transformation-Loading, that is, data extraction, transformation and loading), and so on. The processing speed of Storm is amazing: each node can process 1 million data tuples per second. Storm is extensible, fault-tolerant, and easy to set up and operate.

IV. Apache Drill

In order to help business users find more effective and faster ways to query Hadoop data, the Apache Software Foundation recently launched an open source project called Drill. Apache Drill implements Google's Dremel.

According to Tomer Shiran, product manager of Hadoop manufacturer MapR Technologies, "Drill" has been operated as an Apache incubator project and will be continuously promoted to software engineers around the world.

The project will create an open source version of Google's Dremel Hadoop tool (which Google uses to speed up the Internet application of the Hadoop data analysis tool). And "Drill" will help Hadoop users to query massive data sets faster.

The "Drill" project is also inspired by Google's Dremel project: it helps Google analyze and process massive data sets, including analyzing and crawling Web documents, tracking application data installed on Android Market, analyzing spam, analyzing test results on Google's distributed build system, and so on.

By developing the "Drill" Apache open source project, organizations are expected to establish the API interface to which Drill belongs and a flexible and powerful architecture to help support a wide range of data sources, data formats, and query languages.

5. RapidMiner

RapidMiner is the world's leading data mining solution, with advanced technology to a great extent. It covers a wide range of data mining tasks, including a variety of data arts, which can simplify the design and evaluation of the data mining process.

Functions and features

Provide data mining technology and library free of charge

100% with Java code (can run on the operating system)

The process of data mining is simple, powerful and intuitive

Internal XML ensures a standardized format to represent the exchange data mining process

Large-scale processes can be automated with a simple scripting language

A multi-level view of data to ensure valid and transparent data

Interactive prototype of graphical user interface

Automatic large-scale application of command line (batch mode)

Java API (Application programming Interface)

Simple plug-ins and promotion mechanisms

Powerful visualization engine, visual modeling of many cutting-edge high-dimensional data

Supported by more than 400 data mining operators

Yale University has been successfully applied in many different application fields, including text mining, multimedia mining, functional design, data flow mining, integrated development methods and distributed data mining.

VI. Pentaho BI

Pentaho BI platform is different from traditional BI products, it is a process-centric, solution-oriented (Solution) framework. Its purpose is to integrate a series of enterprise BI products, open source software, API and other components to facilitate the development of business intelligence applications. With the emergence of it, a series of independent products for business intelligence, such as Jfree, Quartz and so on, can be integrated to form a complex and complete business intelligence solution.

The Pentaho BI platform, the core architecture and foundation of the Pentaho Open BI suite, is process-centric because its central controller is a workflow engine. The workflow engine uses process definitions to define business intelligence processes that are executed on the BI platform. Processes can be easily customized or new processes can be added. The BI platform contains components and reports to analyze the performance of these processes. At present, the main elements of Pentaho include report generation, analysis, data mining, workflow management and so on. These components are integrated into the Pentaho platform through J2EE, WebService, SOAP, HTTP, Java, JavaScript, Portals and other technologies. The release of Pentaho is mainly in the form of Pentaho SDK.

Pentaho SDK consists of five parts: Pentaho platform, Pentaho sample database, Pentaho platform that can run independently, Pentaho solution example and a preconfigured Pentaho network server. Among them, Pentaho platform is the most important part of Pentaho platform, including the main body of Pentaho platform source code. Pentaho database provides data services for the normal operation of Pentaho platform, including configuration information, Solution-related information, etc., which is not necessary for Pentaho platform, and can be replaced by other database services through configuration. The stand-alone Pentaho platform is an example of the stand-alone operation mode of the Pentaho platform, which demonstrates how to make the Pentaho platform run independently without application server support; the Pentaho solution example is an Eclipse project that demonstrates how to develop relevant business intelligence solutions for the Pentaho platform.

The Pentaho BI platform is built on servers, engines and components. These provide the system's J2EE server, security, portal, workflow, rule engine, chart, collaboration, content management, data integration, analysis and modeling functions. Most of these components are standards-based and can be replaced with other products.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.