Shulou(Shulou.com)06/03 Report--

Background

1. "Yunda Kaiwu", the four hot information technologies

1.1 the development of business is increasingly affected by technological progress. Business innovation is inseparable from technological innovation. Technology serves the business?

two。 Panoramic view of big data's technology stack:

Distributed programming

Distributed file system

Column databases (HBase, Cassandra, BigTable)

Column database (Greenplum, BigQuery)

Key-value database (Redis, Amazon DynamoDB, Bolt)

Document database (MongoDB, RethinkDB)

Relation database

New SQL Database (HANA)

Time series database

SQL engine (Hive, PrestoDB, SparkSQL)

Data extraction

Service programming

Dispatching

Machine learning

Benchmark and Security

System deployment

Application program

Search engine and Framework

MySQL 、 PostgreSQL 、 Memcached

Embedded database

Business intelligence

Data visualization

Internet of things data

Stream computing engine

Data pipeline

Big Data

Public datasets-(data openness, data cloud services)

Hadoop-big data distributed data Storage and processing Framework

Data Engineering

Streaming

Design philosophy

Embrace open source and open platform

Encourage autonomy and data ecology

Support innovation and development ecology

Data sharing and opening / data Portal / data Ecology

Build using ckan.

Big data crowdsourcing and big data operation

Data ecology

Data governance vs data autonomy (search engine is a typical data autonomy; autonomy-> ecology)

Machine learning

Machine learning method is a method that computers use existing data to obtain (train) a certain model, and use this model to predict the future. Machine learning can perform functions that cannot be accomplished by direct programming (traditional programming).

Methods of machine learning:

1. Regression algorithm

two。 Neural network

3.SVM (support vector machine)

4. Clustering algorithm

5. Dimensionality reduction algorithm

6. Recommendation algorithm

Supervised learning algorithms: linear regression, logical regression, neural network, SVM

Unsupervised learning algorithm: clustering algorithm, dimensionality reduction algorithm

Special algorithm: recommendation algorithm

Before 2010, the application of machine learning played a great role in some specific fields, such as license plate recognition, network prevention, handwritten character recognition and so on. However, since 2010, with the rise of the concept of big data, a large number of applications of machine learning are highly coupled with big data. Big data can almost be regarded as the best scene for the application of machine learning.

1. Big data, small analysis: the idea of OLAP analysis in the field of data warehouse, that is, the idea of multi-dimensional analysis.

two。 Big data, big analysis: this represents data mining and machine learning analysis.

3. Flow analysis: this mainly refers to the event-driven architecture.

4. Query analysis: the classic representative is NoSQL database.

A subcategory of machine learning-deep learning

Deep learning is the development of traditional neural networks to multiple hidden layers. When the neural network is expanded to more than two hidden layers, its training speed will be very slow.

Artificial intelligence:

Big data platform and IaaS/PaaS

IaaS:OpenStack Keystone (Authentication Service), OpenStack Swift (object Storage)

The cooperation of big data platform and IaaS layer can realize automatic deployment of big data platform, increase or decrease of nodes, multi-tenant isolation and so on.

PaaS: based on Docker technology

Hosting and auto scaling of big data front-end (front-end) applications

Hadoop as a Service

Cloudbreak

Oriented user

Data provider

Data analyst

Developer

Operation and maintenance engineer

Safety

Keystone, ldap, oauth and social accounts, basic authentication

Integration issues, such as ckan and owncloud, have their own users

Api store/data store/app store

Development ecology

Hosting of big data's application

API Store: reflects the reuse of technology, reduces the learning threshold, and facilitates debugging.

Does crawler service count as API?

Deployment

Use common deployment tools

Docker deployment

Vagrant

Cloudbreak

Apache Ambari

HDP deployment

Using the ambari deployment tool, it is best to use the official source + homemade source mode. Sinicize the official source and add self-made services.

Material

Chrome adds face recognition, printed recognition OCR, bar code recognition, one line of code to achieve the above recognition. IDAP also has face recognition, OCR, bar code recognition and other industry cases, packaged as API services?

Big data API service

Face recognition

License plate recognition

× × × recognition

OCR

QR code recognition

Integration

Unification of users (tenants)

Docking of data sets (pipes)

Internationalization

Frontend-backend (REST API)

Cloud computing model

There are differences in operation models, such as resource management, charging, and tenant management.

Public cloud billing, virtual data center

Sharing Yunda application + approval + accounting after the event

Exclusive Cloud (Private Cloud) small Application + approval + Statistics

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.