Shulou(Shulou.com)05/31 Report--

This article will explain in detail the example analysis of big data's safety specification. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Big data safety code

I. Overview

Big data's security system is divided into five levels: perimeter security, data security, access security (authentication-authentication and authorization-authorization), visible access behavior, error handling and exception management. The following is explained in turn:

1. Perimeter security technology is the traditional network security technology, such as firewall and so on.

two。 Data security includes encryption and decryption of data, which can be subdivided into storage encryption and transmission encryption, and desensitization of data.

3. Access security mainly includes two aspects: authentication and authorization of users:

User authentication (Authentication)

That is, to check the identity of the user and confirm that the user is the declared identity, including the authentication of the user and the service.

User authorization (Authorization)

That is, permission control, authorizing or denying access to specific resources and specific access users. User authorization is based on re-user authentication. There is no user authorization without reliable user authentication.

Access security also includes data authentication (data validation)

1 > type. Int string et al.

2 > format. Phone email et al.

3 > length.

4 > range.

5 > precense or absence.

6 > match in lookup tables.

7 > other bussiness rules

4. Visible access behavior refers to recording the user's access behavior to the system (audit and log): such as which file to view; which queries are run; access behavior monitoring, on the one hand, in order to give a real-time alarm and quickly deal with dangerous access behavior; on the other hand, in order to investigate and collect evidence afterwards, analyze and locate specific purposes from the long-term data access behavior.

5. Error handling and exception management

This is mainly aimed at error detection, the general approach is to establish and gradually improve the monitoring system, early warning or warning of what may have happened or has happened. It also includes the monitoring of abnormal attacks, and the methods found to deal with attacks are:

1 > attack chain analysis, analyze according to the time of threat detection, and describe the attack chain

2 > merge statistics for the same type of attacks

3 > abnormal traffic learns normal access traffic and gives an alarm when the traffic is abnormal.

Among these five levels, the third layer (access security) has the most direct relationship with the business: the multi-tenancy and access control of the application directly depend on the technical implementation of this layer, so we will also focus on this layer. It is well known that the authentication (mainly kerberos) provided by hadoop itself is not easy to maintain, and the authorization (mainly ACL) is very coarse-grained, so we decided to use Hortonworks open source Ranger after comparing the open source security services of two heavyweight companies (Cloudera and Hortonworks) (see blog post). Ranger provides many security suites for enterprise hadoop ecological services, providing users / groups with authentication and authorization control of files, folders, databases, tables and columns through centralized rights management, as well as auditing (querying through solr). The newly launched RangerKMS also supports encryption of hdfs data, etc.

II. Access Security of big data platform Security Specification

2.1 user authentication

Authentication is achieved through the user / group synchronization function provided by Ranger. Ranger can integrate Unix or LDAP for user authentication management.

2.2 user Rights Management

2.2.1 account Management

The account is divided into operation and maintenance account and development user account.

The OPS account is divided into multiple accounts according to the service, and different accounts operate different services, as shown below:

Service

User

Flume

Flume

HDFS

Hdfs

MapReduce

Mapred

HBase

Hbase

Hive

Hive

Kafka

Kafka

Oozie

Oozie

Ranger

Ranger

Spark

Spark

Sqoop

Sqoop

Storm

Storm

YARN

Yarn

ZooKeeper

Zookeeper

Ambari Metrics

Ams

To develop user accounts, each user has one account, which is grouped by team. Different accounts or groups operate different files or tables. If you need to manipulate other people's data, you need to be authorized by operation and maintenance.

2.2.2 Directory and file specification

Catalogue

Rules

/ source

The original collected logs are mainly stored. The storage rules are as follows: / source/ {business name} / {date}, where:

Business name: such as sending records, etc.

Date: unified format as yyyyMMdd

/ data

The storage specification is the same as source, the temporary directory of files before the data warehouse

Cleaning time to be determined

/ workspace

Workspace, storage rules are as follows: / workspace/ {team name} / {Business name | Product name}

Each other

/ user

User space, which stores user's private data, and only users can access it. According to the developer

Own custom organization storage files, which are used to store users' test data

Cleaning time to be determined

When the employee's turnover account is cancelled, the space storage is reclaimed.

/ user/hive/warehouse

Store the hive warehouse and create the library according to the team; the public log is created according to the business name

Each team can create a hive library that belongs to the team.

/ temp

Used to store some temporary files

Clean up once a month

2.2.3 user Rights Management

There are two schemes for privilege management, ACL (coarse grained) and ranger (fine grained). Based on our data requirements, we first consider using the fine grained access control provided by ranger.

Use the Ranger UI interface to manage permissions. Currently, the permissions provided by various services are as follows:

Service

Service details

Authority

HDFS

Hdfs path

Read 、 Write 、 Execute

HBase

Table 、 column family 、 column

Read 、 Write 、 Create 、 Admin

Hive

Database, table | function, column

Select 、 Update 、 Create 、 Drop 、 Alter 、 Index 、 Lock 、 All

YARN

Queue

Submit-job 、 Admin-queue

Kafka

Topic

Publish 、 Consume 、 Configure 、 Describe 、 Kafka Admin

Team permission assignment

party

Team member group

Service

Authority

Dp (data platform)

Dp

HDFS

Read 、 Write 、 Execute

HBase

Read 、 Write

Hive

Select

YARN

Submit-job

Kafka

Publish 、 Consume 、 Configure 、 Describe

Dm (data Mining)

Dm

HDFS

Read 、 Write 、 Execute

HBase

Read 、 Write

Hive

Select

YARN

Submit-job

Da (data application)

Da

HDFS

Read 、 Write 、 Execute

HBase

Read 、 Write

Hive

Select

YARN

Submit-job

Op (operation and maintenance)

Hadoop administrator

HDFS 、 HBase 、 Hive 、 YARN 、 Kafka

All

Personal account: online operations should be accurate to personal

Apply for permission process:

The leader of each team shall apply to the administrator and pass the review before granting the corresponding permissions.

This is the end of the example analysis of big data's security specification. I hope the above content can be helpful to everyone and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.