Shulou(Shulou.com)11/24 Report--

Generally speaking, big data platform includes the following four types of data production domains-production ecological environment (formal production environment), development and testing environment, training and demonstration environment, and disaster preparedness environment. While each production domain is guaranteed by the platform for resources, security, monitoring and fault recovery, different production domains also need to be strictly isolated to ensure the reliability, availability and security of data production. Specific to the real enterprise environment, the planning of production domain is more complex.

This article focuses on the planning and construction of independent production domains, sharing standard processes and points for attention.

Why should we focus on new data production domains?

In the process of development, for business expansion, safety compliance, organizational adjustment and other requirements, it is often necessary to carry out independent production domain planning for the existing big data platform to match the emerging data requirements.

To give a few examples:

1. With the rapid expansion of business, can the data production domain replicate synchronously and quickly?

The rapid replication of business is a common practice of large-scale expansion of enterprises. Correspondingly, big data platform should also provide enough data space for these services to be put into use. For example, after a manufacturing enterprise has a mature practice of building a factory in a certain place, the new local factories also need a new data production domain, and the data operations between the factories do not affect each other.

two。 Security compliance requirements, can independent data isolation and management be ensured?

When carrying out cross-border business, enterprises are required to abide by the relevant laws and regulations on data security in the place where the business is located, and local data are not allowed to be transmitted and exchanged without principles. Therefore, enterprises need to create multiple physically isolated and data-independent production domains to ensure local data business security and compliance. In China, the financial data of listed companies often have compliance requirements for independent data management, which means that finance should have an independent data production domain.

3. Match the organizational structure, can each format not interfere with each other and operate independently?

Large group enterprises with multiple subsidiaries, sub-brands and formats must set up multiple independent data production domains on the big data platform, and at the same time, it is also convenient for each subsidiary to complete independent data cost accounting at the group level.

It must be noted in the above scenario that one data production domain after another represents security, isolation, and stability, but it does not mean that a data island has been re-established.

Taking the group data cloud service of "both isolated and unified" as an example, the Group big data platform unifies to provide accounting resources, operation and maintenance services and security for each subsidiary and sub-brand (independent data production domain). And retain the ability to analyze and insight into the group-wide data assets. The more technologically mature big data platform should also support "reuse" under the premise of compliance, such as replicating the data business logic of the standard space to the new space, in order to keep up with the pace of rapid business expansion and avoid rebuilding from scratch again and again.

5 steps to complete the planning of the new data production domain

DataSimba, the singularity cloud platform, has the capability of cross-cloud, multi-domain and multi-tenancy. Relying on DataSimba, enterprises can uniformly build and manage global data assets, or create multiple Workspace (workspaces, that is, independent data production domains) to meet multi-cloud, multi-brand, multi-format and other management requirements.

The authority management and control system of DataSimba is orderly, flexible and refined. Tenants can establish their own Project (project) and divide permissions under one Workspace, or they can establish different projects in different Workspace. A project can be set up by multiple tenants in a Workspace, or it can be monopolized by one tenant.

In the data cloud platform DataSimba, planning and creating a new Workspace can be summarized into the following five steps:

1. Big data cluster evaluation

1.1 overall research: research on the current situation of enterprise business and data

Confirm the overall business objectives and business scope; explore the current situation of data, clear data distribution and data flow; IT system research, research the status of enterprise IT infrastructure; organizational structure research, understand the overall organizational structure of the enterprise.

1.2 Resource assessment: assess the total amount of resources needed

Through the survey data, estimate the overall amount of data in the next few years; based on the future development of the overall business, estimate the overall number of tasks.

1.3 component evaluation: select components based on the business scenario of the survey to meet the needs of the business.

2. Account planning

Account (account) refers to the tenant account, which will be bound with several User (user sub-accounts). Each sub-account can be assigned a different Role (role), and each role can control the permissions of the functions that can be accessed. At the same time, each sub-account can bind data permissions.

Specific steps include: identifying users who need to establish sub-accounts based on organizational structure research; planning different roles and required functional permissions based on users' positions and responsibilities; planning data permissions setting according to enterprise data security requirements; and finally, binding Account to resource nodes.

3. Project planning

Project (project) is a logical management unit for tasks, jobs, and data.

First of all, according to the business situation of the enterprise, select the appropriate division dimensions for Project planning. After the division is completed, the users related to the above Account are assigned to the corresponding Project.

Common dimensions of planning Project are as follows:

Environmental use dimensions: such as development environment, test environment, pre-release environment, formal environment, etc. From an economic point of view, it is usually divided into development environment and formal environment.

Business domain dimensions: such as order domain, financial domain, etc.

Organizational structure dimension: such as production department, market operation department, e-commerce department and so on.

Geographical location dimension: divided according to the geographical location of the business, such as Europe, North America, etc.

4. Quota planning

Quota (quota) refers to the allocation and restriction of resources (such as CPU, memory, GPU, etc.) used by different users or departments.

With the resources and projects planned above, you can start Quota planning, usually following the following principles:

Business priority principle: important business spaces have higher quotas to ensure that tasks are effectively completed.

Resource utilization principle: Quota sharing can be used as much as possible to improve resource utilization without affecting business usage.

Combined with the above principles and business scenarios, judge the high-priority business. On the premise of ensuring the effective execution of high-priority services, configure Quota. Then judging the medium priority service, you can choose to share a Quota with the low priority service according to the actual business requirements. After the Quota is planned, it is assigned to the corresponding Project.

5. Task and data migration

After the above links are planned, the related tasks and data will be formally migrated.

DataSimba has a built-in migration client, which supports automatic migration of data sources, jobs, tasks, services and other objects of the existing big data system to DataSimba.

In addition, DataSimba provides complete tools to completely copy the data business logic of the existing Workspace to the new Workspace to meet the needs of rapidly creating and replicating independent production domains.

At the end: to create a data production domain is to create an example of an object system.

The underlying layer of the data cloud platform DataSimba is the data cloud operating system kernel (SimbaOS Kernel). The kernel abstracts the common functions of big data, such as storage, computing, service, scheduling, security, tenant and so on, into a set of standard object modules. This set of standard objects, coupled with the relationships between objects, can meet the needs of almost all business scenarios.

As shown in the figure above, Workspace (workspace), Account (account), Project (project) and User (sub-account) are all "objects". Create a data production domain in DataSimba, that is, an instance of an object system:

1. Create an instance of Account and associate with Workspace

two。 Select and create a Project instance

3. Create the corresponding Quota instance

4. Finally, data migration is carried out, and instances such as Task (task), Job (job) are created.

It is more advantageous to build a data production domain by creating an object system (instance of):

Encapsulate the underlying technology to improve ease of use: take Project (project) as an example, data cloud platform users (engineers) only need to create a project through this object and complete various modifications to the project without paying attention to the underlying technical details

Improve the maintainability of the system: the adjustment of a single object has little impact on the whole. For example, the object Workspace (Workspace) needs to support new features, and only changes to this object are needed, without affecting other objects and relationships.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.