Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "what are the advantages and disadvantages of web data quality monitoring". The editor shows you the operation process through actual cases, and the operation method is simple, fast and practical. I hope this article "what are the advantages and disadvantages of web data quality monitoring" can help you solve the problem.

I. Overview

1. Current situation

Recently, there are all kinds of problems in the data collected by SaaS platform, APP and other products, such as the title is parsed into JavaScript code, or contains a useless character, or there is a garbled string and so on.

The disadvantages of the previous monitoring mechanism seem to be getting bigger and bigger, and can no longer meet the needs of data monitoring.

With the increasing number of data types, customized collection scripts and personnel involved, and the increasing difficulty of collection, a variety of problems occur frequently.

In order to develop a system that can really monitor data quality in real time, locate problems quickly, and quickly iterate collectors or scripts at the same time, on the basis of the original decentralized monitoring, add another layer of centralized monitoring at the data push interface.

two。 Advantages and disadvantages

Decentralized monitoring means that each collector or script monitors the quality of the data on its own. But sometimes because of the urgent task, or to save trouble, there is no monitoring module added at all.

Centralized monitoring refers to the processing of data quality, weight, etc., at the kafka unified push interface.

1) advantages and disadvantages of decentralized monitoring:

(1) advantages

① can reduce the pressure at the unified push interface and shorten the time for data to enter kafka.

② reduces the frequency of interface exceptions

(2) shortcomings

① related personnel may modify the monitoring indicators, resulting in confusion, unable to achieve the effect of data quality monitoring, and unable to locate the problem.

② may not add monitoring and scheduling mechanism at all because of urgent task or to save trouble, resulting in a large number of data duplication and poor quality data, affecting ES performance and seriously affecting the user experience of the product.

③ wastes resources. Because each collector or custom script needs to consider the monitoring problem, it virtually increases a lot of repetitive work and increases the cost of manpower.

④ product iterations are slow. Laziness is human nature. Without the supervision of processes and mechanisms, most people will deal with problems in the most convenient way. Even think that this is a small problem, it doesn't matter, put it off and forget it.

2) advantages and disadvantages of centralized monitoring:

(1) advantages

① reduces waste of manpower and other resources

② unified standard monitoring mechanism

Lead to ③ abnormal problems to improve the user experience of the product.

④ reduces human risk.

According to the monitoring results, ⑤ urges the relevant personnel to iterate the product continuously through the process and monitoring system.

For managers, ⑥ can understand the problems existing in each part of the collection in real time, think from the overall situation, optimize the collection strategy and so on.

⑦ can provide the basis of performance appraisal for managers to a certain extent according to the monitoring results.

(2) shortcomings

① increases the complexity of unified interface logic processing and increases the probability of exceptions.

② reduces the speed of data processing. Comprehensive consideration, within the acceptable range, or when the demand is met, can not be considered for the time being.

This is the end of the content on "what are the advantages and disadvantages of web data quality monitoring". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.