Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the interview questions for Flink". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what are the interview questions for Flink"?

1. How to ensure accurate one-time consumption by Flink

Flink ensures accurate one-time consumption mainly depends on two Flink mechanisms.

1. Checkpoint mechanism

2. Two-phase submission mechanism

Checkpoint mechanism

The main reason is that when Flink opens Checkpoint, a barrir is inserted into the source end, and then the barrir flows along with the data flow. When it flows into an operator, the operator begins to make checkpoint, which creates the state of the current operator before the arrival of barrir, and writes the state to the back end of the status. Then flow down the barrir, when flowing to the keyby or shuffle operator, for example, when the data of an operator depends on multiple streams, there will be barrir alignment, that is, when all the barrir come to this operator to make checkpoint, flow in turn, when the flow to the sink operator, and the completion of the sink operator checkpoint will report to jobmanager the completion of the checkpoint n production.

Two-phase submission mechanism

Flink provides two interfaces: CheckpointedFunction and CheckpointListener. There is a snapshotState method in CheckpointedFunction. Each time checkpoint triggers the execution method, the cached data is usually put into a state, which can be understood as a hook. Pre-submission can be realized in this method, notifyCheckpointComplete method in CheckpointListyener, and notification method after checkpoint is completed. Some additional operations can be done here. For example, FLinkKafkaConumerBase uses this to complete the submission of Kafka offset, which can be implemented in this method. It is mentioned in 2PC that if the corresponding process, such as a checkpoint, fails, then the checkpoint will be rolled back without affecting data consistency, so if it fails after notifying checkpoint that it is successful, then the transaction will be committed in the initalizeSate method, which ensures data consistency. The most important judgment is based on the status file of checkpoint.

2. The difference between flink and spark

Flink is an "open source technology stack" similar to spark, because it also provides batch processing, streaming computing, graph computing, interactive query, machine learning, and so on. Flink is also memory computing, similar to spark, but the difference is that spark's computing model is based on RDD, streaming computing is regarded as a special batch, and its DStream is actually RDD. Flink treats batch processing as a special streaming computing, but the layers of batch and streaming computing have two engines, abstracting DataSet and DataStream. Flink also performs well in performance, and the delay of streaming computing is less than that of spark, so it can achieve real streaming computing, while spark can only be quasi-streaming computing. And in batch processing, when there are more iterations, flink is faster than spark, so if flink came out earlier, it might be more popular than the current Spark.

3. What can the status of Flink be used for?

There are two main ways to use Flink state:

Data recovery of checkpoint

Logical calculation

4. WaterMark mechanism of Flink, Flink watermark transmission mechanism.

The watermark mechanism in Flink is used to deal with disorder, and the time of flink must be event time. A simple example is that if the window is 5 seconds and the watermark is 2 seconds, then the total is 7 seconds. When will the calculation be triggered? assuming that the initial time of the data is 1000, then the calculation of 5999 windows will be triggered when 6999, and the next window will trigger 10999 windows when 13999.

In fact, this is the mechanism of watermark. In multiple parallelism, for example, when all partitions are reached in kafka, the window will be triggered.

5. The temporal semantics of Flink

The time when the Event Time event was generated

The time when the Ingestion time event entered the Flink

The time when the processing time event entered the operator

6 、 Flink window join

1. Window join, that is, inner join according to the specified fields and scrolling sliding windows and session windows

2. CoGoup is actually left join and right join

3. Interval join, that is, there are some problems with join in the window, because some data really arrive after the meeting, and it is still a long time, so there is an interval join at this time, but it must be the event time, and you have to specify watermark and water level as well as obtain the event timestamp. And set the offset interval, because join can't wait forever.

7. What are the flink window functions?

Tumbing window

Silding window

Session window

Count winodw

8. How keyedProcessFunction works. If it's event time.

KeyedProcessFunction has an ontime operation. If it is event time, then the time to call is to check whether the watermark of event is greater than the time of trigger time. If it is greater, it is calculated. If it is kafka, then the minimum time of the partition key is used to trigger by default.

9. How does flink handle offline data, such as association with offline data?

1 、 async io

2 、 broadcast

3. Async io + cache

4. Read in the open method, and then refresh the thread regularly. The cache update is deleted first, and then another one is responsible for writing to the cache.

10. Data types supported by flink

DataSet Api and DataStream Api, Table Api

11. What if the data tilt occurs in Flink

How to view Flink data skew:

The data skew can be seen in the web ui of flink, that is, the amount of data processed by each subtask varies greatly, for example, some are only one M and some are 100m, which is a serious data skew.

Data skew on the KafkaSource side

For example, if there is a data hotspot problem with the specified key when the upstream kafka is sent, do a load balancer after the connection (provided that the downstream is not keyby).

Aggregation operator data skew

Prepolymerization plus global polymerization

12. How to do flink dimension table association

1 、 async io

2 、 broadcast

3. Async io + cache

4. Read in the open method, and then refresh the thread regularly. The cache update is deleted first, and then another one is responsible for writing to the cache.

13. How to solve the timeout problem of Flink checkpoint.

1. Whether there is a network problem

2. Whether it is a barrir problem

3. Check webui to see if there is any data skew.

4. If there is data tilt, then after solving the data tilt, there will be improvement.

14. The difference between flinkTopN and offline TopN

Topn is a common function in both offline and real-time computing. Unlike topn in offline computing, real-time data is continuous, which brings great difficulties to the calculation of topn, because it is necessary to maintain a topn data structure in memory and update this data structure when new data comes.

15. The difference between checkpoint in sparkstreaming and flink

Checkpoint of sparkstreaming will lead to repeated consumption of data

But flink's checkpoint can guarantee accurate one-time, incremental, fast checkpoint, there are three state backends, memery, rocksdb, hdfs

16. A brief introduction to cep state programming

Complex Event Processing (CEP):

FLink Cep is a complex time processing library implemented in FLink. CEP allows the detection of event patterns in endless time flows, giving us the opportunity to master an important part of the data. One or more time streams composed of simple events match through certain rules, and then output the data that users want, that is, complex events that meet the rules.

17. What are the options for Flink cep continuous events 18, how to use flink's CEP to realize payment delay reminder 19, which business scenarios have you used in Flink cep 20, how does the bottom layer of cep work 21, how does cep age 22, cep performance tuning 23, the back pressure of Flink, introduce the backpressure of Flink, how do you monitor and find it.

Flink does not use any complex mechanism to solve the reverse pressure problem, and Flink uses distributed blocking queues during data transmission. We know that in a blocking queue, the sender will be naturally blocked when the queue is full, and this blocking function is equivalent to providing the blocking queue with the ability to reverse pressure.

When there is a backpressure on your task, if your upstream is a Kafka-like messaging system, it is obvious that consumption slows down and Kafka messages pile up.

If your business doesn't have a high demand for data latency, then reverse pressure doesn't really have a big impact. However, for large operations in large clusters, reverse pressure can cause serious "complications". First of all, the task state will become very large, because the data is piled up in the system on a large scale, and the data that is not processed temporarily will also be put into the "state". In addition, Flink will lead to checkpoint timeout due to data accumulation and slow processing speed, and checkpoint is the key to ensure data consistency in Flink, which will eventually lead to data inconsistency.

Flink Web UI

The background page of Flink is the first choice for us to find reverse pressure problems. The background page of Flink can see the running status of the current job intuitively and clearly.

Web UI, it should be noted that the calculation of the backpressure state is triggered only when the user visits and clicks on a job. In the default setting, the TaskManager of Flink triggers backpressure monitoring every 50ms for a total of 100 times, and the calculation results are fed back to JobManager. Finally, JobManager calculates the backpressure ratio and then displays it.

In a production environment, there are three types of Flink tasks: OK, LOW, and HIGH.

OK is normal.

LOW in general

HIGH high load

24. CBO of Flink, logical execution plan and physical execution plan

The optimized execution of Flink is actually an execution plan generated by the optimizer of the database.

CBO, the cost optimizer, the least expensive execution plan is the best execution plan. In traditional databases, the cost optimizer makes the optimal execution plan based on statistical information. The same is true of Flink's cost optimizer. Flink optimizes the execution logic and physical execution plan for each query before providing final execution. These optimization tasks are left to the bottom to complete. Further optimizations are performed based on the query cost, resulting in potentially different decisions: how to sort joins, what type of joins to perform, parallelism, and so on.

/ / TODO

25. Data aggregation in Flink, how to achieve aggregation without using windows

ValueState is used to save a single value

ListState is used to save list elements

MapState is used to save a set of key-value pairs

ReducingState provides the same method as ListState, returning an aggregated value of ReducingFunction.

AggregatingState is similar to ReducingState in that it returns an aggregated value within AggregatingState.

26. What are the storage methods of state in Flink

Memery 、 RocksDB 、 HDFS

27. How to deal with Flink abnormal data

In our scenario, abnormal data is generally divided into missing fields and outlier data.

Outliers: for example, data on the age of a baby, for example, for the maternal and infant industry, the age of a baby is a vital data, which can be said to be the most important, because babies who are older than 3 years old almost do not buy goods from mothers and babies. Like ours, we have the same day, the unknown, and a long time. This is an exception field, and we will show the data to the store manager and the regional manager to let them know how many ages are not allowed. If you want to deal with it, you can make real-time correction according to the time he bought, such as maternity clothing, the level of milk powder, the size of diapers, and pacifiers, ah, those that can distinguish between age groups. We don't process the data in real time, we have a low-level strategic task, Night Uygur, to run once a week.

Missing fields: for example, some fields are really missing and can be repaired as soon as possible. Give up if you can't fix it, just like the news recommendation filter in the last company.

28. Flink monitors how you do

1. We monitored whether the task of Flink stopped.

2. We monitored the LAG of Flink's Kafka.

3. We will carry out real-time data reconciliation, such as sales.

Is there any possibility of data loss in Flink

Flink has three kinds of data consumption semantics:

At Most Once consumption may be lost if it fails at most once.

At Least Once may repeat at least once failure.

If a fault occurs at a precise time, Exactly-Once can also guarantee that the data will not be lost or repeated.

The new version of flink no longer provides At-Most-Once semantics.

30. Flink interval join, can you simply write DataStream keyed1 = ds1.keyBy (o-> o.getString ("key")) DataStream keyed2 = ds2.keyBy (o-> o.getString ("key")) / / time stamp-5s on the right

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.