Shulou(Shulou.com)06/02 Report--

How to use set in flink to calculate the website uv of the day in real time. For this question, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Background

For web sites, we generally have such a need to calculate the uv of the site in real time and display it as soon as possible. Today we will talk about the set collection based on java to do real-time uv statistics.

Simple requirements:

Real-time calculation of the zero point of the day to the current time (android,ios,h6) under each end (uv) update statistical results every second case to explain the simulation of source

First of all, let's simulate and generate the simplest data, generating a flink binary Tuple2. Denotes classification and user id respectively

Public static class MySource implements SourceFunction {

Private volatile boolean isRunning = true

String category [] = {"Android", "IOS", "H5"}

@ Override

Public void run (SourceContext ctx) throws Exception {

While (isRunning) {

Thread.sleep (10)

/ / which end is the user?

String type = category [(int) (Math.random () * (category.length))]

/ / randomly generate int type data less than 10000 as userid

Int userid = (int) (Math.random () * 10000)

Ctx.collect (Tuple2.of (type, userid))

}

}

@ Override

Public void cancel () {

IsRunning = false

}

}

Define window

Next we define a sliding window with a period of one day, because we want to output the data of the window every second, so we define a 1-second trigger immediately after the window.

DataStream dataStream = env.addSource (new MySource ())

DataStream.keyBy (0) .window (TumblingProcessingTimeWindows.of (Time.days (1), Time.hours (- 8)

.trigger (ContinuousProcessingTimeTrigger.of (Time.seconds (1)

New MyAggregate (), new WindowResult ()

.print ()

Custom aggregation operator

Next, let's customize an aggregation operator to implement this function.

For an understanding of the aggregation operator, please refer to this article:

Https://mp.weixin.qq.com/s/ZCWexNGzhSchRpxipa1x-g

Public static class MyAggregate

Implements AggregateFunction {

@ Override

Public Set createAccumulator () {

Return new HashSet ()

}

@ Override

Public Set add (Tuple2 value, Set accumulator) {

Accumulator.add (value.f1)

Return accumulator

}

@ Override

Public Integer getResult (Set accumulator) {

Return accumulator.size ()

}

@ Override

Public Set merge (Set a, Set b) {

A.addAll (b)

Return a

}

}

Processing output result

Here we output the results to the console, and in actual production we can write the data to redis or hbase and so on.

1 > Result {, dateTime='2020-06-21 1915 19V 2315 30mm typewritten IOSs, uv=136}

2 > Result {, dateTime='2020-06-21, 1915, 1915, 2315, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915, 1915,

1 > Result {, dateTime='2020-06-21 1915 19V 2315 30mm typewriter H5mm, uv=134}

1 > Result {, dateTime='2020-06-21 1915 19V 23V 31mm typewritten IOSs, uv=164}

2 > Result {, dateTime='2020-06-21 1915 1915 23lV 31mm typewritten Android, uv=177}

1 > Result {, dateTime='2020-06-21 1915 19V 23V 31mm typewriter H5mm, uv=167}

2 > Result {, dateTime='2020-06-21, 1915, 1915, 2313, type 32, uv=205}

1 > Result {, dateTime='2020-06-21 1915 19V 2315 32mm typewritten IOSs, uv=193}

1 > Result {, dateTime='2020-06-21 1915 19V 2315 32mm typewriter H5mm, uv=198}

This is the answer to the question about how to use set in flink to calculate the uv of the website in real time. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.