Shulou(Shulou.com)06/01 Report--

Flink time and time zone how to solve the problem, I believe many inexperienced people are helpless about this, this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. epoch

The so-called "time epoch" is January 1, 1970 at 0:00, 0:00, referring to the beginning time. Java class code:

Date date = new Date(0);

System.out.println(date);

Printed results:

Thu Jan 01 08:00:00 CST 1970

It is also January 1, 1970. In fact, the time and minutes are 0:00:00. The printed time here is 8:00 instead of 0:00. The reason is that there is a problem between system time and local time. In fact, the system time is still 0:00. However, our computer time zone is set to East 8, so the printed result is 8:00.

Just set the time zone to GMT+0 to print 0:00:00:00.

System.setProperty("user.timezone","GMT+0");

In fact, the time zone problem is to add/subtract a certain offset based on this time epoch.

2. Flink time

What does java epoch have to do with the flink time problem discussed in this article?

Flink is based on the concept of epoch when it uses the concept of time. For example, first of all, our time zone is East Eight, UTC-0 time in our field of vision should be added with an 8-hour offset, which is the time we see, so when using flink window, it is often 8 hours less than our current time.

There is also a flink window for it, which is also based on epoch time. For example, the following example has three window functions

1).5min Scroll window

The window started at 14:16:391, the scrolling window time is 5 minutes, you will find that it is not waiting for five minutes before the result is output, but at 14:20:00.0, the result is directly output.

2).30min Scroll window

The scrolling window started at 14:27:11 was output directly at 14:30:00 instead of waiting for half an hour.

3).1hour scrolling window

The one-hour scrolling window started at 15:54:48, and the output time was 16:00 sharp.

The time difference is eight hours, but the alignment is an integer unit based on time epochs.

3. Solving the eight-hour problem

Flink's output time difference is annoying when actually used, but there is no way flink currently does not support configuring time zones, but blink supports, waiting for the merger.

In fact, there are many solutions to the time zone problem. If you want to avoid hurting your bones, you will mainly introduce the following three types:

Flink is not processed. That is, 8 hours offset is added to the data read.

Use operators such as udf to add an 8-hour offset to the timestamp.

Sink is processed internally.

1).Udf implementation

sink-side processing

import org.apache.flink.table.functions.ScalarFunction;

import java.sql.Timestamp;import java.text.ParseException;import java.text.SimpleDateFormat;import java.util.Date;import java.util.TimeZone;

public class UTC2Local extends ScalarFunction { public Timestamp eval(Timestamp s) { long timestamp = s.getTime() + 28800000; return new Timestamp(timestamp); }

}

Register udf

tEnv.registerFunction("utc2local",new UTC2Local());

using UDF

Table table1 = tEnv.sqlQuery("select count(number),utc2local(TUMBLE_END(proctime, INTERVAL '1' HOUR)) from res group by TUMBLE(proctime, INTERVAL '1' HOUR)");

2). Sink internal support

The implementation of the sink side is also relatively simple, mainly to determine the output field type, and then add 8 hours offset. You can refer to blink's printtablesink implementation.

override def invoke(in: JTuple2[JBool, Row]): Unit = { val sb = new StringBuilder val row = in.f1 for (i 0) sb.append(",") val f = row.getField(i) if (f.isInstanceOf[Date]) { sb.append(DateTimeFunctions.dateFormat(f.asInstanceOf[JDate].getTime, "yyyy-MM-dd", tz)) } else if (f.isInstanceOf[Time]) { sb.append(DateTimeFunctions.dateFormat(f.asInstanceOf[JDate].getTime, "HH:mm:ss", tz)) } else if (f.isInstanceOf1654075720) { sb.append(DateTimeFunctions.dateFormat(f.asInstanceOf[JDate].getTime, "yyyy-MM-dd HH:mm:ss.SSS", tz)) } else { sb.append(StringUtils.arrayAwareToString(f)) } }

if (in.f0) { System.out.println(prefix + "(+)" + sb.toString()) } else { System.out.println(prefix + "(-)" + sb.toString()) } } After reading the above content, do you know how to solve the time and time zone problem of flink? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.