Shulou(Shulou.com)06/02 Report--

Flink development of how to batch applications, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

Demand

Word frequency statistics, that is, give a file, count the number of times each word appears in the file, the delimiter is\ t. The contents of this document are as follows:

Hello world welcomehello welcome

The statistical results are printed directly on the console. Generally Sink to the destination in a production environment.

Using Flink + java to realize the requirement environment

JDK:1.8

Maven:3.6.1 (minimum Maven 3.0.4)

Create a project mvn archetype:generate-DarchetypeGroupId=org.apache.flink-DarchetypeArtifactId=flink-quickstart-java-DarchetypeVersion=1.8.1-DarchetypeCatalog=local

GroupId: com.vincent artifactId: springboot-flink-train version:1.0 thus creates a project and imports it using Idea. The project structure is as follows:

There are two java classes that are automatically prepared for us.

Development steps

Step 1: create a batch context

/ / set up the batch execution environmentfinal ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment ()

Step 2: read the data

Env.readTextFile (textPath)

Step 3: transform operations, such as filter () flatMap () join () coGroup (), which is the core of development, which is generally business logic.

Step 4: execute program

Concrete operation

Step 1: read the data

Hello welcome

Step 2: split the data of each row according to the specified delimiter

Hellowelcome

Step 3: assign each word a number of times

(hello,1) (welcome,1)

Step 4: merge operation

Code implementation / * use Java API to develop batch applications for Flink * / public class BatchWCJavaApp {public static void main (String [] args) throws Exception {String input = "E:/test/input/test.txt"; / / step1: get the runtime environment ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment (); / / step2: read data DataSource text = env.readTextFile (input) / / step3: transform / / FlatMapFunction 0) {out.collect (new Tuple2 (token, 1));}) .groupBy (0) .sum (1) .print ();}} run result (world,1) (hello,2) (welcome,2) use Flink + scala to implement the demand environment

JDK:1.8

Maven:3.6.1 (minimum Maven 3.0.4)

Create a project in the same way as using java: mvn archetype:generate-DarchetypeGroupId=org.apache.flink-DarchetypeArtifactId=flink-quickstart-scala-DarchetypeVersion=1.8.1-DarchetypeCatalog=local

GroupId: com.vincent artifactId: springboot-flink-train-scala version:1.0 thus creates a project and imports it using Idea:

The next development steps are the same as those implemented using java: here is the

Code implementation import org.apache.flink.api.scala.ExecutionEnvironment/** * use Scala to develop batch applications for Flink * / object BatchWCScalaApp {def main (args: Array [String]): Unit = {val input = "E:/test/input/test.txt" val env = ExecutionEnvironment.getExecutionEnvironment val text = env.readTextFile (input) / / introduce implicit conversion import org.apache.flink.api.scala._ text .flatMap (_ .toLowerCase.split ("\ t")) .filter (_ .nonEmpty) .map ((_ 1)) .groupBy (0) .sum (1) .print ()}} Java and Scala implementation comparison of operators and simplicity

That is, although the principle of the transform part is the same, it is implemented in a different way, and scala is more concise.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.