Shulou(Shulou.com)06/01 Report--

This article shows you. Net for Spark how to achieve WordCount application and debug pit analysis, the content is concise and easy to understand, absolutely can make your eyes bright, through the detailed introduction of this article, I hope you can get something.

.net for Spark to realize WordCount application and debug into the pit

1. Overview

INeuOS cloud operating system now has the functions of Internet of things, view business modeling and machine learning, but it lacks a computing platform product. Recently, I have been investigating what language to use for development, and studying the technical route of implementation. INeuOS is developed using C#/JS/CSS/PYTHON, so C # is preferred to implement the development of computing platform, and of course, languages such as scala and python are not excluded. Microsoft recently released .net for Spark components that are compatible with .NET Standard 2.0 and can be used on Linux, macOS, and Windows systems, just like the rest of .NET. A survey was conducted online, and the following figure shows the performance comparison of .NET Core, Python and Scala on TPC-H query sets. As shown below:

Read the source code on github, the code quality is relatively high. The application and testing have been carried out in the past two days, and the overall process has been worked out and feels relatively refreshing.

two。 System environment

Operating system: Ubuntu 16.04

Development language: .Net Core 2.2

Computing framework: .Net for Spark

Development tool: VSCODE

Testing tool: netcat

3. Environment deployment 3.1 install .net Core 2.2

See installation procedure: https://dotnet.microsoft.com/download/linux-package-manager/ubuntu16-04/sdk-current

3.2 install .net for Spark and code implementation

See installation procedure: https://github.com/dotnet/spark/blob/master/docs/getting-started/ubuntu-instructions.md

For the code of Program.cs implementation, please see: https://github.com/dotnet/spark/blob/master/examples/Microsoft.Spark.CSharp.Examples/Sql/Streaming/StructuredNetworkWordCount.cs

3. 3 install netcat

one

Command line: apt-get install netcat

4. Development and debugging 4.1 start netcat

It is mainly used to send strings in real time to realize string statistics.

one

Command line: nc-lk 9999

4.2 start debug mode

Under the assembly directory: HelloSpark/bin/Debug/netcoreapp2.2.

one

Command line: spark-submit-class org.apache.spark.deploy.DotnetRunner-master local microsoft-spark-2.4.x-0.3.0.jar debug

If you do not execute this command, an exception will occur during vscode debugging: [Exception] [JvmBridge] Connection refused 127.0.0.1 Connection refused 5567.

4.3 start vscode debugging

Add a line of test code to the first line of the main function:

one

Args=new string [] {"localhost", "9999"}

It is mainly used to connect to the netcat service, receive the string sent by netcat, and count the string. As shown below:

4.4 debugging process

Open the "4.1Startup netcat" and "4.2Startup Debug Mode" windows. Netcat is used to send data, and the debug mode window can receive the data and count it. As shown below:

The above content is. Net for Spark how to achieve WordCount application and debug into the pit analysis, have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.