Shulou(Shulou.com)06/01 Report--

The infrastructure side of the company wants to extract slow jobs and learn about the waste of resources, so install a dr elephant. LinkIn open source system for performance analysis and tuning recommendations for yarn-based mr and spark jobs.

Most of the DRE development is based on java, while the spark monitoring part uses scala development and uses the play stack framework. This is a framework similar to Django in Python, based on java?scala? I don't know much about it. I can use it directly. I need more than java1.8.

Prerequest list:

Java 1.8

PlayFramework+activator

Nodejs+npm

Scala+sbt

The compilation server is a CVM set up in Silicon Valley in the United States. Java,maven,ant,scala,sbt and other compilation tools have been installed for bigtop, so download activator and decompress it to / usr/local and add PATH.

Then a dr-elephant from github clone, open compile.conf, modify the hadoop and spark versions to the current version,: wq save exit, run compile.sh to compile, after a short wait, because the United States server, the next dependent fast. There will be a dist folder, which will pack a dr-elephant-2.0.x.zip, copy it and decompress it.

DRE itself requires support above mysql 5.5, or the latest stable version of mariadb 10.1 is also available. There will be a problem here, that is, these three lines in DRE/conf/evolutions/default/1.sql:

Create index yarn_app_result_i4 on yarn_app_result (flow_exec_id); create index yarn_app_result_i5 on yarn_app_result (job_def_id); create index yarn_app_result_i6 on yarn_app_result (flow_def_id)

Because in some database cases, the index length exceeds the limit of the database itself, you need to modify the index length to avoid not being able to start.

Create index yarn_app_result_i4 on yarn_app_result (flow_exec_id); create index yarn_app_result_i5 on yarn_app_result (job_def_id); create index yarn_app_result_i6 on yarn_app_result (flow_def_id)

And then there should be no problem.

Create a database called drelephant in the database, and give the relevant access users

Next, you need to configure DRE:

Open app-conf/elephant.conf

# Play application server port# web port port=8080# Database configuration# database host that play framework listens to after starting dre, username password library name db_url=localhostdb_name=drelephantdb_user= "root" db_password=

Other defaults are fine. No change is required.

And then GeneralConf.xml.

Drelephant.analysis.thread.count 3 Number of threads to analyze the completed jobs drelephant.analysis.fetch.interval 60000 Interval between fetches in milliseconds drelephant.analysis.retry.interval 60000 Interval between retries in milliseconds drelephant.application.search.match.partial true If this property is "false", search will only make exact matches

Modify drelephant.analysis.thread.count. The default is 3. It is recommended that the reading speed from jobhistoryserver is too slow if it is modified to 10. If it is read too fast if it is higher than 10, it will cause great pressure on jobhistoryserver. The following two are the time period for reading and the interval period for retrying reads.

Then perform start.sh startup under bin. And then, show smile to the yellow elephant .

After loading and looking at this thing, in fact, the principle itself is not complicated, that is, it is not impossible to read all kinds of jmx,metrics and log information and write one by yourself. The main function is to put the contents of the job information into a screen to display, saving a point in the JHS page.

That's it, so easy

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.