Shulou(Shulou.com)06/01 Report--

PinPoint distributed full link monitoring

All package download paths required for installation:

Link: https://pan.baidu.com/s/1sl9dP9r password: jsux

Introduction:

PinPoint is an open source APM (Application Performance Management/ Application performance Management) tool based on JAVA large-scale distributed systems.

Emulate GOOLE DAPPER,PinPoint to provide a solution by tracking calls between distributed applications to help analyze how the overall structure of the system and internal modules relate to each other

Server map (ServerMap):

Understand the system topology by visualizing the modules of the distributed system and the relationship between them. Clicking on a node will show the details of the module, such as its current status and number of requests.

Real-time active thread diagram (Realtime Active Thread Chart)

Monitor active threads within the application in real time.

Request / response Distribution Chart (Request/Response Scatter Chart)

Long-term visualization of the number of requests and response patterns to identify potential problems. You can choose to request more details by dragging on the chart

Call stack (CallStack)

Generate a code-level view for each call in a distributed environment, locating bottlenecks and points of failure in a single view

Inspection (Inspector)

View other details on the application, such as CPU usage, memory / garbage collection, TPS, and JVM parameters

Distributed system call chain monitoring features:

1. After the application architecture evolves from centralized to distributed, the whole invocation relationship becomes complex.

2. The distributed architecture is composed of complex and large-scale clusters. Brother applications are quite independent and may be implemented by different teams and different languages.

3. A complete calling process of the system may span multiple services and data centers.

4. It is difficult to locate the problem after the complex call causes the system problem.

5. It is impossible to know the overall system performance and operation accurately.

Data structure in Pinpoint

In Pinpoint, the core data structure consists of Span, Trace, and TraceId.

1. Span: the basic unit of RPC (remote procedure call / remote procedure call) tracking; when a RPC call arrives, it indicates that the work has been processed and contains trace data.

To ensure code-level visibility, Span has substructures with SpanEvent tags as data structures. Each Span contains a TraceId

2. Trace: a collection of multiple Span; composed of associated RPC (Spans). Span in the same trace shares the same TransactionId. Trace is sorted into inheritance tree structure by SpanId and ParentSpanId.

3. TraceId: a collection of key consisting of TransactionId, SpanId, and ParentSpanId. TransactionId specifies the message ID, while SpanId and ParentSpanId represent the parent-child relationship of RPC.

TransactionId (TxId): the ID; of messages sent / received by a single transaction between distributed systems must be globally unique across the entire server cluster.

SpanId: the ID; of the work that is processed when a RPC message is received is generated when the RPC request reaches the node.

ParentSpanId (pSpanId): the SpanId of the parent span that initiated the RPC call. If the node is the starting point of the transaction, there will be no parent span-in this case, a value of-1 is used to indicate that the span is the root span of the transaction.

Differences in terminology between Google Dapper and NAVER Pinpoint

Full-link performance monitoring

A complete call chain of a request may be like the following figure, through multiple system services, the invocation relationship:

Focus on the performance metrics of each call, such as throughput (TPS), response time, error logging, etc.

Throughput: the actual throughput of components, platforms, and physical devices can be calculated according to the topology

Response time: including the response time of the overall call and the corresponding time of each service, etc.

Error record: count the number of exceptions per unit of time returned according to the service

Full-link performance monitoring displays various indicators from the global dimension to the local dimension, and centrally displays the performance information of all the call chains across applications, which is convenient to measure the overall and local performance.

And it is convenient to find the source of the fault, and the troubleshooting time can be greatly shortened in production.

Environmental preparation:

System environment: centos 6.x or above, not too low

JDK preferably 7 or 8

Deploy HBASE (ZOOKEEPER for distributed use can use the built-in, but not a single point)

Build process:

Pinpoint consists of four parts.

1.Pinpoint Agent, 2. Pinpoint Collector, 3. HBase, 4.Pinpoint Web UI

1. Deploy JDK

The specific process is as follows:

A. Configure environment variable vim / etc/profile

Export JAVA_HOME=/usr/local/jdk

Export JRE_HOME=$ {JAVA_HOME} / jre

Export CLASSPATH=.:$ {JAVA_HOME} / lib:$ {jre_home} / lib

Export PATH=$ {JAVA_HOME} / bin:$PATH

# # configure environment variable execution: source / etc/profile

B. Check whether the configuration is successful

[root@mysql-jiachi local] # java-version

Java version "1.8.0,112"

Java (TM) SE Runtime Environment (build 1.8.0_112-b15)

Java HotSpot (TM) Server VM (build 25.112-b15, mixed mode)

2. Install and start hbase

Decompress hbase-1.1.11-bin.tar.gz

1. Tar-zxvf / root/pp/hbase-1.1.11-bin.tar.gz-C / usr/local/ # # it is recommended to rename it to hbase = > / usr/local/hbase

2 、 cd / usr/local/hbase/conf

Vim hbase-env.sh

27 export JAVA_HOME=/usr/local/jdk # refers to your corresponding JAVA_HOME. Default is java6.

Vim hbase-site.xml

Hbase.rootdir

File:///data/hbase # here we specify Hbase to store data locally, and production environments store data recommendations in HDFS.

3. Start the service

/ usr/local/hbase/bin/start-hbase.sh

Note: it is explained here that we are the simplest way to start Hbase

There is no separate deployment of zookeeper for Hbase cluster management, but directly uses the zk module that comes with Hbase. In addition, Hbase strongly recommends storing the data in HDFS. At present, we have a local Hbase.

4. Verify whether HBASE starts successfully

Jps

8114 Jps

7820 HMaster

5. HBASE initializes the table statement

/ usr/local/hbase/bin/hbase shell / usr/local/hbase-create.hbase # # specify the location of the table data

Finally, output a few table names, no exception means successful, look at the log.

3. Install Pinpoint-collector, whose service is based on tomcat, so you need to extract the tomcat package before deployment.

1. Tar-zxvf apache-tomcat-8.0.36.tar.gz-C / usr/local/ # # it is recommended to rename it to pinpoint-collector = > / usr/local/pinpoint-collector

2. Modify the configuration of pinpoint-collector 's Tomcat, mainly modify the port to avoid port conflicts with pp-web 's Tomcat (if multiple machines deploy TOMCAT, there is no need to modify the port)

3 、 cd / usr/local/pinpoint-collector/conf

Sed-I's bind port = "8005" / port= "18005" / g 'server.xml

Sed-I's bind port = "8080" / port= "18080" / g 'server.xml

Sed-I's bind port = "8443" / port= "18443" / g 'server.xml

Sed-I's bind port = "8009" / port= "18009" / g 'server.xml

Sed-I's server.xml redirect Port = "8443" / redirectPort= "18443" / g'Port

Sed-I "s/localhost/ `ifconfig eth0 | grep 'inet addr' | awk' {print $2}'| awk-F:'{print $2}'`/ g" server.xml

4. Rm-rf / usr/local/pinpoint-collector/webapps/*

5. Decompress the package pinpoint-collector-1.5.2.war

# mkdir / usr/local/pinpoint-collector/webapps/ROOT

# unzip pinpoint-collector-1.5.2.war-d / usr/local/pinpoint-collector/webapps/ROOT

6. Start tomcat

# / usr/local/pinpoint-collector/bin/startup.sh

4. Install pinpoint-web, the specific process is exactly the same as Pinpoint-collector, decompress a tomcat package.

1. Tar-zxvf apache-tomcat-8.0.36.tar.gz-C / usr/local/ # # it is recommended to rename it to pinpoint-web = > / usr/local/pinpoint-web

2. Modify the configuration of pinpoint-web 's Tomcat, mainly modify the port to avoid port conflicts with pinpoint-collector 's Tomcat (if multiple machines deploy TOMCAT, there is no need to modify the port)

3 、 cd / usr/local/pinpoint-web/conf

Sed-I's bind port = "8005" / port= "28005" / g 'server.xml

Sed-I's bind port = "8080" / port= "28080" / g 'server.xml

Sed-I's bind port = "8443" / port= "28443" / g 'server.xml

Sed-I's bind port = "8009" / port= "28009" / g 'server.xml

Sed-I's server.xml redirect Port = "8443" / redirectPort= "28443" / g'Port

Sed-I "s/localhost/ `ifconfig eth0 | grep 'inet addr' | awk' {print $2}'| awk-F:'{print $2}'`/ g" server.xml

4. Rm-rf / usr/local/pinpoint-web/webapps/*

5. Decompress the package pinpoint-web-1.5.2.war

# mkdir / usr/local/pinpoint-web/webapps/ROOT

# unzip pinpoint-web-1.5.2.war-d / usr/local/pinpoint-web/webapps/ROOT

# here is an explanation:

Hbase.properties configures the data source from which our pp-web gets the collected data. Here we only specify the zookeeper address of Hbase.

Connection authentication configuration for jdbc.properties pp-web to connect to its own Mysql database.

The sql directory pp-web itself has some data that needs to be stored in the MySQL database. Here you need to initialize the table structure.

Pinpoint-web.properties the configuration file for the pp-web cluster here, if you need a pp-web cluster.

These applicationContext-* .xml files will be used in subsequent tuning efforts.

Log4j.xml log related configuration

6. Start tomcat

# / usr/local/pinpoint-web/bin/startup.sh

5. Deploy pinpoint-agent to collect monitoring data

1. Decompress tar-zxvf pinpoint-agent-1.5.2.tar.gz

2 、 mv pinpoint-agent-1.5.2 / data/pp-agent

3. Edit the configuration file vim pinpoint.config # the test environment does not change on one machine, which is the default.

The main modification of IP, only need to specify the installation of Pinpoint-collector IP on the line, after the installation of Pinpoint-collector startup, automatically opened 99949995j9996 port.

Don't worry about it here. If you have port requirements, go to Pinpoint-collector 's configuration file ("Pinpoint-collector/webapps/ROOT/WEB-INF/classes/pinpoint-collector.properties") to modify these ports.

Profiler.collector.ip=192.168.245.136 (need to be determined here)

5. Add a probe and add a file at the entrance to the service user's access. Generally, the file name is "vim catalina.sh" under the startup directory.

# it's easy to deploy the collector, just add 3 words. I do a test Tomcat here to simulate deployment and store catalina.sh files in the bin directory under tomcat.

# pinpoint agent path

CATALINA_OPTS= "$CATALINA_OPTS-javaagent:/usr/local/pinpoint-agent-1.5.1/pinpoint-bootstrap-1.5.1.jar"

# the identification number of the monitored project using agent

CATALINA_OPTS= "$CATALINA_OPTS-Dpinpoint.agentId=0000002"

# name of the project being monitored

CATALINA_OPTS= "$CATALINA_OPTS-Dpinpoint.applicationName=172.16.10.43_3306_ customization"

#

The first line is the jar package location of pp-agent

The second line is the ID of agent. This ID is unique. I named it with pp + today's date, as long as it does not repeat the ID of other projects.

The third line is the name of the collection project, which can be chosen as long as each item is not repeated.

After configuring the catalina.sh file, start the tomcat file

6. Access pinpoint-web service

Other details can be studied more.

At this point, the whole deployment process is over. It is worth noting that:

If Hbase is not installed on the same machine as pinpoint-web and Pinpoint-collector, you need to install zookeeper, just install it, and do start port 2181.

If zookeeper is installed on a separate machine, you need to modify the configuration file pinpoint-collector.properties,pinpoint-web.properties of Pinpoint-collector and pinpoint-web, otherwise it will cause both modules to fail to start.

If a new version is recommended in the latest version, there are some problems with the old version.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.