Shulou(Shulou.com)06/03 Report--

Background:

The company built its own IDC computer room and built big data cluster based on IDC. It needs to monitor the cluster resources. The cluster uses CDH cluster, and the collection is mainly divided into two parts:

HDFS and YARN related indicators are collected.

Collect the indexes of the IDC machine itself.

Note: some people may wonder why it is necessary to display the monitoring charts already provided in the CM interface. The reason is that this information needs to be integrated into the internal data platform to make corresponding data reports and visually display on their own data platform.

The implementation ideas can be roughly divided into two types:

Use the Java API provided by CM to get the REST API provided by CM.

In fact, the two are essentially the same. The Java API provided by CM is also implemented according to the set of REST API, and the two are consistent.

The core code is as follows:

Public class IdcHostResource {private static final Logger LOGGER = LoggerFactory.getLogger (IdcHostResource.class); static RootResourceV18 apiRoot;// TODO... Dead, need to improve static {apiRoot = new ClouderaManagerClientBuilder () .withHost ("cm ip") .withPort (7180) .withUsernamePassword ("user", "passwd") .build () .getRoot V18 ();} / * fixed access to basic resource information of Host * / public static List getAllHostResource () {List hosts = new ArrayList () HostsResourceV10 hostsResourceV10 = apiRoot.getHostsResource (); List hostLists = hostsResourceV10.readHosts (DataView.SUMMARY). GetHosts (); LOGGER.info ("Total" + hostLists.size () + "Host"); for (ApiHost hostList: hostLists) {IdcHostBasicInfo host = formatHost (hostsResourceV10.readHost (hostList.getHostId (); LOGGER.info ("HostName:" + host.getHostName ()); LOGGER.info ("Host HealthSummary:" + host.gethostHealthSummary ()) LOGGER.info ("Host Physical Memory:" + host.getTotalPhysMemBytes ()); hosts.add (host);} return hosts;} public static IdcHostBasicInfo formatHost (ApiHost apiHost) {IdcHostBasicInfo idcHostBasicInfo = new IdcHostBasicInfo (); idcHostBasicInfo.sethostHealthSummary (apiHost.getHealthSummary (). ToString ()); idcHostBasicInfo.setHostName (apiHost.getHostname ()); idcHostBasicInfo.setTotalPhysMemBytes (apiHost.getTotalPhysMemBytes ()); return idcHostBasicInfo } / * dynamically obtain the corresponding metrics info * * @ param query * @ param startTime * @ param endTime * @ return * / public static List getHostMetrics (String query, String startTime, String endTime) throws ParseException {TimeSeriesResourceV11 timeSeriesResourceV11 = apiRoot.getTimeSeriesResource (); ApiTimeSeriesResponseList responseList = timeSeriesResourceV11.queryTimeSeries (query, startTime, endTime); List apiTimeSeriesResponseList = responseList.getResponses (); List metrics = formatApiTimeSeriesResponseList (apiTimeSeriesResponseList); return metrics } public static List formatApiTimeSeriesResponseList (List apiTimeSeriesResponseList) throws ParseException {List metrics = new ArrayList (); DateUtils dateUtils = new DateUtils (); for (ApiTimeSeriesResponse apiTimeSeriesResponse: apiTimeSeriesResponseList) {List dataList = new ArrayList (); List apiTimeSeriesResponseLists = apiTimeSeriesResponse.getTimeSeries (); for (ApiTimeSeries apiTimeSeries: apiTimeSeriesResponseLists) {LOGGER.info ("query sql is:" + apiTimeSeries.getMetadata (). Get_Expression ()); IdcMetricInfo metric = new IdcMetricInfo () Metric.setMetricName (apiTimeSeries.getMetadata (). GetMetricName ()); metric.setEntityName (apiTimeSeries.getMetadata (). GetEntityName ()); metric.setStartTime (apiTimeSeries.getMetadata (). GetStartTime (). ToString ()); metric.setEndTime (apiTimeSeries.getMetadata (). GetEndTime (). ToString ()); for (ApiTimeSeriesData apiTimeSeriesData: apiTimeSeries.getData ()) {MetricData data = new MetricData () / / insert EntityName into Data to avoid the generation of duplicate data data.seHostname (apiTimeSeries.getMetadata (). GetEntityName ()); / / the default time format of CM is EEE MMM dd HH:mm:ss' CST' yyyy, and the conversion time format is yyyy-MM-dd HH:mm:ss data.setTimestamp (dateUtils.parse (apiTimeSeriesData.getTimestamp (). ToString () Data.setType (apiTimeSeriesData.getType ()); data.setValue (apiTimeSeriesData.getValue ()); dataList.add (data);} metric.setData (dataList); metrics.add (metric);}} return metrics;}

Note:

The DateUtils involved in the code needs to be implemented by itself.

Through this part of the code, we can get the metric information of the corresponding idc cluster by passing in tsquery; in the following code, we only need to achieve the acquisition code of the corresponding monitoring indicators through ServiceImpl.

If you want to integrate with spring boot through cm api, you will encounter two more problems:

The problem of dependency conflict is mainly manifested in the conflict between jackson and cxf, which can be solved by arranging jar packets.

Regular parsing error, which is a pit in the use of cm and is still under investigation. The specific forms are as follows:

There is a space in it, so regular parsing errors are reported directly during compilation; however, we can find that this problem is no longer present in the api version of cm 6.x:

Therefore, we can directly upgrade the version of api to solve this problem, but the resulting problem is that it is inconsistent with the online version of cm (the online version is 5.13.2), so we still need to think about how to solve it. However, after testing, it is found that the use of cm 6.x version of api does not affect the relevant indicators of the frontline version.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.