Shulou(Shulou.com)05/31 Report--

Most people do not understand the knowledge of this "what is Splunk" article, so the editor summarizes the following content, detailed content, clear steps, and has a certain reference value. I hope you can gain something after reading this article. Let's take a look at this "what is Splunk" article.

What is Splunk?

Splunk is the best commercial software implementation in the field of logging / streaming data, with only one core competence:

Search all generated logs within the enterprise like Google

The power of this is very great, today's enterprises are not short of data, what is lacking is the ability to mine data effectively. It is clear that most enterprises do not have the ability of Google to do search, so Splunk provides such capabilities. A competing open source implementation is Logstash.

Splunk ≈ LogstashLogstash = Redis (transfer) + ElasticSearch (search) + Kibana (presentation) ElasticSearch = Lucene + Search so, where can I buy it? # #

It's on Splunk's website, so I won't advertise for them. Anyway, it's very expensive. Ten thousand dollars can buy 1 gigabyte of traffic every day. To get back to the point, let me analyze some of the functional features of this thing.

First of all, Splunk has a cool interface.

As you can see, the main way to use Splunk is that search box, enter a search language called SPL, you can get all kinds of information you want. Splunk can filter, aggregate and count the data in the background, and finally get all kinds of reports and images.

SPL is a language that pays homage to SQL (chao). The syntax is very similar, except that SPL searches not a relational database, but all the log data entered into the Splunk system. Here are a few specific examples:

As you can see, for a line of SPL search statements

Sourcetype = syslog ERROR | top user | fields-precent

That's what Splunk did.

First of all, search the log with the field sourcetype (source type) of syslog on the hard disk, and the log contains the keyword ERROR.

Through the pipe character, aggregate the above search results according to the user field, and take out the top 10 that appear the most

Then through the pipe symbol, remove the percentage field, and finally get the result.

Finally, what did this search do? It instantly counts the top ten users with the most errors in the log, so that subsequent programmers can track why these errors occur and then start to solve them.

| | where distance/time > 100 |

Using where, divide the two fields in the log and compare them.

Cause and effect structure diagram

Splunk did three main things.

Parse the original log format and decompose it into meaningful fields. Some log collection schemes parse the log and send only the fields of concern in the first stage, in order to save bandwidth.

Associate log entries according to timestamps, request ID,session ID,user ID, etc., in order to clarify the status of each subsystem at that time.

According to the purpose of the analysis to do filtering, aggregation, statistics, etc., and finally a beautiful report out.

The outstanding feature of Splunk is that...

WEB's UI is excellent and plug-in, turning it into a platform that allows many third-party companies to publish applications on it.

Search syntax is powerful, such as finding the recent frequency of HTTP 503 errors, such as the list of items most visited by users in a given region, such as page traffic rankings. Basically, SPL can do any search you can think of that can be done by SQL.

Automatically guess some log fields, and you can manually adjust how to parse the source log on Web.

All of the above can be done by non-programmers who master the SPL language, which means that Splunk can be controlled by the product manager or the operations team. And it can also visualize the data.

Streaming search, real-time filtering logs and then alarm, which is very useful for the operation and maintenance team.

The above points determine that the market of Splunk is very large. The concept of this company is a data warehouse in the field of streaming data. 2012 of it is listed on NASDAQ, but it has been shorted and its stock has plummeted in the past two years. Because many cloud computing vendors can provide this service, such as Aliyun 1MB/S is free.

Competitive product analysis-- Logstash, Kafka##

# Splunk vs Logstash###

Logstash is an open source log search tool and an integrated out-of-the-box product. Basically, it can achieve 60% of the power of Splunk. Web is not that strong, nor does it have such a simple language as SPL. ElasticSearch needs to query through Json, and Kibana's search statement ability is limited. At present, it can be said that the Logstash project is still in the mature stage. It takes a lot of follow-up work to do it well.

# Splunk vs Kafka #

This comparison is actually not very fair.

Kafka only solves the problem of unified collection, transmission, serialization and storage of logs. Splunk does more and does deep processing of data indexing.

At the same time, Kafka needs to use schema at the source to define the data format, which is strict and convenient for later consumer programs.

However, Splunk has less requirements for source data and small changes to the existing system, because it is an enterprise software and needs to pursue compatibility.

In terms of high availability, Splunk currently has no case of collecting several T of data a day, and there is absolutely no problem with Kafka's ability in this area.

The Kafka is a good body frame, but it still lacks a powerful engine and a lot of interiors; the Splunk is a fully functional car, but it is very expensive and has never been driven more than 150 yards.

So, for Kafka, the possible overall solutions are:

Kafka + YARN + Hadoop = Samza (Linkin) Kafka + Strom + MySQLKafka + ElasticSearch + Kibana above is the content of this article about "what is Splunk?", I believe you all have some understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about it, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.