Shulou(Shulou.com)06/01 Report--

Today, the editor will share with you the relevant knowledge points about how to use logstash's ruby module. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's take a look at it.

Arbitrary ruby code can be executed. But when I really realized it, I stepped on a lot of words and reported a lot of mistakes. When the ruby script is called in the first step, it always prompts "Script does not define a filter! Please ensure that you have defined a filter methods." the specified function is not loaded at all. Look through the logstash-filter-ruby source code and know that

There must be a filter function in the method, which can be solved when the function name is changed to filter, but the subsequent prompt does not define event. Continue to look through the source code to see this.

The event was passed when the custom ruby script was called. So change the event to the parameter passing solution. The final production configuration is as follows: filter {# executes parsing logic ruby {id= > "ls133" path = > ". / rubyLogParse.rb"}

# format date date {match= > ["requestTime", "yyyy-MM-dd HH:mm:ss"] target = > "@ timestamp" locale= > "en" timezone = > "+ 00:00"} # remove redundant fields and reduce es storage space mutate {remove_field = > ["@ version", "offset", "message", "beat", "input", "prospector"]}}

The conversion relationship between String and json in uby is as follows:

String--- > JSON JSON.parse String

JSON--- > String JSON.to_json

The definition method is as follows: # parse json format log def parseJsonLog (event,logdata) # get log content loginfo=logdata [(logdata.index (']') + 2)... (logdata.rindex ('}') + 1)] # convert it into json object logInfoJson=JSON.parse loginfo # get the serial number event.set ('transactionID',logInfoJson [' transactionID']) # get the business account event.set ('businessNumber' LogInfoJson ['businessNumber']) # get province code event.set (' provinceCode',logInfoJson ['provinceCode']) # get execution time-consuming event.set (' callTime',logInfoJson ['callTime']) # get interface code event.set (' serverCode',logInfoJson ['serverCode']) # get request ip event.set (' reqIp',logInfoJson ['reqIp']) # get channel event.set (' channle' LogInfoJson ['reqMessage'] [' channel']) # get the return message event.set ('respMessage',logInfoJson [' respMessage']. To_json) # get the request message event.set ('reqMessage',logInfoJson [' reqMessage']. To_json) if (loginfo.include?'errorInfo') event.set ('errorInfo',logInfoJson [' errorInfo']) end # set the mark bit event.set ('tags','log_success') end

# parsing xml format def parseXmlLog (event,logdata) # intercepting the serial number field if (logdata.include?'transactionID') & & (logdata.include?'businessNumber') transactionID=logdata [(logdata.index ('transactionID') + 16)... (logdata.index (' businessNumber')-3)] event.set ('transactionID') TransactionID) # judge the channel if (logdata.include?'Channel') channle=logdata [(logdata.index (') + 9)... (logdata.index ('))] event.set ('channle') according to the serial number Channle) end # intercepts the return message if (logdata.include?'respMessage') respMessage=logdata [(logdata.index ('respMessage') + 14)... (logdata.index (' reqMessage')-3)] event.set ('respMessage',respMessage) end omits part of the code. # set the mark bit event.set ('tags','log_success') end end

Def filter (event) # fetch log information logdata=event.get ('message') # if the log is not SMS log, cache monitoring log, cache log Access nodechanged if (logdata.include?'reqMessage') # parse log level and request time And save it to the event object logleve=logdata [(logdata.index ('[) + 1)... logdata.index (']')] event.set ('logleve',logleve) event.set (' requestTime',logdata [0mage logdata.index ('- [)]) # if it is the rest interface json log if (logdata.include?'restJson') parseJsonLog event,logdata else parseXmlLog event Logdata end end return [event] end above is all the content of the article "how to use the ruby Module of logstash" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.