Shulou(Shulou.com)06/03 Report--

The company has a flying tower firewall, which is based on utm, and many functions have expired. I configured the cracked anlayer to collect and analyze firewall policy logs, which is very easy to use, but this thing costs money and can only be tried for 30 days. I set the system time to 2017 to avoid expiration, but the collection time is not accurate and very troublesome.

As follows:

So I want to use graylog3.0 to collect and analyze the logs of Feita Firewall.

First of all, graylog is a big data analysis platform, only used to collect logs a little overqualified, but I do not understand big data =. =

1. Content packs and marketplace of graylog

Content pack (content packs) content packs are a convenient way to share configurations. A content pack is an JSON file that contains the configuration of a set of Graylog components. You can upload this JSON file to an Graylog instance and install it. Taking the time to create inputs for specific types of log formats, users of pipes and dashboards can easily share their efforts with the community.

Graylog marketplace is a market for sharing and buying content packs.

It can be simply understood that there are many gods who provide different ways of data analysis for people to download and use to meet the impassable needs of data analysis.

The address is

Https://marketplace.graylog.org/

Search for forita related

The search results are as follows:

I use the first one.

Contains the github address and configuration method

The github address is https://github.com/juiceman84/Fortigate_Content_Pack

2. Graylog imports content packs

After logging in to graylog, configure the content pack in system- > connet packs

You can see that the system comes with content packs. Click create a content pack to import.

Then fill in as follows

Remember to fill in the github address https://github.com/juiceman84/Fortigate_Content_Pack for url

And then next, you don't have to adjust.

And then the next step.

Don't worry about the rest, because I don't know what it is, and then click create.

Click install when you are finished

At this time, the content pack has been installed, isn't it very simple?

But you think it's simple because you don't know what so many options are for. =

3. Configure input

Configure the log receiving method in system- > input

Since the firewall logs are all in syslog format, create a new input for syslog udp

It can be configured as follows. The port is 30000, which is required by the content pack description. Udp or tcp protocol depends on whether the firewall supports it. The company's Feita firewall only supports udp protocol.

After completion, configure log sending on the firewall. The port needs to be consistent.

Then configure input's extrators (extractor), the so-called extractor, here are the official instructions

To put it simply, the syslog protocol is too simple, and you need an extractor to configure certain formats, such as display time, host, content, etc., otherwise the collected logs will be a mess of cvs text.

Then click import extrators in action

Here you are required to enter a json format code

I work in operation and maintenance, json who knows bro Mao.

Never mind, the extractor's json is already included in the github of the content pack, but we just have to write it manually

Then copy and paste the code

Then click add

After the addition is successful, you can see that many extractors appear, which are previously configured in the content pack

After the configuration is finished, then you can collect it. Click start.

Then you can see the information collected.

4. Configuration

The log information collected is very messy, including policy log, information log, and the content is very miscellaneous, so it is necessary to adjust

First, configure the fields displayed by the information (fields), including date, source address, destination address, etc. Do not select all of them, select the important ones, and check them according to the firewall policy quintuple information (source destination port, original destination address, protocol).

Secondly, in the search box, enter the type of information to be searched, and use type=traffic to filter out the information of traffic type.

Description:

1. In the search box, enter the field type=taffic to represent traffic

2. These selectable fields are used because content packs are configured, but content packs are not configured, and these fields are not filtered and selected.

Briefly explain the function of the field.

Timestamp: timestamp

Source: the source device that sends the log

Dst: destination addr

Dst_int: destination interface

Level: log level

Msg: processing action

Policyid: the hit policy id

Proto: protocol type

Servie: type of service

Sercetiy: security level

Src: source addr

Src_int: source interfac

Tpye: log type

Basically, with these things, you can analyze the policy log.

Then we click on a message.

You can see the specific class content of the information, which is basically a firewall policy information, very detailed.

Then we can save the search results so that we can use them directly next time

At the same time, you can put the search results in the dashboard.

In addition to being able to search, you can also configure field statistics, such as statistical source address ranking, policy ranking, etc.

For example, if you configure the ranking of the number of policies, you can see which policies are most referenced.

Configure generate chart (universal icon) to view the number of field information and the size of saved space within a certain period of time, which can be added to the dashboard

Configure Quick values, you can count a certain number, graphics, default pie chart statistics, can be added to the dashboard

Field statistics, counting the number, maximum and minimum values of this field, etc., which can be added to the dashboard

The last world map, you need to configure a geographic information extension to show how often the field appears in the geographic area, but not this time

All right, let's take a look at the dashboard. Once configured, it can be displayed directly.

Can basically meet the needs of daily operation and maintenance.

5. Thinking and expansion

1. Graylog supports streaming computing and alarm, that is, it supports real-time filtering of collected data according to certain rules, and then alarms according to the configured alarm rules, but I have not successfully configured it, so it seems that I need to learn the relevant knowledge of big data.

2. The search syntax of graylog is too simple and not allowed. It feels like a simple regular matching. Compared with splunk, there are other logging software with its own search syntax, which is too far away.

3. The log volume of the firewall is very huge. Basically, the peak of the transmission volume is 1500 messages per second, the peak of ipos is 30, and the peak bandwidth 30MBps takes up about 5 GB of storage space. My graylog almost failed several times, IO can not bear, actual production use, do not use virtual machine, you can also consider distributed multi-node deployment, sharing the pressure

4. The storage pressure is simply tested. 1000 pieces of data are wonderful, the data write size is more, 4-8kB is more, 64-512kB is the second, about 70% random write IO, the average IO write latency 100ms, there are still storage requirements.

5. Cpu and memory pressure, the average footprint of 16g memory is 99.5%, and the long-term occupation of 4-core 8vcpu in route 2 is 88.5%. At this time, the graylog search and response are obviously slow.

The advantages of graylog are open source, simple, easy to use out of the box, using content packs, and can collect and analyze almost any type of data.

But for firewall log collection, it is better to have special hardware.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.