Shulou(Shulou.com)06/01 Report--

This article introduces you how to use Elasticsearch real-time monitoring VPC flow log, the content is very detailed, interested friends can use for reference, I hope it can be helpful to you.

Keywords: ElasticSearch, vpc flow log, CloudWatch log

Readability: basic knowledge of AWS

Readable occupation: operation and maintenance, operation and management

Application scenario: ElasticSearch log analysis

Process architecture diagram

Step 1 create an Elasticsearch domain

Log in to the Elasticsearch console

Click "create New Domain", which is basically the default setting, and fill in the ES domain name

Everything else can be set by default, "next"

In order to facilitate the setting of access "public access"

Access policy setting "allow public access to domains"

Note: in general, you should select "allow access to a domain from a specific ip", but always report "User: anonymous is not authorized to perform: es:ESHttpGet" when accessing kinana during testing. For solutions that report errors, please refer to the official website document https://docs.aws.amazon.com/zh_cn/elasticsearch-service/latest/developerguide/aes-handling-errors.html. The simplest solution is to allow public access. Therefore, this operation practice is configured to allow public access.

Continue to "next", all use the default settings, click "OK" to successfully create the ES domain.

Step 2 create a CloudWatch log group

Switch to the CloudWatch console and select the log bar

Click "create Log Group", enter the log group name, and complete the log group creation, as shown in the following figure:

Click the created log group log-vpcflow to go in and view its contents

Because there is no log coming in yet, it is empty, and the next step is to generate a log.

Step 3 turn on VPC flow log

Two tasks need to be done in this section: 1) to generate vpc flow log, we can create an ec2 instance in vpc; 2) push vpc flow log to the CloudWatch log group.

1) create a new ec2 instance. The basic operation is omitted.

2) enable vpc flow log and configure

Switch to the vpc console and select the "your VPC" bar

In the current area, there are two vpc, a default vpc and a VPC created by myself (usually I use the VPC I created). The previous ec2 instance is also in this "vpc-xuyi", so select the VPC, you can see four tab columns in Xiamen, select "flow logs".

Click "create flow log" to start creating vpc flow logs

The filter selects "all" and the target selects "send to CloudWatch logs". In the target log group, select the log group "log-vpcflow" created in step 2. Now you also need to configure the IAM role, click "set up permission" in the blue font.

This is the permission to use the default configuration of the wizard. We do not need to modify it. Click "allow" to create a role "flowlogsRole". We can switch to the IAM console to view the role.

Ok, now we need to go back to the previously unfinished process and go to the VPC console-> Select the appropriate vpc- > create vpc flow Log-> configure the role

Click "create"

The Vpc flow log has been created.

Then switch to the CloudWatch console, enter the log group we created, and you can see "log-vpcflow" in the log group.

There is a flow log in the

Note: it may take some time for the flow log to appear here. Click the refresh icon on the right and brush it a few more times. The time is hard to say. It may be several minutes or very soon. Anyway, wait patiently.

We can click on the flow log to check it out (there is nothing to see)

Step 4 configure CloudWatch log groups

Now you also need to transfer the logs of the CloudWatch log group to the ElasticSearch domain, or the log group interface you created earlier.

Select streaming to ES

Select the account "this account", the ES cluster select the previously created es domain "es-vpc-log", and the lambda role select "create IAM role"

This is also the default role given by the configuration wizard. Just allow it directly, and a corresponding role will be created.

Once allowed, you have selected the role you just created in the roles section, and click "next"

Select "vpc flow Log" as the log format, click "next", and confirm it all the way.

Finally, you can see that our cloudwatch log group has been subscribed by a lambda function that streams our logs to the es domain. We can also go to the lambda console to see the architecture of the function (if you don't want to see it, you don't have to go in to see it, heh heh)

So the lambda function above is created by using the wizard, which is the advantage of the wizard, and I don't understand the code.

Step 5 Monitoring vpc flow log

Switch to the Elasticsearch console and select the domain we created

Select Index

The "cwl-2019.11.07" is the collected vpc flow log, named November 7, 2019.

Then select the Overview page

Click the link after "kibana" to go to the kibana page

It takes a long time to refresh the page. Wait, select the "manage" icon on the left in the image above.

Select "index patterns"

The index here is the log "cwl-2019.11.07" we saw earlier. Fill the index into the index pattern field, click "next", and you will see that the structure of the index is listed.

Then select the Discovery icon on the left

So far, we have implemented the monitoring of vpc flow log, and we can visually display the logs through Elasticsearch's kibana tool. Kibana itself is a powerful visualization tool, and I know a lot about it, so I can't give further guidance.

On how to use Elasticsearch real-time monitoring VPC flow log to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.