In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-17 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Development >
Share
Shulou(Shulou.com)06/03 Report--
This article mainly introduces the Go language Elasticsearch data cleaning tool how to use, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian take you to understand.
First, clean up the train of thought
You can see that the index data ends with a date. We can match the index data according to the date and delete the index. You need to consider here that some Es services enable index protection mechanism, which cannot be deleted through * index, but can only be deleted by the full name of the index. So our overall process is as follows:
1. Get all the index data in the es service.
2. Get the date to be deleted based on the current time-number of days of retention.
3. Determine whether the index contains the date to be deleted by string matching, and delete it if so.
4, tool-friendly, we can configure ES service address, date format type, retention days and other information through the configuration file.
2. Code implementation 2.1.All index data in ES service can be obtained.
To get all the index data in the Es service, we first connect to the Es server, here we use the github.com/olivere/elastic/v7 library to operate the Es.
Connect to the ES:
Func GetEsClient (data Data) * elastic.Client {Init () file: = ". / eslog.log" logFile, _: = os.OpenFile (file, os.O_RDWR | os.O_CREATE | os.O_APPEND, 0766) client, err: = elastic.NewClient (elastic.SetURL (data.Host), elastic.SetSniff (false), elastic.SetInfoLog (log.New (logFile, "ES-INFO:", 0)) Elastic.SetTraceLog (log.New (logFile, "ES-TRACE:", 0)), elastic.SetErrorLog (log.New (logFile, "ES-ERROR:", 0)),) if err! = nil {return nil} return client}
We connect to ES through the GetEsClient method and return client for use by subsequent methods. The Data here contains information such as the address of the ES service. We will give the data structure of Data later.
Get all index data
Func getIndex (data Data) map [string] interface {} {client: = GetEsClient (data) mapping: = client.GetMapping () service: = mapping.Index ("*") result, err: = service.Do (context.Background ()) if err! = nil {fmt.Printf ("create index failed, err:% v\ n", err) return nil} return result}
Get all the index data in the es service through client.GetMapping (). Index ("*") API and return it in the following format:
WX20211008-110537room2x
This time we get all the indexes complete.
2.2. Get the date to be deleted based on the current time-number of days of retention
We get the date data that needs to be deleted based on the current time-the number of days to be retained. We implement this function through GoLang's built-in function library time.
CurrentTime: = time.Now () / / get current time oldTime: = currentTime.AddDate (0,0, data.Day) / / obtain retention days through configuration file format: = oldTime.Format (data.IndexFmt) / / get serialized date format 2.3.Through string matching, determine whether the index contains the date to be deleted, and delete it if so
Here, string matching is used to determine whether the index data needs to be deleted.
Func delIndex (data Data) {currentTime: = time.Now () oldTime: = currentTime.AddDate (0,0, data.Day) format: = oldTime.Format (data.IndexFmt) index: = getIndex (data) / / get all indexes for k: = range index {/ / traversal index data fmt.Println ("key:", k, "format:", format) if find: = strings.Contains (k, format) Find {/ / determines whether the index contains the date format to be deleted DelIndex (data, k) / / call the DelIndex method to delete}} / / DelIndex to delete indexfunc DelIndex (data Data, index... string) bool {client: = GetEsClient (data) response, err: = client.DeleteIndex (index...) .Do (context.Background ()) if err! = nil {fmt.Printf ("delete index failed, err:% v\ n" Err) return false} return response.Acknowledged}
Deletes the specified data through DeleteIndexAPI.
2.4. Configure data flexibly through configuration files
Here we define Config and Data objects, and the object structure is as follows:
Type Config struct {Data [] Data `json: "data" `} type Data struct {Host string `json: "host" `IndexFmt string `json: "index_fmt" `Day int `json: "day"`}
The configuration file is as follows:
{"data": [{"host": "http://ip1:9200",// service IP" index_fmt ":" 20060102 ", / / date format" day ":-1 / / retention for 1 day}, {" host ":" http://ip2:9200/", "index_fmt": "20060102", "day":-1} {"host": "http://ip3:32093"," index_fmt ":" 2006.01.02 "," day ":-7 / / retention days}]}
We load the configuration file to Config through the Init method
Var config Config func Init () {JsonParse: = NewJsonStruct () / / the relative path is used below. The config.json file and the main.go file are in the same directory JsonParse.Load ("config/config.json", & config)} type JsonStruct struct {} func NewJsonStruct () * JsonStruct {return & JsonStruct {} func (jst * JsonStruct) Load (filename string, v interface {}) {/ / ReadFile function reads the entire contents of the file. The result is returned to data as [] byte type, and the data read by err: = ioutil.ReadFile (filename) if err! = nil {return} / / is in json format and needs to be decoded err = json.Unmarshal (data, v) if err! = nil {return}}
Write the Main method to run the program:
Func main () {Init () for I, datum: = range config.Data {fmt.Printf ("config data Host is [% s], fmt is [% s]\ n", datum.Host, datum.IndexFmt) println (I) delIndex (datum)}}
Here we still traverse multiple service configurations in the configuration file. Multiple Es services can be managed simultaneously.
Complete code package main import ("encoding/json", "fmt", "io/ioutil", "strings"time") type Config struct {Data [] Data `json: "data" `} type Data struct {Host string `json: "host" `IndexFmt string `json: "index_fmt" `Day int `json: "day"`} var config Config func Init () {JsonParse: = NewJsonStruct () / / the relative path is used below Config.json files and main.go files are in the same directory JsonParse.Load ("config/config.json", & config)} type JsonStruct struct {} func NewJsonStruct () * JsonStruct {return & JsonStruct {} func (jst * JsonStruct) Load (filename string, v interface {}) {/ / ReadFile function reads the entire contents of the file The result is returned to data as [] byte type, and the data read by err: = ioutil.ReadFile (filename) if err! = nil {return} / / is in json format. Err = json.Unmarshal (data, v) if err! = nil {return}} func delIndex (data Data) {currentTime: = time.Now () oldTime: = currentTime.AddDate (0,0, data.Day) format: = oldTime.Format (data.IndexFmt) index: = getIndex (data) for k: = range index {fmt.Println ("key:", k, "format:" Format) if find: = strings.Contains (k, format) Find {DelIndex (data, k)}} func main () {Init () for I, datum: = range config.Data {fmt.Printf ("config data Host is [% s], fmt is [% s]\ n", datum.Host Datum.IndexFmt) println (I) delIndex (datum)}} package main import ("context"fmt"github.com/olivere/elastic/v7"log"os"time") / / GetEsClient initializes client func GetEsClient (data Data) * elastic.Client {Init () file: = ". / eslog.log" logFile, _: = os.OpenFile (file) Os.O_RDWR | os.O_CREATE | os.O_APPEND, 0766) / / error should be judged Abbreviated here client, err: = elastic.NewClient (elastic.SetURL (data.Host), elastic.SetSniff (false), elastic.SetInfoLog (log.New (logFile, "ES-INFO:", 0)), elastic.SetTraceLog (log.New (logFile, "ES-TRACE:", 0)), elastic.SetErrorLog (log.New (logFile, "ES-ERROR:", 0)) ) if err! = nil {return nil} return client} / / IsDocExists determines whether the index stores func IsDocExists (data Data, id string, index string) bool {client: = GetEsClient (data) defer client.Stop () exist, _: = client.Exists (). Index (index) .ID (id) .Do (context.Background ()) if! exist {log.Println ("ID may be incorrect!") Id) return false} return true} / / whether PingNode is connected to func PingNode (data Data) {start: = time.Now () client: = GetEsClient (data) info, code, err: = client.Ping (data.Host) .Do (context.Background ()) if err! = nil {fmt.Printf ("ping es failed, err:% v" Err)} duration: = time.Since (start) fmt.Printf ("cost time:% v\ n", duration) fmt.Printf ("Elasticsearch returned with code% d and version% s\ n", code, info.Version.Number)} / / GetDoc get the document func GetDoc (data Data, id string, index string) (* elastic.GetResult, error) {client: = GetEsClient (data) defer client.Stop () if! IsDocExists (data, id) Index) {return nil, fmt.Errorf ("id does not exist")} esResponse, err: = client.Get (). Index (index) .Id (id) .Do (context.Background ()) if err! = nil {return nil, err} return esResponse, nil} / / CreateIndex create indexfunc CreateIndex (data Data, index, mapping string) bool {client: = GetEsClient (data) result Err: = client.CreateIndex (index) .BodyString (mapping) .Do (context.Background ()) if err! = nil {fmt.Printf ("create index failed, err:% v\ n", err) return false} return result.Acknowledged} / / DelIndex Delete indexfunc DelIndex (data Data, index... string) bool {client: = GetEsClient (data) response Err: = client.DeleteIndex (index...) .Do (context.Background ()) if err! = nil {fmt.Printf ("delete index failed, err:% v\ n", err) return false} return response.Acknowledged} func getIndex (data Data) map [string] interface {} {client: = GetEsClient (data) mapping: = client.GetMapping () service: = mapping.Index ("*") result Err: = service.Do (context.Background ()) if err! = nil {fmt.Printf ("create index failed, err:% v\ n", err) return nil} return result} Thank you for reading this article carefully I hope the article "how to use the Elasticsearch data cleaning tool in Go language" shared by the editor will be helpful to you. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.