Shulou(Shulou.com)06/01 Report--

In this article Xiaobian for you to introduce in detail "Go how batch operation excel into mongodb", detailed content, clear steps, details handled properly, I hope that this "Go how batch operation excel into mongodb" article can help you solve doubts, the following follow the editor's ideas slowly in-depth, together to learn new knowledge.

Requirements: complete a command tool to batch process some excel under a directory, import these excel data into mongodb, and synchronize them to mysql.

Code directory:

├─ cmd | └─ ecc.go # command ├─ configs ├─ data ├─ internal │ └─ importing # main logic processing ├─ pkg # │ ├─ files │ ├─ mongo mongo └─ mysql ├─ queue └─ tools1. Select command line package

The commonly used command toolkits are:

Urfave/cli

Spf13/cobra

Urfave/cli package is used here, which is relatively simple.

Var DirPath = ".. / data" / / default location var dir = DirPathapp: = & cli.App {Name: "Ecc", Usage: "Ecc is a tools for batch processing of excel data", Flags: [] cli.Flag {& cli.StringFlag {Name: "model" Aliases: [] string {"m"}, Usage: "The model of searching", Value: "model", Destination: & model,} & cli.StringFlag {/ / set a parameter of-d Used to determine the destination folder location Name: "dir", Aliases: [] string {"d"}, Usage: "Folder location of data files", Destination: & dir Value: DirPath,}, Action: func (c * cli.Context) error {importing.Load (".. / configs/cfg.yaml") / / introduce the configuration file Read mongodb, mysql and other configuration importing.Handle (dir) # # specific logic processing return nil} 2. Read configuration, connect to database

Read the configuration using the spf13/viper library. You need to read the configuration and connect to mongodb.

Var C Configtype Config struct {Env string `yaml: "env" `Mongo struct {DNS string `yaml: "dns" `Db string `yaml: "db" `Collection string `yaml: "collection" `} `yaml: "mongo" `Mysql struct {Alias string `yaml: "alias" `Dns String `yaml: "dns" `} `yaml: "mysql"`} func Load (cf string) {var err error viper.SetConfigFile (cf) if err = viper.ReadInConfig () Err! = nil {log.Fatal ("fatal error config file:% s\ n", err)} if err = viper.Unmarshal (& configs.C); err! = nil {log.Fatal (fmt.Errorf ("unmarshal conf failed, err:%s\ n", err)) if err = mongo.Conn (configs.C.Mongo.DNS, configs.C.Mongo.Db) Err! = nil {log.Fatal (color.RedString ("% s:\ n% v", "mongo connect err", err)) if mongo.CheckCollection (configs.C.Mongo.Collection) {if err = mongo.DelCollection (configs.C.Mongo.Collection) Err! = nil {log.Fatal (color.RedString ("% s:\ n% v", "mongo del collection err", err))} if err = mongo.CreateCollection (configs.C.Mongo.Collection); err! = nil {log.Fatal (color.RedString ("% s:\ n% v", "mongo create collection err", err)) 3. Read a file

First determine the file permissions and whether the file exists

Func ReadDir (dir string) ([] os.FileInfo, error) {perm: = checkPermission (dir) if perm = = true {return nil, fmt.Errorf ("permission denied dir:% s", dir)} if isNotExistDir (dir) {return nil, fmt.Errorf ("does not exist dir:% s", dir) files Err: = ioutil.ReadDir (dir) if err = = nil {return files, err return nil, fmt.Errorf ("ReadDir:% s, err:% v", dir, err)}

After getting the file, you should read the data of each excel file concurrently.

The requirement here is that all files must be read in one task, and if there is any error in any file, exit the program.

:: so you need to define an exception exit channel and a complete read two channels, and the total data is written securely using sync.Map.

3.1. Read rWait = truerDone = make (chan struct {}) rCrash = make (chan struct {}) read (f, dir, data) for rWait {/ / use the for loop to block reading the file select {case

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.