Shulou(Shulou.com)06/02 Report--

This article introduces you what is the introductory guide to operate Jingdong Cloud object Storage OSS based on Go SDK. The content is very detailed. Interested friends can refer to it. I hope it will be helpful to you.

Before the actual operation, let's take a look at the API interface support scope and capabilities of Jingdong Cloud OSS. As can be seen from the API documents of COS, JD.com Cloud provides two sets of APIs:

1. It is compatible with S3 API, supports AWS S3 interface, and is compatible with most important functions as the focus of subsequent development and optimization. Since it is not fully compatible with S3 interfaces, you need to focus on reading the list of compatible interfaces, as described in

Https://docs.jdcloud.com/cn/object-storage-service/compatibility-api-overview

2. The old version of OSS API and the restful interface encapsulated independently by JD.com Cloud support basic service, bucket, object and other operations. This version can still be used at present, but it will not be developed later.

Obviously, JD.com Cloud provides S3-compatible interfaces, which can quickly support existing applications based on AWS S3 on the one hand, and help customers migrate their data from AWS S3 on the other. It is recommended that you use S3-compatible interfaces, while Jingdong Cloud provides rich multi-language versions of SDK, which can be selected in different languages according to the needs of your project development.

For a related introduction, see

Https://docs.jdcloud.com/cn/object-storage-service/introduction-3

Here, you need to check the list of compatible APIs and compare the compatibility of Jingdong Cloud OSS and AWS S3 interfaces. For example, the Put Bucket API only uses the general request header, and the default is to create a bucket with the permission of private. ACL (i.e. Canned ACL) cannot be set with standard permissions because it does not support request headers such as x-amz-acl, xmuramzMurgrantMurray *. If you need to specify bucket ACL when you create it, you need to do it through another interface, which will be mentioned below.

List of compatible interfaces:

Https://docs.jdcloud.com/cn/object-storage-service/compatibility-api-overview

The following describes the basic operation of Jingdong Cloud OSS with Go SDK to achieve the functions of uploading and downloading object resources.

Environmental preparation

1. Install or update Go 1.5 or above (refer to 1-Mac to install GE locale configuration).

2. (optional) build Go compilation environment through visual IDE Atom. Compared with sublime text or traditional vim editing, Atom is a more advanced text code editor and a programming development tool created by Github. In addition to the beautiful interface, there are a variety of powerful plug-ins. This article takes the Atom development environment as a demonstration (refer to 2-Mac to build a Go development environment based on Atom)

3. Before initiating a request using Go SDK, you need to apply for accesskey and secretKey key pairs (referred to as AK/SK) in advance on the AccessKey management page under the account management of JD.com Cloud user Center. This key pair will be used in subsequent program initialization.

Download and install

1. Command line installation

Go get github.com/aws/aws-sdk-go

2. Atom installation (optional)

Menu [Packages]-> [Go]-> [Get Package], then type github.com/aws/aws-sdk-go, and wait a moment to download and install. The code is downloaded to the first path src directory in the GOPATH environment variable, which is the same as the command line installation method.

Overview of SDK composition

Before coding, it is recommended to understand the composition of AWS SDK, which mainly includes two parts: SDK core and service clients. SDK core is applicable to all AWS services, and client in service is only applicable to the corresponding service, which is invoked as the client of the service.

SDK core includes generic classes that make it easier to construct API parameters, such as Config, Logger, and so on. Among them

Awserr: the interface for process exceptions, which returns exceptions and errors encountered in the process, corresponding to error codes and information.

The call to credential:API requires authentication, which needs to be verified by using JD.com Cloud's AK/SK, and the default Config configuration item needs to be modified.

Endpoints: the invocation entry of the service, which has a region attribute and needs to be configured in Config.

Session: provides initialization of configuration, which can be initialized by customizing the parameters in the configuration, including region, endpoints, credential, etc.

Request: provides API requests and retries, and you can customize the request and how it is handled.

An example of creating a S3 client

Before initiating a request for OSS (S3 protocol), you need to initialize S3 client. The following is an example of creating a client.

1 ak: = "your accesskey" 2 sk: = "your secretkey" 3 token: = "" / / Token leave blank 4 creds: = credentials.NewStaticCredentials (ak, sk, token) 5 _, err: = creds.Get () 67 config: = & aws.Config {8 Region: aws.String ("cn-north-1"), / / Region9 Endpoint: aws.String ("s3.cn-north-1.jcloudcs.com") where Bucket is located / / Endpoint10 DisableSSL where Bucket is located: aws.Bool (false), 11 Credentials: creds,12} 13 client: = s3.New (session.New (config))

After creating the S3 client, you can request operations on oss resources such as bucket, object, and so on. The following describes the creation of bucket, upload files (PutObject, Upload), and multipart upload files for demo display.

Create bucket space

Goal: create a bucket named go-sdk-sample in JD.com Yun North China, and set ACL to public read and private write (public-read).

Sample code:

1 package main23 import (4 "fmt" 5 "os" 6 "github.com/aws/aws-sdk-go/aws" 7 "github.com/aws/aws-sdk-go/aws/credentials" 8 "github.com/aws/aws-sdk-go/aws/session" 9 "github.com/aws/aws-sdk-go/service/s3" 10) 1112 func main () {13 bucket: = "go-sdk-sample" 14 / / Create S3 service client15 svc: = s3.New (newSession ()) 1617 crParams: = & s3.CreateBucketInput {18 Bucket: aws.String (bucket) 19} 2021 _, err: = svc.CreateBucket (crParams) 2223 if err! = nil {24 exitErrorf ("Unable to create bucket% Q,% v", bucket, err) 25} 2627 / / Wait until bucket is created before finishing28 fmt.Printf ("Waiting for bucket% q to be created...\ n", bucket) 2930 err = svc.WaitUntilBucketExists (& s3.HeadBucketInput {31 Bucket: aws.String (bucket) 32}) 33 if err! = nil {34 exitErrorf ("Error occurred while waiting for bucket to be created,% v", bucket) 35} 3637 fmt.Printf ("Bucket% q successfully created\ n", bucket) 3839 puParams: = & s3.PutBucketAclInput {40 Bucket: aws.String (bucket), 41} 42 puParams.SetACL ("public-read") / / set bucket ACL4344 _ Err = svc.PutBucketAcl (puParams) 45 if err! = nil {46 exitErrorf (err.Error ()) 47} 48 fmt.Println ("Set", bucket, "ACL to public-read") 4950} 5152 func newSession () * session.Session {53 ak: = "your accesskey" 54 sk: = "your secretkey" 55 token: = "" / / Token leave 5657 creds: = credentials.NewStaticCredentials (ak, sk) Token) 58 creds.Get () 5960 config: = & aws.Config {61 Region: aws.String ("cn-north-1"), / / Region62 Endpoint: aws.String ("s3.cn-north-1.jcloudcs.com") where Bucket is located, / / Endpoint63 DisableSSL: aws.Bool (false), 64 Credentials: creds,65} 66 return session.New (config) 67} 6869 func exitErrorf (msg string) Args... interface {}) {70 fmt.Fprintf (os.Stderr, msg+ "\ n", args...) 71 os.Exit (1) 72}

As you can see in the example, we create bucket through the interface of CreatBucket and set the ACL of bucket through the interface of PutBucketAcl. Currently, JD.com Cloud supports three access permissions, including private read and write (private), public read and private write (public-read), and public read and write (public-read-write).

Execution result:

If the execution is successful, the console shows that the corresponding bucket is created successfully.

Upload files (PutObject)

Goal: upload an object to bucket and add MD5 check to ensure data integrity.

Sample code:

1 package main23 import (4 "crypto/md5" 5 "encoding/hex" 6 "fmt" 7 "io" 8 "os" 910 "github.com/aws/aws-sdk-go/aws" 11 "github.com/aws/aws-sdk-go/aws/credentials" 12 "github.com/aws/aws-sdk-go/aws/session" 13 "github.com/aws/aws-sdk-go/service/s3" 14) 1516 Func main () {17 filename: = "your file path" / / file path1819 file Err: = os.Open (filename) 20 if err! = nil {21 exitErrorf ("Unable to open file% Q,% v", err) 22} 23 defer file.Close () 2425 md5_file: = CreateMd5 (filename) 2627 / / Create S3 service client28 svc: = s3.New (newSession ()) 2930 input: = & s3.PutObjectInput {31 Body: file,32 Bucket: aws.String ("go-sdk-sample") 33 Key: aws.String (filename), 34 ContentMD5: aws.String (md5_file), / / MD5 check (optional) 35} 3637 result, err: = svc.PutObject (input) 3839 if err! = nil {40 exitErrorf ("PutObject Error,% v", err) 41} 4243 fmt.Println (result) 4445} 46 / / MD5 check 47 func CreateMd5 (filename string) string {4849 f Err: = os.Open (filename) 50 if err! = nil {51 exitErrorf ("Unable to open file% Q,% v", err) 52} 53 defer f.Close () 5455 md5hash: = md5.New () 56 io.Copy (md5hash) F) 5758 return hex.EncodeToString (md5hash.Sum (nil)) 5960} 6162 func newSession () * session.Session {63 ak: = "your accesskey" 64 sk: = "your secretkey" 65 token: = "" / / Token left blank 6667 creds: = credentials.NewStaticCredentials (ak, sk, token) 68 creds.Get () 6970 config: = & aws.Config {71 Region: aws.String ("cn-north-1") / / Region72 Endpoint where Bucket is located: aws.String ("s3.cn-north-1.jcloudcs.com"), / / Endpoint73 DisableSSL where Bucket is located: aws.Bool (false), 74 Credentials: creds,75} 76 return session.New (config) 77} 7879 func exitErrorf (msg string, args... interface {}) {80 fmt.Fprintf (os.Stderr, msg+ "\ n", args...) 81 os.Exit (1) 82}

As can be seen in the example, we upload the local file through the PutObject API, and in order to verify the accuracy of the data, we add the MD5 value of the file generated by the CreateMd5 method, and pass the parameter to ContentMD5. The OSS server will verify the file. If the same value is the same, it will return the successful result, that is, the ETAG value of the file.

Execution result:

You can see from the console that the file has been uploaded correctly, and the ETag matches the MD5 value calculated by the file. You can try to change the parameter of ContentMD5 to another value, which will return an error result, which will not be shown here.

Upload files (Uploader)

If the file is uploaded by PutObject, the mechanism of multipart, concurrency and retry is generally implemented through multipart upload to ensure the upload of large and large files. Here, we need to use CreateMultipartUpload, UploadPart, CompletedPart, AbortMultipartUpload and other interfaces to complete the whole process, while we need to implement retry, parallel and other logic, which is relatively complex. S3 provides S3 Upload Manager to help users achieve the above logic, and can automatically determine the file size to choose a single upload or multipart upload to complete. At the same time, users can customize parameters such as block size (PartSize), number of concurrency (Concurrency) and maximum number of blocks (MaxUploadParts) to meet the needs of different scenarios.

Sample code:

1 package main23 import (4 "fmt" 5 "os" 67 "github.com/aws/aws-sdk-go/aws" 8 "github.com/aws/aws-sdk-go/aws/credentials" 9 "github.com/aws/aws-sdk-go/aws/session" 10 "github.com/aws/aws-sdk-go/service/s3/s3manager" 11) 1213 func main () {14 bucket: = "go-sdk-sample" / / bucket name15 filename: = "your file path" / / file url1617 file Err: = os.Open (filename) 18 if err! = nil {19 exitErrorf ("Unable to open file% Q,% v", err) 20} 21 defer file.Close () 2223 sess: = newSession () 24 uploader: = s3manager.NewUploader (sess) 2526 upParams: = & s3manager.UploadInput {27 Bucket: aws.String (bucket), 28 Key: aws.String (filename), 29 Body: file,30} 3132 / / 33 result Err: = uploader.Upload (upParams, func (u * s3manager.Uploader) {34 u.PartSize = 10 * 1024 * 1024 / / Custom chunk size 35 u.LeavePartsOnError = true / / true for each part of 10m. Upload error will retain 36 u.Concurrency of successfully uploaded parts = 3 / / define the number of concurrency That is, the number of goroutines 37 u.MaxUploadParts = 10000 / / defines the maximum number of uploaded blocks 38}) 3940 if err! = nil {41 exitErrorf ("Put Object Error,% v") Err) 42} 4344 fmt.Printf ("Successfully uploaded!\ n") 45 fmt.Println ("File URL:" + result.Location) 46} 4748 func newSession () * session.Session {49 ak: = "your accesskey" 50 sk: = "your secretkey" 51 token: = "" / / Token leave 5253 creds: = credentials.NewStaticCredentials (ak, sk) Token) 54 creds.Get () 5556 config: = & aws.Config {57 Region: aws.String ("cn-north-1"), / / Region58 Endpoint: aws.String ("s3.cn-north-1.jcloudcs.com") where Bucket is located, / / Endpoint59 DisableSSL: aws.Bool (false), 60 Credentials: creds,61} 62 return session.New (config) 63} 6465 func exitErrorf (msg string) Args... int Erface {}) {66 fmt.Fprintf (os.Stderr, msg+ "\ n", args...) 67 os.Exit (1) 68}

As you can see, unlike the previous PutObject, s3manager.NewUploader is used to construct the upload service and upload files through the uploader.Upload method. The specific implementation logic of Upload can be found in the source code:

Https://github.com/aws/aws-sdk-go/blob/master/service/s3/s3manager/upload.go

For example, the logic for automatically determining whether to upload multipart files is as follows:

Execution result:

The size of the uploaded file is 16m, which is larger than 10m of PartSize, so multipart upload will be performed. During the upload process, you can see the multipart task that is being uploaded on the console-multipart Management, indicating that multipart upload is in progress. When upload is uploaded successfully, the file access address location will be returned.

At this point, we have introduced the main examples of OSS operations, and we will continue to organize the rest of the operations according to actual needs. For example, instead of using s3manager, we can directly use basic APIs such as MultipartUpload to achieve multipart upload. You can give it a try and we will decompose it next time.

Refer to the installation of GE language environment configuration under 1:Mac

1. Install Go

Install using brew, brew is a package management tool under Mac, similar to yum under CentOS, you can easily install, uninstall and update various software packages.

Brew install go

After installation, enter go version in the terminal to view the installed version, my display go version go1.12.5 darwin/amd64, indicating that the installation is v1.12.5 version.

2. Configure environment variables

View the environment variable settings for go

Building Go Development Environment based on Atom under go env reference 2:Mac

1. Install Atom

Https://atom.io/ Atom official website, you can download the software and install it directly.

2. Install the GE language environment (refer to the steps in 1)

3. Install the go-plus plug-in

Go-plus is an open source Golang development environment plug-in on Atom, project address:

Https://github.com/joefitzgerald/go-plus

You can find the install menu in Preference in Atom and enter go-plus.

Click: install, the go-plus,go-plus plug-in will be installed automatically and the corresponding dependent plug-in will be installed automatically. If the corresponding Golang class library is not installed, you can install it using go get.

After installation, we can code. In order to facilitate compilation, testing and compilation, we need to cooperate with terminal. We can open the terminal through view-Terminal without switching the interface.

Go-plus has many features, such as real-time feedback on syntax errors and compilation errors. Each time you save a file, go-plus will run your pre-configured go tools to execute in the background, such as go vet, go oracle, go build, etc., and then display errors and warnings at the bottom of the editor.

Go-plus can also display compilation error messages and error messages for that line on the corresponding line of the editor, so you can quickly locate which line is wrong. You can give it a try.

This is the end of the introduction guide to how to operate Jingdong Cloud object Storage OSS based on Go SDK. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.