Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to use S3 JAVA SDK in AWS. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

S3 JAVA SDK

S3 architecture design is independent of programming language and provides REST and SOAP interfaces. SOAP support on HTTP is deprecated, but can still be used on HTTPS. SOAP will not support the new S3 feature, so REST API is recommended.

With REST, buckets and objects can be created, extracted, and deleted using standard HTTP requests. Direct use of REST API for code development is complex, AWS SDK wraps the underlying REST API, which can simplify programming tasks.

Configure AWS Credentials

To use AWS SDK, you must provide AWS credentials and create them in ~ / .aws/credentials (the Windows user is C:\ Users\ USER_NAME.aws\ credentials):

[default] aws_access_key_id = your_access_key_idaws_secret_access_key = your_secret_access_keyPOM UTF-8 com.amazonaws aws-java-sdk-s3 com.amazonaws aws-java-sdk-bom 1.11.433 pom import

If you want to use all SDK, you do not need to use BOM, simply declare:

Com.amazonaws aws-java-sdk 1.11.433 S3 basic Operation

Demonstrates the basic operation of S3, such as createBucket, listBuckets, putObject, getObject, listObjects, deleteObject, deleteBucket and so on.

Package org.itrunner.aws.s3;import com.amazonaws.HttpMethod;import com.amazonaws.regions.Regions;import com.amazonaws.services.s3.AmazonS3;import com.amazonaws.services.s3.AmazonS3ClientBuilder;import com.amazonaws.services.s3.model.*;import java.io.File;import java.net.URL;import java.util.Date;import java.util.List;public class S3Util {private static AmazonS3 S3; static {S3 = AmazonS3ClientBuilder.standard () .withRegion (Regions.CN_NORTH_1) .build () } private S3Util () {} / * * Create a new S3 bucket-Amazon S3 bucket names are globally unique * / public static Bucket createBucket (String bucketName) {return s3.createBucket (bucketName);} / * * List the buckets in your account * / public static List listBuckets () {return s3.listBuckets () } / * * List objects in your bucket * / public static ObjectListing listObjects (String bucketName) {return s3.listObjects (bucketName);} / * * List objects in your bucket by prefix * / public static ObjectListing listObjects (String bucketName, String prefix) {return s3.listObjects (bucketName, prefix) } / * * Upload an object to your bucket * / public static PutObjectResult putObject (String bucketName, String key, File file) {return s3.putObject (bucketName, key, file);} / * Download an object-When you download an object, you get all of the object's metadata and a stream from which to read the contents. * It's important to read the contents of the stream as quickly as possibly since the data is streamed directly from Amazon S3 and your * network connection will remain open until you read all the data or close the input stream. * / public static S3Object get (String bucketName, String key) {return s3.getObject (bucketName, key);} / * * Delete an object-Unless versioning has been turned on for your bucket, there is no way to undelete an object, so use caution when deleting objects. * / public static void deleteObject (String bucketName, String key) {s3.deleteObject (bucketName, key);} / * Delete a bucket-A bucket must be completely empty before it can be deleted, so remember to delete any objects from your buckets before * you try to delete them. * / public static void deleteBucket (String bucketName) {s3.deleteBucket (bucketName);}} generate pre-signed URL

By default, the S3 object is private and only the owner has access. However, object owners can use their own security credentials to create a pre-signed URL and grant permission to download objects for a limited period of time to share objects with other users, and anyone who receives a pre-signed URL can access the object.

When creating a pre-signed URL, you must provide security credentials, bucket name and object key, HTTP method (specified as GET to download the object), and expiration time.

Public String generatePresignedUrl (String bucketName, String key, int minutes) {/ / Sets the expiration date Date expiration = new Date (); long expTimeMillis = expiration.getTime (); expTimeMillis + = 1000 * 60 * minutes; expiration.setTime (expTimeMillis); / / Generate the presigned URL. GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest (bucketName, key) .withMethod (HttpMethod.GET) .withExpiration (expiration); URL url = s3.generatePresignedUrl (generatePresignedUrlRequest); return url.toString ();} Select content from the object

With Amazon S3 Select, you can use SQL statements to filter the contents of S3 objects and retrieve part of the data you need. Amazon S3 Select is suitable for objects stored in CSV or JSON format, which can be compressed and encrypted on the server side by GZIP or BZIP2.

Requirements and limitations of S3 Select

Request:

You must have s3:GetObject permissions for the object you are querying.

If the queried object is encrypted, https must be used and an encryption key must be provided in the request.

Restrictions:

The maximum length of an SQL expression is 256 KB.

The maximum length recorded in the result is 1 MB.

SQL syntax

Amazon S3 Select supports partial SQL, and the syntax is as follows:

SELECT column_name FROM table_name [WHERE condition] [LIMIT number]

Where table_name is S3Object.

The SELECT clause supports *.

When the file format is CSV, reference columns can use column numbers or column names, starting with 1:

Select s._1 from S3Object sSelect s.name from S3Object s

When using column names, FileHeaderInfo must be set to Use in the program.

You can use double quotes to indicate that column names are case sensitive:

SELECT s. "name" from S3Object s

Column names are case-insensitive without double quotation marks.

For example, the CSV file is as follows:

Username,emailJason,jason@163.comCoco,coco@163.com

The SQL statement can be:

Select s.email from S3Object s where s.usernamekeeper Jason'

For more SQL information, see the SQL reference for Amazon S3 Select and Amazon Glacier Select.

Query CSV files

The following example saves the query results in an outputPath file:

Public static void selectCsvObjectContent (String bucketName, String csvObjectKey, String sql, String outputPath) throws Exception {SelectObjectContentRequest request = generateBaseCSVRequest (bucketName, csvObjectKey, sql); final AtomicBoolean isResultComplete = new AtomicBoolean (false); try (OutputStream fileOutputStream = new FileOutputStream (new File (outputPath)) SelectObjectContentResult result = s3.selectObjectContent (request) {InputStream resultInputStream = result.getPayload (). GetRecordsInputStream (new SelectObjectContentEventVisitor () {/ * * An End Event informs that the request has finished successfully. * / @ Override public void visit (SelectObjectContentEvent.EndEvent event) {isResultComplete.set (true);}}); copy (resultInputStream, fileOutputStream);} / * * The EndEvent indicates all matching records have been transmitted. If the End Event is not received, the results may be incomplete. * / if (! isResultComplete.get ()) {throw new Exception ("S3 Select request was incomplete as End Event was not received.");}} private static SelectObjectContentRequest generateBaseCSVRequest (String bucket, String key, String query) {SelectObjectContentRequest request = new SelectObjectContentRequest (); request.setBucketName (bucket); request.setKey (key); request.set_Expression (query); request.setExpressionType (ExpressionType.SQL); InputSerialization inputSerialization = new InputSerialization (); CSVInput csvInput = new CSVInput () CsvInput.setFileHeaderInfo (FileHeaderInfo.USE); inputSerialization.setCsv (csvInput); inputSerialization.setCompressionType (CompressionType.NONE); request.setInputSerialization (inputSerialization); OutputSerialization outputSerialization = new OutputSerialization (); outputSerialization.setCsv (new CSVOutput ()); request.setOutputSerialization (outputSerialization); return request } this is the end of the article on "how to use S3 JAVA SDK in AWS". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.