Amazon S3 – Upload/download large files to S3 with SpringBoot Amazon S3 MultipartFile application

In the previous post, we had learned how to upload a file to Amazon S3 in a single operation. But when file size reaches 100 MB, we should consider using multipart uploads with the advantages: improved throughput, quick recovery from any network issues. So Today, JavaSampleApproach will guide how to upload/download large files to S3 with SpringBoot Amazon S3 MultipartFile application.

Related post:
Amazon S3 – Upload/Download files with SpringBoot Amazon S3 application


I. Technologies

– Java 8
– Maven 3.6.1
– Spring Tool Suite: Version 3.8.4.RELEASE
– Spring Boot: 1.5.4.RELEASE
– Amazon S3

II. SpringBoot Amazon S3 MultipartFile application

AWS SDK for Java exposes a high-level API TransferManager class that simplifies multipart upload/download files.

SpringBoot Amazon S3 MultipartFile - upload-download - architecture

1. Upload/Download files

– Create an instance of the TransferManager:

-> minimumUploadPartSize & multipartUploadThreshold define the range size of each parts for uploading.
-> multipartCopyPartSize & multipartCopyThreshold define the range size of each parts for downloading.

– Use TransferManager.upload and TransferManager.download to upload/download files:

Note: TransferManager processes all transfers asynchronously, so this call will return immediately.
For block and wait for the upload/download to finish, we use:

2. Abort Multipart Uploads

Amazon S3 will charge for all storage associated with uploaded parts, so you need to abort the failed multipart upload to remove any uploaded parts from S3.

-> The above code will abort all uploadings which are not completely after 1 day processing.

3. Tracking Multipart Upload Progress

Amazon S3 APIs provides a listen interface, ProgressListener, to track the progress when uploading/downloading data using the TransferManager class.

III. Practice

In the tutorial, we create a SpringBoot project to upload/download file with TransferManager class.

SpringBoot Amazon S3 MultipartFile - upload-download - project structure

Step to do:
– Create SpringBoot project
– Configure TransferManager
– Implement S3 upload/download services
– Implement S3 client
– Run and check results

1. Create SpringBoot project

Using Spring Tool Suite to create a SpringBoot project. Then add aws-java-sdk dependency:

2. Configure TransferManager

Create a S3Config.java to config TransferManager:

– Open application.properties, add configuration:

– For logging in file, under /src/main/resources folder, adding logging.file=jsa-app.log then creating a logback-spring.xml file:

3. Implement S3 upload/download services

– Create an S3Services interface:

– Implement S3ServicesImpl:

4. Implement S3 client

In main class of SpringBoot application, using S3Services to upload/download a file:

5. Run and check results

Prepare:
– Setup Amazon S3 by the guide.
– In folder: C:\s3\upload, prepare a upload file:

SpringBoot Amazon S3 MultipartFile - upload-download - prepare upload file

Build and run above SpringBoot application with commandlines {mvn clean install, mvn spring-boot:run}. Waiting a minutes, on console log, we got:

-> Results:

– Upload file:

SpringBoot Amazon S3 MultipartFile - upload-download - result upload file

– When downloading, open folder C:\s3\download, we got:

SpringBoot Amazon S3 MultipartFile - upload-download - result download file in progress

– After downloading, open folder C:\s3\download, we got:

SpringBoot Amazon S3 MultipartFile - upload-download - result download done

– Open log file jsa-app.log, see:

IV. Sourcecode

SpringS3AmazonMultipartFile


Related Posts


Got Something To Say:

Your email address will not be published. Required fields are marked *

*