Using an Presigned Url With Postman Upload

When it comes to file uploads performed by client apps, "traditionally," in a "serverful" world, we might use the following approach:

1. on the client side, the user submits a form and the upload begins
2. one time the upload has been completed, we do all of the necessary piece of work on the server, such as check the file type and size, sanitize the needed data, maybe do image optimizations, so, finally, motility the file to a preferred location, be information technology another storage server or perhaps S3.

Although this is pretty straight forward, there are a few downsides:

1. Uploading files to a server tin can negatively touch on its organization resources (RAM and CPU), especially when dealing with larger files or prototype processing.
2. If you are storing files on a dissever storage server, you lot also don't have unlimited deejay space, which means that, as the file base grows, you'll need to exercise upgrades.
3. Oh, yeah, and did I mention backups?
4. Security — there are never enough preventive steps that you can implement in this department.
5. We constantly need to monitor these servers in lodge to avoid reanimation and provide the best possible user experience.

Woah! 😰

Just, luckily, there'southward an easier and better manner to perform file uploads! By using pre-signed Post data, rather than our own servers, S3 enables u.s.a. to perform uploads directly to it, in a controlled, performant, and very safe manner. 🚀

You might be asking yourself: "What is pre-signed Mail data and how does it all work together." Well, sit back and relax, considering, in this brusque post, we'll encompass everything y'all demand to know to get you started.

For demonstration purposes, we'll also create a simple app for which we'll utilise a fiddling bit of React on the frontend and a simple Lambda function (in conjunction with API gateway) on the backend.

Let'southward go!

How does it work?

On a high level, it is basically a two-pace process:

1. The client app makes an HTTP asking to an API endpoint of your choice (1), which responds (2) with an upload URL and pre-signed POST information (more than information about this presently). Annotation that this asking does not contain the actual file that needs to be uploaded, just information technology can contain additional data if needed. For instance, you might want to include the file name if for some reason you need it on the backend side. You are gratis to transport annihilation you demand, but this is certainly not a requirement. For the API endpoint, equally mentioned, we're going to utilize a simple Lambda function.
2. Once it receives the response, the client app makes a multipart/course-data POST request (3), this time directly to S3. This one contains received pre-signed Mail service data, forth with the file that is to be uploaded. Finally, S3 responds with the 204 OK response lawmaking if the upload was successful or with an appropriate error response code if something went incorrect.

Alright, at present that we've gotten that out of the way, you might still be thinking what pre-signed Post information is and what information it contains.

Information technology is basically a set of fields and values, which, get-go of all, contains information almost the actual file that's to exist uploaded, such equally the S3 key and destination bucket. Although not required, it'south also possible to set additional fields that further depict the file, for example, its content type or immune file size.

It also contains information most the file upload request itself, for example, security token, policy, and a signature (hence the name "pre-signed"). With these values, the S3 determines if the received file upload request is valid and, even more than chiefly, immune. Otherwise, everyone could only upload any file to it as they liked. These values are generated for you by the AWS SDK.

To cheque it out, let's accept a await at a sample effect of thecreatePresignedPost method phone call, which is part of the Node.js AWS SDK and which nosotros'll later use in the implementation section of this post. The pre-signed Post data is independent in the "fields" key:

            {     "url": "https://s3.us-east-two.amazonaws.com/webiny-cloud-z1",     "fields": {         "key": "uploads/1jt1ya02x_sample.jpeg",         "bucket": "webiny-cloud-z1",         "X-Amz-Algorithm": "AWS4-HMAC-SHA256",         "X-Amz-Credential": "A..../us-east-2/s3/aws4_request",         "10-Amz-Date": "20190309T203725Z",         "X-Amz-Security-Token": "FQoGZXIvYXdzEMb//////////...i9kOQF",         "Policy": "eyJleHBpcmF0a...UYifV19",         "X-Amz-Signature": "05ed426704d359c1c68b1....6caf2f3492e"     } }          

As developers, we don't really need to concern ourselves too much with the values of some of these fields (once nosotros're certain the user is actually authorized to asking this information). It's important to annotation that all of the fields and values must be included when doing the actual upload, otherwise the S3 will answer with an fault.

Now that we know the nuts, nosotros're set up to move onto the actual implementation. We'll first with the client side, after which we'll prepare our S3 bucket and finally create our Lambda office.

Client

As we've mentioned at the beginning of this post, nosotros're going to utilize React on the client side, so what we accept hither is a simple React component that renders a push, which enables the user to select any type of file from his local system. Once selected, nosotros immediately start the file upload procedure.

Allow's accept a expect:

	import React from "react" ;
	import Files from "react-butterfiles" ;
	/**
	* Retrieve pre-signed POST data from a defended API endpoint.
	* @param selectedFile
	* @returns {Promise<any>}
	*/
	const getPresignedPostData = selectedFile => {
	return new Promise ( resolve => {
	const xhr = new XMLHttpRequest ( ) ;
	// Set the proper URL here.
	const url = "https://mysite.com/api/files" ;
	xhr . open ( "POST" , url , true ) ;
	xhr . setRequestHeader ( "Content-Type" , "application/json" ) ;
	xhr . send (
	JSON . stringify ( {
	proper name: selectedFile . name ,
	type: selectedFile . blazon
	} )
	) ;
	xhr . onload = part ( ) {
	resolve ( JSON . parse ( this . responseText ) ) ;
	} ;
	} ) ;
	} ;
	/**
	* Upload file to S3 with previously received pre-signed Mail data.
	* @param presignedPostData
	* @param file
	* @returns {Promise<any>}
	*/
	const uploadFileToS3 = ( presignedPostData , file ) => {
	render new Hope ( ( resolve , reject ) => {
	const formData = new FormData ( ) ;
	Object . keys ( presignedPostData . fields ) . forEach ( key => {
	formData . append ( key , presignedPostData . fields [ key ] ) ;
	} ) ;
	// Actual file has to be appended final.
	formData . suspend ( "file" , file ) ;
	const xhr = new XMLHttpRequest ( ) ;
	xhr . open ( "Mail" , presignedPostData . url , true ) ;
	xhr . send ( formData ) ;
	xhr . onload = function ( ) {
	this . status === 204 ? resolve ( ) : refuse ( this . responseText ) ;
	} ;
	} ) ;
	} ;
	/**
	* Component renders a simple "Select file..." button which opens a file browser.
	* Once a valid file has been selected, the upload process will start.
	* @returns {*}
	* @constructor
	*/
	const FileUploadButton = ( ) => (
	< Files
	onSuccess = { async ( [ selectedFile ] ) => {
	// Step 1 - get pre-signed POST data.
	const { data: presignedPostData } = await getPresignedPostData ( selectedFile ) ;
	// Step two - upload the file to S3.
	try {
	const { file } = selectedFile . src ;
	await uploadFileToS3 ( presignedPostData , file ) ;
	console . log ( "File was successfully uploaded!" ) ;
	} catch ( e ) {
	console . log ( "An fault occurred!" , due east . message ) ;
	}
	} }
	>
	{ ( { browseFiles } ) => < button onClick = { browseFiles } >Select file...< / button > }
	< / Files >
	) ;

For an easier file selection and cleaner code, we've utilized a small package chosen react-butterfiles. The writer of the package is actually me, so if you have any questions or suggestions, feel free to allow me know! 😉

Other than that, there aren't any boosted dependencies in the code. We didn't fifty-fifty bother to use a 3rd party HTTP client (for example axios) since we were able to achieve everything with the born XMLHttpRequest API.

Note that we've used FormData for assembling the request torso of the second S3 asking. As well appending all of the fields independent in the pre-signed POST data, too brand certain that the actual file is appended as the last field. If yous do that before, S3 will render an fault, then watch for that i.

S3 saucepan

Permit'southward create an S3 bucket, which volition store all of our files. In case you don't know how to create it, the simplest way to do this would be via the S3 Management Panel.

Once created, we must suit the CORS configuration for the bucket. Past default, every bucket accepts only GET requests from some other domain, which ways our file upload attempts (POST requests) would exist declined:

            Access to XMLHttpRequest at 'https://s3.amazonaws.com/presigned-post-test' from origin 'http://localhost:3001' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.          

To fix that, merely open your saucepan in the S3 Direction Console and select the "Permissions" tab, where yous should be able to see the "CORS configuration" button.

Looking at the default policy in the above screenshot, we but need to suspend the following rule:

            <AllowedMethod>Mail service</AllowedMethod>          

The complete policy would and so exist the following:

            <CORSConfiguration>     <CORSRule>         <AllowedOrigin>*</AllowedOrigin>         <AllowedMethod>Become</AllowedMethod>         <AllowedMethod>POST</AllowedMethod>         <MaxAgeSeconds>3000</MaxAgeSeconds>         <AllowedHeader>Authorisation</AllowedHeader>     </CORSRule> </CORSConfiguration>          

Alright, permit's motility to the last piece of the puzzle and that's the Lambda function.

Lambda

Since it is a scrap out of the scope of this mail, I'll presume you already know how to deploy a Lambda office and expose it via the API gateway, using the Serverless framework. The serverless.yaml file I used for this piddling project can be plant here.

To generate pre-signed POST information, nosotros will apply the AWS SDK, which is by default available in every Lambda role. This is peachy, simply nosotros must be aware that it tin only execute deportment that were allowed by the role that is currently assigned to the Lambda function. This is important because, in our case, if the role didn't take the permission for creating objects in our S3 bucket, upon uploading the file from the customer, S3 would respond with the Admission Denied error:

            <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Admission Denied</Message><RequestId>DA6A3371B16D0E39</RequestId><HostId>DMetGYguMQ+due east+HXmNShxcG0/lMg8keg4kj/YqnGOi3Ax60=</HostId></Error>          

So, before standing, make sure your Lambda function has an acceptable office. For this, we can create a new role, and attach the following policy to it:

            {     "Version": "2012-10-17",     "Statement": [         {             "Sid": "VisualEditor0",             "Result": "Allow",             "Action": "s3:PutObject",             "Resource": "arn:aws:s3:::presigned-mail-data/*"         }     ] }          

A quick tip here: for security reasons, when creating roles and defining permissions, make certain to follow the principle of to the lowest degree privilege, or in other words, assign but permissions that are really needed by the function. No more than, no less. In our case, we specifically allowed s3:PutObject activeness on the presigned-post-data bucket. Avoid assigning default AmazonS3FullAccess at all costs.

Alright, if your role is fix, permit'due south take a look at our Lambda role:

	const S3 = crave ( "aws-sdk/clients/s3" ) ;
	const uniqid = crave ( "uniqid" ) ;
	const mime = require ( "mime" ) ;
	/**
	* Utilise AWS SDK to create pre-signed POST data.
	* Nosotros as well put a file size limit (100B - 10MB).
	* @param key
	* @param contentType
	* @returns {Promise<object>}
	*/
	const createPresignedPost = ( { key, contentType } ) => {
	const s3 = new S3 ( ) ;
	const params = {
	Expires: 60 ,
	Saucepan: "presigned-post-data" ,
	Weather: [ [ "content-length-range" , 100 , 10000000 ] ] , // 100Byte - 10MB
	Fields: {
	"Content-Type": contentType ,
	primal
	}
	} ;
	return new Promise ( async ( resolve , reject ) => {
	s3 . createPresignedPost ( params , ( err , data ) => {
	if ( err ) {
	reject ( err ) ;
	return ;
	}
	resolve ( data ) ;
	} ) ;
	} ) ;
	} ;
	/**
	* Nosotros demand to respond with adequate CORS headers.
	* @type {{"Access-Control-Allow-Origin": string, "Access-Control-Allow-Credentials": boolean}}
	*/
	const headers = {
	"Admission-Command-Allow-Origin": "*" ,
	"Admission-Control-Allow-Credentials": true
	} ;
	module . exports . getPresignedPostData = async ( { torso } ) => {
	try {
	const { name } = JSON . parse ( body ) ;
	const presignedPostData = wait createPresignedPost ( {
	key: ` ${ uniqid ( ) } _ ${ proper noun } ` ,
	contentType: mime . getType ( name )
	} ) ;
	return {
	statusCode: 200 ,
	headers,
	trunk: JSON . stringify ( {
	mistake: false ,
	information: presignedPostData ,
	message: null
	} )
	} ;
	} catch ( eastward ) {
	return {
	statusCode: 500 ,
	headers,
	body: JSON . stringify ( {
	error: true ,
	data: zero ,
	message: e . message
	} )
	} ;
	}
	} ;

Besides passing the basickey and Content-Blazon fields (line 18), we also appended the content-length-range status (line 17), which limits the file size to a value from 100B to 10MB. This is very of import, because without the condition, users would basically exist able to upload a 1TB file if they decided to practice it.

The provided values for the condition are in bytes. Also note that there are other file conditions you can use if needed.

Ane last note regarding the "naive" ContentType detection you might've noticed (line 49). Considering the HTTP request that volition trigger this Lambda function doesn't incorporate the actual file, information technology'southward impossible to check if the detected content type is really valid. Although this will suffice for this mail service, in a existent-world application you would do additional checks once the file has been uploaded. This can be done either via an boosted Lambda function that gets triggered once the file has been uploaded, or you could pattern custom file URLs, which point to a Lambda part and not to the actual file. This way, you can make necessary inspections (ideally simply in one case is enough) before sending the file back to the client.

Let's try it out!

If you lot've managed to execute all of the steps correctly, everything should exist working fine. To try it out, allow's first endeavour to upload files that don't comply with the file size condition. So, if the file is smaller than 100B, we should receive the post-obit mistake message:

            Post https://s3.the states-east-2.amazonaws.com/webiny-deject-z1 400 (Bad Request)  Uncaught (in promise) <?xml version="1.0" encoding="UTF-8"?> <Error><Code>EntityTooSmall</Code><Message>Your proposed upload is smaller than the minimum allowed size</Message><ProposedSize>19449</ProposedSize><MinSizeAllowed>100000</MinSizeAllowed><RequestId>AB7CE8CC00BAA851</RequestId><HostId>mua824oABTuCfxYr04fintcP2zN7Bsw1V+jgdc8Y5ZESYN9/QL8454lm4++C/gYqzS3iN/ZTGBE=</HostId></Error>          

On the other hand, if information technology's larger than 10MB, nosotros should as well receive the following:

            POST https://s3.us-east-2.amazonaws.com/webiny-cloud-z1 400 (Bad Request)  Uncaught (in promise) <?xml version="ane.0" encoding="UTF-8"?> <Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum immune size</Message><ProposedSize>10003917</ProposedSize><MaxSizeAllowed>10000000</MaxSizeAllowed><RequestId>50BB30B533520F40</RequestId><HostId>j7BSBJ8Egt6G4ifqUZXeOG4AmLYN1xWkM4/YGwzurL4ENIkyuU5Ql4FbIkDtsgzcXkRciVMhA64=</HostId></Mistake>          

Finally, if we tried to upload a file that's in the allowed range, nosotros should receive the 204 No content HTTP response and we should be able to run across the file in our S3 bucket.

Other approaches to uploading files

This method of uploading files is certainly not the only or the "right" 1. S3 actually offers a few means to attain the aforementioned thing. Yous choose the i that mostly aligns with your needs and environment.

For example, AWS Amplify customer framework might be a practiced solution for you, just if y'all're non utilizing other AWS services like Cognito or AppSync, yous don't actually need to employ it. The method we've shown here, on the client side, consists of 2 simple HTTP Postal service requests for which we certainly didn't demand to employ the whole framework, nor whatever other packet for that matter. E'er strive to make your client app build equally light as possible.

You lot might've too heard about the pre-signed URL approach. If y'all were wondering what is the difference between the 2, on a high level, it is similar to the pre-signed Postal service information approach, but it is less customizable:

Note: Not all performance parameters are supported when using pre-signed URLs. Certain parameters, such as SSECustomerKey, ACL, Expires, ContentLength, or Tagging must be provided as headers when sending a asking. If you lot are using pre-signed URLs to upload from a browser and demand to employ these fields, see createPresignedPost().

I notable feature that it lacks is specifying the minimum and maximum file size, which in this post we've accomplished with the content-length-rangecondition. Since this is a must-have if you ask me, the approach we've covered in this post would definitely be my get-to choice.

Additional steps

Although the solution we've built does the chore pretty well, there is always room for improvement. One time you hitting production, you will certainly want to add the CloudFront CDN layer, so that your files are distributed faster all over the world.

If you'll be working with prototype or video files, you will besides want to optimize them, considering it tin salve you a lot of bytes (and money of grade), thus making your app work much faster.

Conclusion

Serverless is a actually hot topic these days and it's not surprising since and then much work is abstracted away from u.s.a., making our lives easier as software developers. When comparing to "traditional serverful" architectures, both S3 and Lambda that we've used in this postal service basically require no or very picayune arrangement maintenance and monitoring. This gives united states more time to focus on what really matters, and ultimately that is the bodily product we're creating.

Thanks for sticking until the very stop of this article. Experience free to permit me know if you have whatsoever questions or corrections, I would be glad to check them out!

---

Thanks for reading! My name is Adrian and I work equally a full stack programmer at Webiny. In my spare fourth dimension, I similar to write about my experiences with some of the modern frontend and backend web development tools, hoping it might help other developers. If yous take any questions, comments or just wanna say hi, feel costless to attain out to me via Twitter.

mathewsgetice.blogspot.com

Source: https://www.webiny.com/blog/upload-files-to-aws-s3-using-pre-signed-post-data-and-a-lambda-function-7a9fb06d56c1/

Share this post