Last Updated: September 09, 2019
·
142.7K
· mikhailov

Nginx as proxy for Amazon S3

The page content was out of date, see Stackoverflow for latest updates: https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584

13 Responses
Add your response

Doesn't this mean that you have reduced/removed the scalability or CDN features of S3? Since all requests now go through the nginx server. You are effectively using S3 to store the files, then piping them through this nginx server.

over 1 year ago ·

@cbess CloudFrount CDN is not feature of S3! CloudFront can use S3 just as origin.
For some reason CDN is not an option for certain business purposes, such data privacy, specific geo-location (if should not be replicated to another regions).

over 1 year ago ·

@mikhailov Correct, S3 is not a CDN. However, my point was that you reduce the scalability by having all traffic diverge to your server. Meaning, you take on the entire load of S3 traffic, thereby sidestepping its ability to scale requests. Aren't you taking the hit for bandwidth and CPU load to serve through nginx? Nginx has to serve and buffer the request to the client. Thanks for the clarification.

over 1 year ago ·

@cbess this solution is not for everybody, but for specific requirements. Please re-read the post again. Sure, proxy doubles the traffic and use CPU, in terms of scalability it solves by using array of Nginx servers.

over 1 year ago ·

@mikhailov Thanks for the post. Can you also please guide us as to how to write a custom proxy module for nginx instead of using the configuration. We need to do a check in database before proxy-ing to s3 for some security reasons. We do not want to serve all requests and rules are written in a database. So if a rule with the request does not match we throw a 404.

over 1 year ago ·

@debjitk my advice is do not write custom authentication mechanism, but using built-in functionality http://nginx.org/en/docs/http/ngx_http_auth_request_module.html

it helps you to do backend request (with following database rules detection) easily.

over 1 year ago ·

Thanks for this, I used it with XSendfile which makes this even more awesome! If you have never used XSendfile (or X-Accel-Redirect as nginx calls it) it is worth a look. You can use it with proxy pass, so it makes it ideal for a setup like this.

over 1 year ago ·

Hi
I have tried the public config and I see that it works on my local ec2 <--> s3 setup, however when I am trying to cache I don't see the files being saved locally on the ec2.
any Idea?

over 1 year ago ·

Great article. One minor point - When you say "High SLA" I think you mean "High Redundancy" (S3 actually has NO SLA (Service level agreement) - It could down for a week and you'd have no legal recourse)

over 1 year ago ·

Fix your code for future readers please

proxypass http://$s3bucket/$url_full;

is actually

proxypass http://$s3bucket$url_full;

or

proxypass http://$s3bucket$uri;

Nice article

over 1 year ago ·

Btw the 172.16.0.23 resolver didn't work for me so i used the google resolver and it worked

over 1 year ago ·

@mikhailov Hi, do you still around mikhailov? I have a question, i hope you can answer it.

over 1 year ago ·

@njsaunders It's not great, but they do have one (now) -> https://aws.amazon.com/s3/sla/

over 1 year ago ·