Nginx direct file upload

I found this very useful bit hit a couple of problems with it.

First, by default nginx was configured to store the files to /tmp/ as a different user to that I had running the proxy processes. Editing the user directive in /etc/nginx/nginx.conf was my solution to this.

Second, without adding "proxy_set_header Content-Length 0;" the response from my proxy just 'hung' open and had to be manually closed. This is a little confusing, but it seems to work for me :)

why only upload file successfully by using curl --data-binary?

I have tried normal form upload, but the temp file is form field name and value?

@dawncold because clientbodyinfileonly doesn't support RFC 2388

NGinx will only start passing the file to the backend when the upload is complete. So at least there won't be a backend worker blocked while the client is still sending the data. And your setup will only work with NGinx and the backend having access to the same filesystem. Other than that: nice trick! Thanks!

@niko,

1) it's not right, Nginx only pass the filename instead of full file body to backend. So the backend should not parse and cut the Content-Disposition headers.

2) back-end is not blocked until the file is uploaded by Nginx in any case

3) Nginx and back-end should have the same filesystem, it's right.

I tried your solution, upload works great but the tmp file gets stored as /tmp/00000x (x is a digit), and like the upload is async compared to the rest of the form that has already been saved as a "resource", how to you know what uploaded file belongs to what resource ? How can you know in advance where it's going to store it ?

@eppo back-end receives a request to URL http://backend/file with empty body(!) and the file name is header X-FILE. The storage location is declared by clientbodytemp_path

Ok I didn't notice that the upload request from the client is passed to the backend when the upload finishes within the same http request. So you are sure you're processing this client request. That's what I was wondering.
Thanks for your tip, really handy

@eppo yes, this is the callback, it fires only if file is uploaded and saved on disk successfully.

How to pass file path as GET or POST variable (instead of the header X-File) in the request to the back-end?

@dawncold you are able to send desirable file name in extra header and reuse it on back-end afterwards

@mikhailov, I have tried to making a URL like : http://back-end/file?name=xxx&path=$request_body_file, but I can't get the value of $requestbodyfile, and this variable's value only can be set in header. I don't know why.

@dawncold We do use extra headers with POST that pass through Nginx proxy and comes to back-end unmodified. So try to use X-NAME headers from initial requests and you will catch it up on backend.

Hi mikhailov, here's my problem:
I'm using reverse proxy with Nginx. When I POST a file to the Nginx, it seems that it will store the whole file in local and forward it to the backend server after received the whole file. I want a solution to make Nginx receive & forward data synchronously.

Can clientbodyinfileonly do this?

PS: proxy and backend are different servers.

@goace, the file stores to the local file system that's right. Once it has been uploaded the backend got synchronous callback (with X-FILE header and empty BODY) to the any URL you specified (http://backend/file in my case)

Saved files contain also the headers — is this the supposed behavior or am I doing something wrong? I.e.:

Content-Disposition: form-data; name="liteUploader_id"
fileUpload1

-----------------------------14064867571470422370146962914
Content-Disposition: form-data; name="custom"
tester

-----------------------------14064867571470422370146962914
Content-Disposition: form-data; name="fileUpload1[]"; filename="Gibson SG.jpg"

Content-Type: image/jpeg

...and then the binary image data follows.

@xfrf how do you upload a file?

@mikhailov our fault, already fixed it all. Thank you for the method description!

@xfrf, what was the problem?

I have a similar problem with the file output containing

------WebKitFormBoundaryvG6nluJ9VUrYg1BK
Content-Disposition: form-data; name="fileToUpload"; filename="Screen Shot 2013-09-16 at 8.48.45 AM.png"
Content-Type: image/png

How do I trim this data ? More importantly capture Content-Type as a Request Header perhaps something like X-Content-Type ?

@meson10 change the way you upload the file, get rid of multipart/form-data

How do I access the Request parameters then ?

through custom headers that Nginx preserve, they come with original request

My Bad. Not just content-type I am sending a bunch of request parameters too like:

------WebKitFormBoundaryXvqKozZ4exuycMpX
Content-Disposition: form-data; name="maxlength"

102400

These would obviously not be a part of the Headers, but of the request body.
Which would require some pruning of the file saved. Correct, Or am I missing a Trick here ?

(I am relatively new to DevOps and Advanced Nginx tricks, pardon my naiveness with the concepts.)

Thanks for sharing this approach!
I have one small question: When nginx stores file in /tmp/ directory it sets file access to 'rw' for file owner only (and owner is nginx user, say 'www-data'). Then I need to use this file from backend process which runs under a separate user (say, 'deployer'). I wonder, how do you deal with this case?

@kliuchnikau in case you have separate deployment user web-server may need to get the access to app tmp directory at least. So the approach is to run webserver under deployer user, it should be ok if you can control the application itself.

Анатолий, добрый вечер.

Спасибо за подробный how-to.

Не подскажите, правильно ли я понимаю, что нельзя влиять на именование файлов, которые будут записываться в upload? Если мы не берём возможность level 1-3?

Т.е. я не могу именовать файлы, например, вместо [\d]{10} как [\w]{6} ?

Спасибо.

@2naive, в рассылке были вопросы по этому поводу, но, насколько я в курсе изменений, этот модуль не трогали (лишь добавили auth_request по просьбам), поэтому на имя файла влиять пока нельзя.

Спасибо за ответ.

И ещё один глупый вопрос - могу ли я быть уверенным, что при восстановлении папки на новой площадке из бекапа с такими же настройками - восстановленные файлы не будут перезаписаны nginx'ом (или не будут записаны новые по причине наличия файла с таким же идентификатором)?

Спасибо.

For some reason on nignx 1.4.4 on Ubuntu using the default built in Uploader with nothing more than a max client body side set to 1GB I can get both PHP upload progress working and multipart with the $_FILES array being populated :S.

Not sure how that is but it is.

Hello Mikhailov,

is it possible to get the original uploaded file name? The X-File header only has the temp file name "/tmp/uploads/0000000001"

@pointblank, I don't think it's possible because Nginx has its own internal naming conventions for body request content

Hi,

Nginx upload module is supported in 1.4.4.

@mandrei99 this plugin author's attitude is pretty clear https://github.com/vkholodkov/nginx-upload-module/issues/41

It's a bit dangerous to rely on patches that can break and make core functionality of Nginx and upload-module unstable

@mandrei99 once you are start using non-supported plugins, it stops you from being on the latest stable Nginx version. New release is out, you are waiting for new patch again.
For example SPDY/2 support will be discontinuing soon, so Nginx 1.5.10 is a must.

We decided do not do that. Just using built-in functionality and develop a service on top of it. Core functionality is enough for our tasks.

files created by nginx:
2014/02/26 21:47:23 [notice] 4533#0: *1 a client request body is buffered to a temporary file /var/www/staging/0000000001, client: 127.0.0.1, server:.com, request: "POST /upload HTTP/1.1", host: "localhost"

created file has owner nobody and very restrictive permissions
-rw------- 1 nobody admin 140257 26 Feb 21:47 0000000001

I'd like to read the file and process it's contents in the backend but I can't seem to figure out how to tell nginx to use a different umask (022) for the files it creates. Can anybody help me please?

The exception I get in the backend:
java.nio.file.AccessDeniedException: /tmp/0055119830
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)

@ronnyf what is the user and group you have in Nginx configuration file? I've checked the files created with the user you specified. If that does not help, please take a look at http://en.wikipedia.org/wiki/Setuid#setgid_on_directories

@mikhailov
thanks, I synchronized the nginx user and the backend user to be the same - works alright.

@mikhailov many thanks, but how about this part '- back-end reads the file and cuts few headers Content-Disposition, Content-Type, stores it on disk again' ??

How can you cut the headers ?? can you use the uploaded file (e.g. 00000002) as a string ?

@vaggos2002 please look at RFC 1867. You can upload files one of either way: multipart form data or as a binary.

@mikhailov thanks a lot

How old is this post and comments? A year? 6 months? A week? Todayt?
I must be being really thick, but I don't see the rather essential dates! So when the post says:
"nginx-big-upload too young, nobody uses it in production yet" about something that has been in existence for over a year, it's kinda hard to get an idea of whether I should be reading this or not!

@digitaloast the reference: https://github.com/vkholodkov/nginx-upload-module/issues/41

@mikhailov - Thanks for this writeup. We are experiencing the same problems at the moment with nginx version upgrade problems.

One question I have is that we need to support ie8 (public sector clients) so we can't use XMLHttpRequest2, the only option is to use binary uploads as you mentioned. However using a normal html form, would we still use the multipart/form-data definition at the top? Would this system handle text files for example (non binary data)?

Thanks.
Hamza

@hamzakc, no, RFC 2388 (multipart/form-data) doesn't supported by Nginx clientbodyinfileonly

@mikhailov - Thanks for getting back to me. So how did you get round the IE8, 9 problem then? Did your site just not use nginx for uploads when using those browsers by doing some javascript detection?

nginx -t fails because of motive host not found in upstream "backend" in /etc/nginx/nginx.conf. am I missing something?

Здравствуйте, Анатолий!
Спасибо вам за данную заметку.
Я к сожалению столкнулся с рядом трудностей и отчаявшись, прошу Вас помочь разобраться мне, я обратил внимаение что в профиле вы указали rails, а я как раз его использую. rails 4.2, rack 1.6.0, nginx 1.6.2.

Если использовать данную конфигурации Webrick отдает 500 ошибку bad content body
Если убрать опцию proxysetbody off; то все отлично, НО запрос доходит до rails app и она его пишет в ОЗУ а этого делать не надо.
Если добавить proxysetbody off; и proxysetheader Content-Type "multipart/form-data"; то webrick возвращает ActionController::InvalidAuthenticityToken.

Я уже 3 день ковыряюсь с настройками и опциями, и все никак не могу заставаить это все работать. Мне то и нужено, что бы nginx сохранял файл, сообщал его путь в rails и что бы rails не писал тело файла в ОЗУ после nginx.

View Haml
= form_for :file, url: '/uploadfile', html: { multipart: true } do |f|
  = file_field_tag 'uploaded_files[]', required: true
  = f.submit 'Save'
end

Class
  def uploadfile
    # render :text => request.headers['X-File']
    render :text => env.inspect
  end
end

Route
Rails.application.routes.draw do
  resources :files
  root 'files#index'
  match '/uploadfile' => 'files#uploadfile', via: [:get, :post]
end

Или вот на pastebin

nginx.conf - http://pastebin.com/YuKjNkee
rails app - http://pastebin.com/G3Bwzggu

Если добавить в nignx к location "proxysetheader Content-Type "multipart/form-data";"
А в rails controller "protectfromforgery except: :uploadfile"
То схема заработает. Правда я не знаю, правильно ли и безопасно ли это делать.

Я только сейчас обратил внимание, что все вырезанные параметры и токены и все остальное было записано вместе с файлом, таким образом на . location /upload нужно отправлять только файл, без какой либо лишней информации.

I had to add "clientmaxbodysize" to the authrequest location too, otherwise nginx complained "client intended to send too large body" and "auth request unexpected status: 413"

Gorgeous. I thought there is no alternative to tengine's unbuffered upload.

How could I upload chunked file?

@craigloftus I'm having a possibly similar hanging problem... the file is sent up to the temp file and I get the nginx message

2015/06/04 09:03:53 [notice] 9239#0: *1 a client request body is buffered to a temporary ....

but then the backend doesn't get the call for another minute or so, I assume due to some timeout. I added
proxysetheader Content-Length 0;
to the X-FILE header setting that @mikhailov originally set, but it doesn't seem to help.

It does seem to be related to file size- perhaps it's just that the 'request body is buffered' shows up at the beginning of the transfer, and the forward happens after the transfer.

@mikhailov the 'proxysetbody off;' directive appears to set the body of the post to the backend to 'off'. That's why the Content-Length is set to 3. Perhaps you meant

proxypassrequest_body off;

anyway, cool trick.

This sounds cool. :)

Just wondering if this supports break-n-resume file uploading?

Hy, i just create repo in this github
https://github.com/ardinusawan/Nginx-direct-file-upload-without-passing-them-through-backend

Hope this help

Thank you, very actual article. Can you explain, is it possible to use this feature with reusable file upload? If network fails can I upload only a rest part of file rather than a full file?

Hi , I am completely new to nginx. My requirement is to upload a file using nginx. I have downloaded nginx zip file on windows and able to see "Welcome to nginx!" message on browser after starting it. nginx.conf file is updated as below

location /upload {
authbasic "Restricted Upload";
authbasicuserfile basic.htpasswd;
limit_except POST { deny all; }

clientbodytemppath /tmp/;
clientbodyinfileonly on;
clientbodybuffersize 128K;
clientmaxbody_size 1000M;

proxypassrequestheaders on;
proxysetheader X-FILE $requestbodyfile;
proxysetbody off;
proxyredirect off;
proxy_pass http://backend/file;

Now I am not sure on below
1. How will I leverage specific module to use like - clientbodyinfileonly or nginx-upload-module . What steps is required to use these modules.
2. What will be trigger point/source to invoke module. Will it be command based or java code needs to be written. I wanted to trigger this upload using java code.
3. What do I need to do for proxy_pass. Is it required to implement by my own.

Please suggest that I could proceed. None of source is available which describe the step for using it from end to end.

Is it possible handler more than one file at the same request?