Last Updated: January 19, 2018
·
1.137K
· sukima

Proxy large file downloads through SSH

Ever want to download a large file proxied through another server? Example use case: You want to download a file from a service but you want the user agent to come from another SSH server you have access too. You also don't want to fill up the SSH server's hard drive with the large file.

In my case I had a file that I could not access directly due to a firewall but could access it via curl on a SSH server. But the SSH server account had a user quota of 5 MB. To get the file to my local machine I did the following:

Open two terminals. In the first on SSH into the SSH server and prepare a FIFO:

me@localhost:$ ssh me@example.com
me@example.com:$ mkfifo large-buffer

Then in the second terminal:

me@localhost:$ ssh me@example.com "cat large-buffer" > ~/Downloads/large-file

Finally, back in the first terminal begin downloading and save the output to large-buffer

me@example.com:$ curl http://files.example.com/large-file > large-buffer

The transfer will happen over encrypted channels. The content won't be seen through the SSH session and the hard drive on the other side doesn't get filled up because the FIFO depletes itself as the second SSH session reads from it.