Proxy large file downloads through SSH
Ever want to download a large file proxied through another server? Example use case: You want to download a file from a service but you want the user agent to come from another SSH server you have access too. You also don't want to fill up the SSH server's hard drive with the large file.
In my case I had a file that I could not access directly due to a firewall but could access it via curl
on a SSH server. But the SSH server account had a user quota of 5 MB. To get the file to my local machine I did the following:
Open two terminals. In the first on SSH into the SSH server and prepare a FIFO:
me@localhost:$ ssh me@example.com
me@example.com:$ mkfifo large-buffer
Then in the second terminal:
me@localhost:$ ssh me@example.com "cat large-buffer" > ~/Downloads/large-file
Finally, back in the first terminal begin downloading and save the output to large-buffer
me@example.com:$ curl http://files.example.com/large-file > large-buffer
The transfer will happen over encrypted channels. The content won't be seen through the SSH session and the hard drive on the other side doesn't get filled up because the FIFO depletes itself as the second SSH session reads from it.