Last Updated: February 25, 2016
· bhaskerkode

Send many small packets or few big packets? #tcp

I wish more was said about optimizing the correlation between TCP transmission and CPU / load average.

Here is some code I quickly wrote that doesn't give me all the answers, but is a start.

Start a tcp server on one terminal

git clone
cd examples/tcp_echo
%% optionally comment the echo part
make && ./_rel/bin/tcp_echo_example console 

This starts the ranch tcp server on port 5555

I opened up another terminal for the client

Create a function that returns N bytes of "x"

1> Bin = fun(N)->
   end, <<>>, lists:seq(1,N)) end.
2> Bin(1).
3> Bin(10).

How long does it take to send a byte

4> timer:tc(fun()-> 
   {ok,C} = gen_tcp:connect("localhost",5555,[]),    

Try it out with different sizes

9> timer:tc(fun()-> 
    {ok,C} = gen_tcp:connect("localhost",5555,[]), gen_tcp:send(C,Bin(150000)) end).

Send 10 one after another with the same socket

11> timer:tc(fun()-> 
  {ok,C} = gen_tcp:connect("localhost",5555,[]),
  [gen_tcp:send(C,Bin(1400)) || _X <- lists:seq(1,10)], ok end).

Make a test function, memoize a bunch of stuff

25> Test4 = fun(Size,Packets) -> 
FullBin = Bin(Size), Max = lists:seq(1,Packets),     timer:tc(fun()-> 
{ok,C} = gen_tcp:connect("localhost",5555,[binary]),
[gen_tcp:send(C,FullBin) || _X <- Max], ok 
end) end.

Send 10,000 bytes, but with different size packets vs number of packets

32> [{Size,Packets,Test4(Size,Packets)} || {Size,Packets} <- 
[{1,10000}, % send 10k 1 byte sized packets
{10000,1}]].  % send one 10k byte sized packet

So sending 1 large packet with 10,000 bytes was fastest. I found this paper[1] interesting that says

1bps of network link requires 1Hz of CPU processing

But I wish I could see the impact on CPU, TIME_WAIT, etc which is less trivial to co-relate.