Rusty Russell's Coding Blog | Stealing From Smart People

May/14

27

Effects of packet/data sizes on various networks

I was thinking about peer-to-peer networking (in the context of Pettycoin, of course) and I wondered if sending ~1420 bytes of data is really any slower than sending 1 byte on real networks.  Similarly, is it worth going to extremes to avoid crossing over into two TCP packets?

So I wrote a simple Linux TCP ping pong client and server: the client connects to the server then loops: reads until it gets a ’1′ byte, then it responds with a single byte ack.  The server sends data ending in a 1 byte, then reads the response byte, printing out how long it took.  First 1 byte of data, then 101 bytes, all the way to 9901 bytes.  It does this 20 times, then closes the socket.

Here are the results on various networks (or download the source and result files for your own analysis):

On Our Gigabit Lan

Interestingly, we do win for tiny packets, but there’s no real penalty once we’re over a packet (until we get to three packets worth):

Over the Gigabit Lan

Over the Gigabit Lan

gigabit-lan-closeup

Over Gigabit LAN (closeup)

On Our Wireless Lan

Here we do see a significant decline as we enter the second packet, though extra bytes in the first packet aren’t completely free:

Wireless LAN (all results)

Wireless LAN (all results)

Wireless LAN (closeup)

Wireless LAN (closeup)

Via ADSL2 Over The Internet (Same Country)

Ignoring the occasional congestion from other uses of my home net connection, we see a big jump after the first packet, then another as we go from 3 to 4 packets:

ADSL over internet in same country

ADSL over internet in same country

ADSL over internet in same country (closeup)

ADSL over internet in same country (closeup)

Via ADSL2 Over The Internet (Australia <-> USA)

Here, packet size is completely lost in the noise; the carrier pidgins don’t even notice the extra weight:

Wifi + ADSL2 from Adelaide to US

Wifi + ADSL2 from Adelaide to US

Wifi + ADSL2 from Adelaide to US (closeup)

Wifi + ADSL2 from Adelaide to US (closeup)

Via 3G Cellular Network (HSPA)

I initially did this with Wifi tethering, but the results were weird enough that Joel wrote a little Java wrapper so I could run the test natively on the phone.  It didn’t change the resulting pattern much, but I don’t know if this regularity of delay is a 3G or an Android thing.  Here every packet costs, but you don’t win a prize for having a short packet:

3G network

3G network

3G network (closeup)

3G network (closeup)

Via 2G Network (EDGE)

This one actually gives you a penalty for short packets!  800 bytes to 2100 bytes is the sweet-spot:

2G (EDGE) network

2G (EDGE) network

2G (EDGE) network (closeup)

2G (EDGE) network (closeup)

Summary

So if you’re going to send one byte, what’s the penalty for sending more?  Eyeballing the minimum times from the graphs above:

Wired LAN Wireless ADSL 3G 2G
Penalty for filling packet 30%  15%  5%  0%  0%*
Penalty for second packet 30%  40%  15%  20%  0%
Penalty for fourth packet 60%  80%  25%  40%  25%

* Average for EDGE actually improves by about 35% if you fill packet

RSS Feed

4 Comments for Effects of packet/data sizes on various networks

Jon Smirl | May 27, 2014 at 11:40 pm

Turn off 802.11b support on your router and the wireless overhead should drop. There is a period in between each packet where the router looks for 802.11b devices and that wastes time if there are none.

Author comment by rusty | May 28, 2014 at 4:30 pm

Hmm, I was on 5GHz. Turned of 2.4GHz altogether, and can’t see any difference.

moeller0 | May 29, 2014 at 8:15 pm

For your DSL tests it will be more enlightening if you increase the packet size by 1 byte on each round instead of 100. Currently all ADSL lines (I have seen) run on an ATM link layer which will chop each IP packet into an integer number of 48 byte ATM cells (after adding some per-packet overhead), padding the last cell if need be (its slightly more complicated but that is the gist of it). The consequence of this is that the RTT by packetize plot should show nice quantization steps with a step size of 48 bytes. Since the unavoidable headers typically eat up most of a cell as well you will see larger steps at packet boundaries (typically twice the per ATM cell latency). The caveat with this is that depending on your bandwidth the time increment of each “step” is small enough (1.0239ms at 6656 kbps download and 640 kbps upload, 0.23567 ms at 16402 kbps download, 2558 kbps upload) that you might need a larger sample size than 20 to be able to actually see it. (Typically you can see the quantization best with a more robust estimator like the mimimum or the mean of all data points (per packetize) that fall into the 5 to 95% quantiles).
I note that the time frame and effect size of the ATM quantization would be lost in the variance of your measurements….

Glen Turner | June 9, 2014 at 3:54 pm

I would expect there to be a additional hit per flow establishment in some networks, and most notably in 3G.

Leave a comment!

«

»

Find it!

Theme Design by devolux.org

Tag Cloud