Uncompressed tar file with AVI files: 3206 GB
gzip -9: 3179 GB
bzip2 -9: 3178 GB
xz: 3179 GB
xz -9: 3175 GB
7z -mx=9: 3180 GB
lrzip -M: 2829 GB
lrzip -l: 2840 GB
lrzip -n: 2852 GB

The times were over 40 min. for bzip2, xz and 7z, 22 min. for lrzip -M, less than 5 min. for lrzip -l and less than 2 min. for lrzip -n.

The issue of 32-bit rzip and 900MB was simply the limitations of machines at the time Tridge was doing his thesis. 64 bit is the obvious thing to do these days, as well as much larger window. If I get cycles (ha!) I’d like to revisit it entirely: the table of backrefs and literal sizes should be compressed separately from the data itself, and rzip should never emit matches which are within the range of the backend. Also, LZMA2 should be used (that’s what xz uses I’m told).

For the current kernel sources, xz wins, making it a good choice for the kernel.org mirrors. But an rzip variant should still be able to do at least as well, and better as we get larger.

lrzip only becomes useful once the simple long distance dictionary rzip stage is significantly larger than the compression window of the backend (lzma). I doubt you’ll find the rzip first stage helps the kernel source at its current size when lzma’s compression windows are the size they already are. You’ll only start getting a benefit with a much larger codebase… which will be in not so long for linux kernel :P . If you see the benchmarks I posted, lrzip only has an advantage with massive files (and massive ram) when there is long distance redundancy. So if you start including multiple trees instead of a single kernel version, you’ll start seeing a benefit from the rzip stage. By a (not so strange) coincidence I also used kernel trees to test it. http://ck.kolivas.org/apps/lrzip/README.benchmarks

As for bugs in rzip, I found it spat the dummy on greater than 32bit signed in multiple ways and suspect that’s why it had the artificial 900MB limit. Dealing with 32bit, mmap, write() limitations and so on has been a bit of a pain when trying to scale to 64bit sizes I have to admit. That’s been the bulk of the work that went into lrzip. In the process it actually made the compression a little less efficient and slower than rzip due to the 64bit offsets in the archives. xz’s performance on massive files was not so great, being both much slower than 7z (due to not being multithreaded) and not even compressing as well as it. I often wonder if the ground swell for xz on linux is well founded, but then there’s always more to software selection than merit on purely technical grounds. By that I don’t just mean politically…

OK, now the results for each compressor (-9 –to-stdout or -M for lrzip):

gzip /tmp/linux.2.6.tar: 86388655
gzip /tmp/linux.2.6.sorted.tar: 84904755
bzip2 /tmp/linux.2.6.tar: 67350827
bzip2 /tmp/linux.2.6.sorted.tar: 66784160
xz /tmp/linux.2.6.tar: 54614828
xz /tmp/linux.2.6.sorted.tar: 55100332
lrzip /tmp/linux.2.6.tar: 56500239
lrzip /tmp/linux.2.6.sorted.tar: 56104936

The xz result is flat-out weird; AFAICT any attempt to help xz with compression makes it worse. I wonder if it was actually optimized using the linux kernel source?

(PS. Added comment preview, since it was annoying me).

Tar the files sorted by file extension instead of file name (if you don’t already).

When we did this for the Strawberry Perl MSI installer, we saw a 10% reduction in file size, on the assumption that files of a similar file extension probably have similar content, and thus grouping them will feed the compression algorithm better.