Use Zopfli Compression in your CDN

#google

#azure

#gzip

#deflate

Several months ago Google released Zopfli, which is a slow (~100 times slower than normal deflate) a compression algorithm that achieves about 4-8% better compression.

Zopfli has also been adapted to be used in PNG compression:

ZopfliPNG will do nothing more that read a png file, re-compress the DEFLATED parts (IDAT image data, and if the tool matures other compressed chunks like iTXt, zTXt, and iCCP) and write the modified file.

When should we use this? The best place to use Zopfli is on your static content, so we choose to use it on our Azure CDN.

Zopfli is a compression algorithm that is compatible with the DEFLATE algorithm used in zlib, allowing it to be used seamlessly with already deployed programs and devices that support the standard. Zopfli produces files that are 4-8% smaller than zlib at the expense of being substantially slower to compress a file than other implementations of the DEFLATE algorithm.

Perfect fit, since CDN files are only uploaded once and compression only needs to happen on upload. Clients downloading compressed files will decompress them using the normal DEFLATE supported everywhere at the same speed (or faster, since they will be smaller in size).

Our first step was to get Zopfli into C#. So we created a wrapper and released it under Apache 2.0

https://github.com/echovoice/libzopfli-sharp

Nuget Package is also found here: https://www.nuget.org/packages/libzopfli-sharp

PNG Compression Usage

============================

If you are working with .Net Image objects simply call SaveAsPNG() Image extension method.

Image testImage = Image.FromFile("files/ev.png");
testImage.SaveAsPNG(path_to_save_compressed_PNG);

You can compress *.PNG files directly using the ZopfliPNG.compress() method.

string path = "files/ev.png";
ZopfliPNG.compress(path);

We also implemented a derived class of Stream called ZopfliPNGStream

byte[] uncompressed = File.ReadAllBytes("files/ev.png");
int before = uncompressed.Length;
byte[] compressed;
int after = 0;

using (MemoryStream compressStream = new MemoryStream())
using (ZopfliPNGStream compressor = new ZopfliPNGStream(compressStream))
{
    compressor.Write(uncompressed, 0, before);
    compressor.Close();
    compressed = compressStream.ToArray();
    after = compressed.Length;
}

In addition to using the default compression options Zopfli exposes some additional options to fine tune compression.
We extended this in the ZopfliPNGOptions object.

public class ZopfliPNGOptions
{
    // Allow altering hidden colors of fully transparent pixels
    public Boolean lossy_transparent;

    // Convert 16-bit per channel images to 8-bit per channel
    public Boolean lossy_8bit;

    // Filter strategies to try
    public ZopfliPNGFilterStrategy[] filter_strategies;

    // Automatically choose filter strategy using less good compression
    public Boolean auto_filter_strategy;

    // PNG chunks to keep
    // chunks to literally copy over from the original PNG to the resulting one
    public String[] keepchunks;

    // Use Zopfli deflate compression
    public Boolean use_zopfli;

    // Zopfli number of iterations
    public Int32 num_iterations;

    // Zopfli number of iterations on large images
    public Int32 num_iterations_large;

    // 0=none, 1=first, 2=last, 3=both
    public Int32 block_split_strategy;
}

Gzip, Deflate and Zlib Compression Usage

============================

For all 3 compression types we implemented a derived class of Stream ZopfliStream

byte[] uncompressed = File.ReadAllBytes("files/fp.log");
int before = uncompressed.Length;
byte[] compressed;
int after = 0;

using (MemoryStream compressStream = new MemoryStream())
using (ZopfliStream compressor = new ZopfliStream(compressStream, ZopfliFormat.ZOPFLI_FORMAT_DEFLATE))
{
    compressor.Write(uncompressed, 0, before);
    compressor.Close();
    compressed = compressStream.ToArray();
    after = compressed.Length;
}

The second parameter for our derived Stream class is the type of compression to use.

public enum ZopfliFormat
{
    ZOPFLI_FORMAT_GZIP,
    ZOPFLI_FORMAT_ZLIB,
    ZOPFLI_FORMAT_DEFLATE
};

In addition to using the default options Zopfli exposes some additional options used to fine tune compression.
We extended this in the ZopfliOptions object which can also be passed into the Stream.

public class ZopfliOptions
{
    // Whether to print output
    public Int32 verbose;

    // Whether to print more detailed output
    public Int32 verbose_more;

    // Maximum amount of times to rerun forward and backward pass to optimize LZ77
    // compression cost. Good values: 10, 15 for small files, 5 for files over
    // several MB in size or it will be too slow.
    public Int32 numiterations;

    // If true, splits the data in multiple deflate blocks with optimal choice
    // for the block boundaries. Block splitting gives better compression. Default:
    // true (1).
    public Int32 blocksplitting;

    // If true, chooses the optimal block split points only after doing the iterative
    // LZ77 compression. If false, chooses the block split points first, then does
    // iterative LZ77 on each individual block. Depending on the file, either first
    // or last gives the best compression. Default: false (0).
    public Int32 blocksplittinglast;

    // Maximum amount of blocks to split into (0 for unlimited, but this can give
    // extreme results that hurt compression on some files). Default value: 15.
    public Int32 blocksplittingmax;
}

Where to go from here?

Now that we have Zopfli working in C# you will want to wrap any of your Azure Blob uploads with the Zopfli stream.

// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");

// Retrieve reference to a blob named "myblob".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob");

// Create or overwrite the "myblob" blob with contents from a local file.
byte[] uncompressed = File.ReadAllBytes(@"path\myfile");
int before = uncompressed.Length;
byte[] compressed;

// test deflate stream compression code
using (MemoryStream compressStream = new MemoryStream())
using (ZopfliStream compressor = new ZopfliStream(compressStream,ZopfliFormat.ZOPFLI_FORMAT_DEFLATE))
{
    compressor.Write(uncompressed, 0, before);
    compressor.Close();
    blockBlob.UploadFromStream(compressStream);
}

from MSDN
http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/

MSDN has a great write up on enabling the CDN from a Blob.

http://www.windowsazure.com/en-us/develop/net/common-tasks/cdn/

If you still need help uploading to Azure or how to determine if compression is supported I would suggest viewing this (slightly outdated) tutorial: