Delta encoding (also called delta compression) is a way to update web pages by sending the “diffs” or deltas between versions of a web page. The server (proxy or origin) sends only what has changed in the page since the last access. This technique greatly reduces the amount of data that is sent from server to browser. In some cases with pages that change slightly, the deltas are on the order of a few TCP/IP packets. As about 32% of page accesses are first-time visits, about 68% of page visits are eligible for delta compression.
There are a few different ways to implement delta encoding (1) by saving old versions of pages and sending differences, (2) by using reference files on the same server that are in the user’s cache, and (3) by using “value-based web caching” that employs blocks of data already sent to the client, independent of file boundaries (Savant 2004).
Same URL Delta Compression
Delta compression for pages at the same URL typically achieves higher compression ratios than other schemes, but has some drawbacks. Sending deltas for the same URL assumes that the client has accessed the page in the past. On the Web, this is only true for 30% of web pages, according one study (Mogul 1997). This method also imposes costs to the origin or proxy server to save old versions of the same page to use as reference files.
Different URL Delta Compression
Delta compression for pages at different URLs typically achieves more modest compression ratios than the same URL method, but does not suffer from the overhead of the same URL method. Improvements of 1.7 times for all pages to 2.9 for eligible text or HTML data have been found over gzip compression (Savant 2004).
Real-World Delta Compression
Although delta encoding is part of the HTTP 1.1 specification (Mogul et al. 2001), it has not been widely adopted among browser and server vendors. However, there are delta compression products that use JavaScript instead. These are generally available through website acceleration appliances. Delta encoding is usually performed as an injected JavaScript that reassembles the differences between a base page and subsequent pages. It can reduce the page load down to a couple TCP packets in some cases, particularly when combined with gzip for text. Cisco and Citrix both offer products that use delta encoding.
Delta Encoding and RSS
While browsers and servers have been slow to adopt delta encoding for web sites, there is one area where it is becoming more popular: RSS newsfeeds. The problem with RSS is that most sites poll feeds for updates. For popular sites, this can add up to a lot of bandwidth (Scoble 2004). Delta encoding was proposed as a temporary solution to reduce the overhead of polling while a push-based model is adopted.
Sites that have adopted delta encoding for RSS newsfeeds report that the average request was reduced by 75% (see Figure 1). Bob Wyman estimates if everyone adopted the RFC3229 protocol for RSS newsfeeds the bandwidth for his now-defunct news aggregation site PubSub.com would have been reduced by two-thirds (Wyman 2004).
 |
RSS feeds that are delta compressed are ~25% the size of requests that are not compressed. |
| Figure 1: Delta Encoding Savings for RSS Feeds (used with permission) |
The Windows RSS platform (Vista) supports this feature, as do a number of other RSS clients. WordPress supports delta encoding of RSS feeds, one option is listed in the references (Robinson 2004). To help conserve bandwidth on the Web ask your blog provider if their software supports delta encoding of RSS.
While delta encoded RSS can save bandwidth, sometimes it can bog down your server. For example, dynamically created feeds like those in WordPress can cause servers to become overloaded during traffic spikes. That is one advantage of Movable Type, it can use static RSS files which scale better under higher loads.
Conclusion
While delta encoding of XHTML, CSS, and JavaScript has not been widely adopted, there is one area where it is catching on: RSS newsfeeds. Delta compression of RSS newsfeeds can save 75% off of file sizes. A number of news aggregators now support delta compression of RSS newsfeeds, making this a viable option for blog-based sites. To delta compress other text-based web content, web site owners can employ acceleration appliances available from Cisco and Citrix.
Further Reading
- King, A., “HTTP Compression,”
- Speed Tweak of the Week, Website Optimization, LLC, Dec. 2003. HTTP compression with gzip saved about 75% on average off of text file size.
- Mogul, J., Krishnamurthy, B., Douglis, F., Feldmann, A., Goland, Y., and A. van Hoff, “Delta encoding in HTTP,”
- RFC 3229, Dec 2001, http://www.ietf.org/rfc/rfc3229.txt.
- Mogul, J. C., Douglis, F., Feldmann, A., and B. Krishnamurthy, “Potential benefits of delta-encoding and data compression for HTTP,”
- In Proc. of the ACM SIGCOMM ’97, Cannes, France, Sept. 1997, 181-196.
- Robinson, J., “RSS and Delta Encoding,”
- The Robinson House, Sept. 2004. Delta encoding code for WordPress by James Robinson.
- Savant, A., and T. Suel, “Server-Friendly Delta Compression for Efficient Web Access,”
- in Proc. of the 8th international workshop on web content caching and distributing, 2004, 303-322.
- Scoble, R., “Full text RSS on MSDN gets turned off,”
- Scobleizer.com, Sept. 8, 2004, http://radio.weblogs.com/0001011/2004/09/08.html#a8195.
- Wyman, B., “Using RFC3229 with Feeds,”
- As I May Think, Sept. 13, 2004, http://www.wyman.us/main/2004/09/using_rfc3229_w.html.
- Wyman, B., “Massive Bandwidth Savings Proven!,”
- As I May Think, Oct. 3, 2004, http://wyman.us/main/2004/10/massive_bandwid.html.