DEFLATE: One Stream Format Behind ZIP, zlib, and gzip
Trace DEFLATE across ZIP method 8 and the 1996 RFCs, distinguishing its raw stream from zlib checksums, gzip members, and archive containers.
A .zip archive, a .gz file, and a zlib stream can all contain DEFLATE-compressed data, but they are not the same format. DEFLATE is the compression method. ZIP assigns that method an identifier inside a multi-file archive; zlib and gzip put different framing and integrity data around compressed streams. The history is a story of one algorithm being specified at several layers, not one file extension replacing another.
The primary documents make that layering visible: PKWARE’s ZIP specification records method 8 as “Deflated,” while three May 1996 RFCs describe the DEFLATE bitstream, the zlib wrapper, and the gzip file format separately.
A method inside a container
PKWARE says its ZIP format specification was first published in 1989 and describes ZIP as a cross-platform file storage and transfer format. Its current APPNOTE gives each archive entry a compression-method field: method 0 means stored without compression, method 8 means Deflated, and other identifiers denote other methods. The field belongs to ZIP’s per-entry metadata; it is not a standalone archive format named “DEFLATE.”
That distinction explains why a ZIP can contain stored entries alongside compressed ones. The archive still needs local file headers, a central directory, filenames, lengths, checksums, and other metadata. The compressed bytes for one method-8 entry are only the payload for that entry. A raw DEFLATE stream does not, by itself, supply ZIP’s archive directory or its per-file names.
The current APPNOTE also lists version 2.0 as the version needed to extract an entry using Deflate. That is a compatibility field in the specification; it should not be mistaken for evidence that ZIP itself began in version 2.0 or that the format’s first publication occurred then.
RFC 1951 specifies the algorithmic stream
RFC 1951, published as an Informational RFC in May 1996, defines DEFLATE as a lossless stream format combining LZ77 references with Huffman coding. It describes a series of blocks. Within a block, data can be represented as literal bytes or as a length and backward-distance pair that points into previously produced output. The distance can reach 32 KiB back, and a match can encode up to 258 bytes.
Huffman codes represent literal/length and distance symbols compactly. A block can use fixed codes, dynamic codes described in the stream, or a stored form with no Huffman compression. The encoder can choose an appropriate block type for its input; therefore two valid compressors can encode the same bytes differently while producing streams that conform to the same specification.
RFC 1951 also clarifies its own chronology: version 1.3 was a conversion of the existing specification to RFC style, with no technical changes since version 1.1. The RFC is a public specification milestone, not a claim that DEFLATE was invented in 1996. It says the defined stream is compatible with the then-current gzip utility, an interoperability goal that makes sense only because the format already had implementations.
Wrappers and archives add their own contracts
| Layer | What it frames | Integrity or metadata |
|---|---|---|
| Raw DEFLATE | RFC 1951 blocks only | No wrapper checksum or filename |
| zlib stream | DEFLATE plus a compact header/trailer | Adler-32; optional preset-dictionary identifier |
| gzip member | DEFLATE inside a file-oriented member | Header fields, CRC-32, and uncompressed size |
| ZIP archive | Per-entry stored or method-selected payloads | Filenames, sizes, CRC-32, flags, and a central directory |
RFC 1950 specifies the zlib wrapper, whose method field identifies DEFLATE and whose trailer contains Adler-32. RFC 1952 specifies gzip as a file format: each member has a header, compressed blocks, and a trailer with CRC-32 and the input size modulo 2^32. ZIP uses its own archive records and CRC field. These checksums and framing rules are not interchangeable, even when the compressed algorithm underneath is the same.
A practical consequence is that a decoder must know which layer it is reading. Passing raw DEFLATE bytes to a zlib decoder can produce a header error; treating a ZIP file as if it began with a single gzip stream ignores its archive directory. Extensions are hints for people, not a substitute for parsing the actual format.
Inspect the distinction with Python’s zlib module
Python exposes the wrapper choice through wbits. This small in-memory example round-trips one fixed byte string in raw, zlib, and gzip forms. It requires Python 3.11 or later for the keyword wbits argument to zlib.compress(), and a Python build that includes the optional zlib module:
import zlib
payload = (b"DEFLATE is an algorithm; wrappers are formats.\n") * 8
for label, wbits in (
("raw", -zlib.MAX_WBITS),
("zlib", zlib.MAX_WBITS),
("gzip", zlib.MAX_WBITS | 16),
):
packed = zlib.compress(payload, wbits=wbits)
assert zlib.decompress(packed, wbits=wbits) == payload
print(f"{label}: {len(packed)} bytes; round trip OK")The negative window size selects raw DEFLATE; the default positive window size selects zlib framing; adding 16 selects gzip framing. The exact compressed lengths can vary with the Python and zlib versions, so use the assertion as the verification, not a hard-coded byte count. This example uses a short trusted constant, not untrusted input. A service that accepts user-controlled compressed data should use streaming decompression and enforce an output limit; a small compressed input can expand substantially.
For interoperability checks, keep the producer and consumer’s wrapper explicit. Python’s zlib.decompress() accepts a wbits parameter for this reason. For a ZIP archive, use a ZIP reader that interprets its central directory and entry method instead of stripping bytes and guessing which DEFLATE flavor a file contains.
What the documents do and do not prove
The RFCs establish a useful publication sequence: in May 1996, IETF’s RFC series captured the DEFLATE stream and two wrappers as separate documents. PKWARE’s living APPNOTE preserves ZIP’s container-level method registry and says its specification dates to 1989. The documents show how interoperability was formalized across tools and formats without collapsing them into one file type.
They do not establish every detail of the algorithm’s earlier commercial history. RFC 1951 acknowledges earlier versions but does not identify every implementation or release date. The current ZIP APPNOTE labels method 8 and lists the version needed to extract, but those fields alone do not prove the precise date when a particular PKZIP release first shipped Deflate. Unless a contemporaneous release note or manual is in hand, it is safer to distinguish the documented standard timeline from claims about product launches.
DEFLATE’s longevity comes from a small, reusable stream format that could sit inside different containers. ZIP made it one archive-entry method; gzip added file-oriented metadata and checksums; zlib made a compact stream wrapper for applications. When a compatibility bug appears, first identify which layer disagrees: the block stream, its wrapper, or the outer archive. Only then consider changing the compression algorithm.
Related:
- GIF: The Palette-Based Image Format That Became an Animation Container
- How to Read and Understand Old Internet RFCs
Sources: