Skip to content
Haiku OSDeep Dive Published Updated 7 min readViews unavailable

Reading Haiku HPKG Files: Header, Heap, TOC, and Package Metadata

Decode Haiku HPKG archives safely: inspect the header, chunked heap, TOC, and metadata without confusing package parsing with trust or installation.

Haiku’s .hpkg files are not ordinary directory trees with a different extension. They are structured containers designed to let packagefs expose package contents without first extracting every file into a writable directory. That design makes the on-disk format relevant to package tooling, archive inspection, repository analysis, and incident response.

This article focuses on the binary format, not on installing software or operating packagefs. A well-formed HPKG is not automatically authentic, approved, or safe to install. Parsing metadata for inspection and authorizing an installation are separate decisions.

Three layers share one file

Haiku’s format specification separates HPKG into three conceptual layers:

  1. The data-container format stores structured data and the byte ranges those structures reference.
  2. The archive format describes filesystem entries such as directories, regular files, symbolic links, permissions, timestamps, and file attributes.
  3. The package format extends the archive with package-level metadata such as name, version, architecture, requirements, and licensing.

These layers answer different questions. The outer container explains where encoded data lives. The archive tree says which filesystem objects the package presents. Package attributes describe the software artifact as a package-manager input. Treating all three as a generic compressed tarball loses the indexed structure and the meaning of package attributes.

The container has a fixed-format header followed by a heap. Its table of contents (TOC) and package attributes are stored at the end of the uncompressed heap, in that order. The header records their lengths and the heap’s compressed and uncompressed sizes. Thus a metadata reader can locate structural regions without assuming that file payloads are laid out like a stream of extracted paths.

Read the header as untrusted length metadata

The documented header begins with the four-byte hpkg magic and includes the header size, format version, total file size, minor version, heap compression mode, chunk size, compressed and uncompressed heap lengths, and the sizes of the TOC and package-attribute data and string tables. Multibyte fixed-width numbers use big-endian representation; variable-length values use LEB128 where the format specifies it.

header_size is the absolute offset of the heap. TOC and package-attribute offsets are derived from the uncompressed heap length, not directly from the on-disk compressed byte position. A parser that adds an uncompressed offset to the physical file offset will select the wrong bytes whenever the heap is compressed.

Every length read from an archive must be validated before arithmetic, allocation, decompression, or slicing. For example, a checked-range helper should reject an offset beyond the buffer and compare the requested size against the remaining length rather than calculating offset + size first:

def checked_slice(data: bytes, offset: int, size: int) -> bytes:
    if offset < 0 or size < 0:
        raise ValueError("negative archive range")
    if offset > len(data) or size > len(data) - offset:
        raise ValueError("archive range is outside the input")
    return data[offset:offset + size]

This helper is only a bounds-checking primitive; it does not parse HPKG or establish authenticity. A full reader must also validate the magic, supported format version, declared total size against the actual input, every section boundary, and the relationship among compressed and uncompressed lengths. Avoid trusting a header field simply because its integer decoded successfully.

The heap is chunked, and references use its uncompressed view

The heap concatenates byte sequences without padding or separators. TOC entries and attributes can refer to those bytes by an offset and length. Those offsets always address the uncompressed heap, even when the stored heap is compressed.

The specification divides the uncompressed heap into 64-KiB chunks. With zlib compression enabled, each chunk can be stored compressed when compression saves space or stored as-is when it does not. A table of 16-bit compressed chunk sizes is present in the compressed representation; the final chunk’s uncompressed length follows from the total uncompressed heap size, while its compressed-size entry is omitted because it is implied by the remaining compressed-size accounting. With no compression, the chunk-size table is omitted. The TOC and package attributes themselves are part of the heap and occupy its end.

This layout supports bounded access to regions instead of requiring a single monolithic decompression operation, but it does not make decompression automatically safe. A reader should cap total uncompressed bytes, bound per-chunk and cumulative work, reject impossible chunk tables, and ensure a decoder produces the expected output size. These are defensive parser rules for processing untrusted files, not claims that the HPKG specification provides a security sandbox.

An important offset invariant is:

package-attributes-start = uncompressed-heap-size - attributes-length
toc-start                = uncompressed-heap-size - attributes-length - toc-length

The expressions describe offsets inside the uncompressed heap. Validate each subtraction first: a section length greater than the available heap is malformed input, not a value to wrap or clamp.

The TOC is a typed attribute tree

The TOC contains attribute entries organized as trees. Each entry has an ID, a data type, a value, and optionally child entries. Data types include signed and unsigned integers, UTF-8 strings, and raw bytes. Values may be inline, refer to the shared-string table, or point into the heap, depending on their type and encoding.

An unsigned LEB128 attribute tag packs the value encoding, whether children follow, the data type, and the attribute ID. A tree ends with a zero byte. The string table contains null-terminated UTF-8 strings and is itself terminated. Those sentinels make it essential for a parser to check that termination occurs before the relevant section boundary; scanning past a boundary can reinterpret payload bytes as metadata.

The archive layer uses directory-entry attributes recursively. An entry’s name is one filename component, not a path containing arbitrary separators. A directory can contain child entries; a regular file can reference raw data in the heap; a symbolic link stores a target path. Extended file attributes are represented as named attributes with a type and associated data. Correct tooling preserves these distinctions instead of flattening them into a host filesystem without an explicit policy for permissions, links, timestamps, and Haiku-specific attributes.

Package metadata is not file content

The package-attribute section uses the same general tree structure but a different set of attribute IDs. It describes the package itself: identity, version components, architecture, summary, license, dependencies, provided capabilities, conflicts, and other package-management declarations. The archive TOC describes entries supplied by the package; the package attributes describe how a package manager should understand the artifact.

That distinction matters when building inventory tools. A package name and version are declarations in the file, not a cryptographic identity. A parser should report what the package says, the digest of the exact file that was inspected, and the repository or acquisition context as separate fields. Do not turn a parsed vendor string, source URL, or package name into proof of publisher identity.

A safe inspection workflow

For an inventory or forensic reader, keep the parser and the installer separate:

  1. Preserve the original file and record its byte length and a cryptographic digest.
  2. Parse only enough header data to establish bounded section ranges; reject inconsistent lengths and unsupported major versions.
  3. Decode the heap with strict output and work limits. Validate every chunk table entry and require each decoded chunk to match its expected size.
  4. Parse TOC and package attributes within their own section boundaries. Bound recursion depth, number of nodes, string length, and aggregate raw data references.
  5. Render paths as data first. Reject traversal, absolute-path, duplicate-entry, or platform-conflicting cases before any extraction operation.
  6. Report package claims and file contents without treating successful parsing as repository authorization or execution approval.

If the goal is to install or extract software, prefer Haiku’s supported package tools and the trust policy for the specific repository and system. Reimplementing the format is justified for specialized tooling, but it creates a second parser that must handle future minor attributes, malformed lengths, compressed chunks, symbolic links, and extended attributes correctly.

HPKG’s compact, indexed structure explains how packagefs can present files from intact package images. The format is also a reminder that efficient random access and safe parsing are different properties: every offset, length, tree, and decompression result still needs a strict boundary.

Related:

Sources:

Comments