Haiku BUnicodeChar: UTF-8 Validation, Classification, and Character Boundaries

BUnicodeChar provides static Unicode classification, simple casing, and UTF-8 conversion helpers in Haiku’s Locale Kit. Its public methods generally accept a uint32 code point rather than a UTF-8 byte. The class is useful after decoding, but its convenience decoding methods are not a validation boundary for arbitrary network or file content.

The distinction is visible in Haiku’s implementation: classification delegates to ICU, while FromUTF8 and ToUTF8 use ICU macros whose names explicitly include UNSAFE. An application importing untrusted bytes should establish length and encoding validity first. Calling a character helper and receiving an integer does not establish that the original sequence was well-formed or wholly contained within the supplied buffer.

Keep bytes, scalars, and graphemes separate

A UTF-8 code point occupies one to four bytes. A Unicode scalar excludes surrogate values and lies within the Unicode code-point range. A visible text element may consist of several scalars, such as a base letter followed by a combining mark or an emoji sequence joined by special characters.

Consequently, a byte offset, a scalar count, and a grapheme-cluster index describe different positions. BUnicodeChar classifies individual code points. It does not implement the grapheme segmentation rules needed for user-facing cursor movement or deletion. Use an appropriate Unicode segmentation implementation when the interface promises operations on user-perceived characters.

Write units into variable names and serialized schemas. A maximum of 100 UTF-8 bytes and a maximum of 100 displayed characters are different product requirements. An import path should retain the byte length supplied by the file or protocol rather than recomputing it with strlen, which stops at an embedded NUL and cannot describe arbitrary binary buffers.

Validate with an explicit byte boundary

In the inspected upstream source, FromUTF8(const char**) invokes U8_NEXT_UNSAFE, advances the supplied pointer, and returns the code point. It has no end pointer or byte-length argument. A truncated multibyte sequence can therefore cause a read beyond the intended data boundary. The overload taking const char* delegates through a local pointer and does not add validation.

ICU’s bounded U8_NEXT macro accepts a byte length and reports malformed input with a negative code point. The following helper rejects invalid UTF-8 and embedded NUL because its intended result will subsequently be treated as NUL-terminated application text. Rejecting NUL is an application policy, not a requirement of UTF-8 itself.

#include <unicode/utf8.h>
#include <climits>
#include <cstddef>
#include <cstdint>

bool ValidateCStringUTF8(const char* bytes, size_t byteLength) {
    if ((bytes == nullptr && byteLength != 0) || byteLength > INT32_MAX)
        return false;

    int32_t offset = 0;
    const int32_t length = static_cast<int32_t>(byteLength);
    const auto* input = reinterpret_cast<const uint8_t*>(bytes);
    while (offset < length) {
        UChar32 codePoint;
        U8_NEXT(input, offset, length, codePoint);
        if (codePoint < 0 || codePoint == 0)
            return false;
    }
    return true;
}

The helper does not supply a terminator or copy the data. Before passing the bytes to a NUL-terminated API, copy them into storage with space for an explicit trailing zero. Keep the validated storage immutable while iterating; validation of one buffer does not authorize decoding a later mutated version of it.

This approach uses ICU directly, so include and link against the ICU development environment appropriate to the target build. Do not hard-code an ICU library filename from another system or assume that an arbitrary Haiku image has the same ICU release as a development machine. The BUnicodeChar API itself belongs to Haiku’s main library, but direct ICU calls have their own build dependency.

Encoding also needs preconditions

ToUTF8(uint32, char**) uses U8_APPEND_UNSAFE and advances the output pointer. It does not accept buffer capacity or append a C-string terminator. Validate that the value is at most 0x10ffff and is not in 0xd800..0xdfff, then reserve up to four output bytes plus a terminator when producing a C string.

#include <UnicodeChar.h>

bool EncodeScalar(uint32 codePoint, char (&output)[5]) {
    if (codePoint > 0x10ffff ||
        (codePoint >= 0xd800 && codePoint <= 0xdfff)) {
        return false;
    }
    char* cursor = output;
    BUnicodeChar::ToUTF8(codePoint, &cursor);
    *cursor = '\0';
    return true;
}

Encoding scalar zero produces a NUL byte, which is valid Unicode but may be unusable as a visible C-string character. Apply the format’s own policy separately. For encoding a whole string into a limited destination, track remaining capacity before every append or use ICU’s checked append facilities rather than assuming every code point is one byte.

The length limit is not a buffer limit

UTF8StringLength(string) walks a NUL-terminated string with FromUTF8. Its second overload stops after maxLength decoded characters or a NUL terminator. The implementation compares the decoded count against that limit; it does not treat maxLength as the maximum number of readable bytes.

That distinction makes the overload unsuitable as protection for a nonterminated packet buffer. Even a limit of one character permits the decoder to access several bytes for a multibyte sequence. Use a bounded decoder for protocol data, and use this convenience length operation only under its valid, terminated-string preconditions.

Classification names encode specific Unicode policies

IsSpace delegates to ICU’s Java-space-character predicate, which recognizes separator categories. IsWhitespace uses ICU’s Java-style whitespace predicate, including certain controls while excluding designated nonbreaking spaces. They therefore are not interchangeable trim rules.

For example, a tab can satisfy IsWhitespace without satisfying IsSpace; a nonbreaking space can satisfy IsSpace without satisfying IsWhitespace. Write fixture tests for those distinctions before using either predicate to tokenize configuration input. Protocol grammars that specify ASCII spaces or tabs should implement those exact characters rather than expanding the grammar with every Unicode whitespace-like character.

Similarly, IsDigit recognizes Unicode decimal digits, not only the ASCII bytes 0 through 9. DigitValue delegates to ICU’s decimal digit conversion. A Unicode-friendly input widget and an ASCII wire-format parser may legitimately need different policies. Classification does not determine whether a code point is permitted in an identifier, safe in a filename, or visually distinguishable from another character.

Simple casing is not full text casing

ToLower, ToUpper, and ToTitle delegate to ICU single-code-point mappings. Their return type can represent one code point only. Full casing of strings can expand output, depend on neighboring characters, and vary by language. Case-insensitive identifier matching also needs a defined folding policy, not a loop that blindly lowercases each code point.

Use appropriate string casing, folding, normalization, or collation services for the actual feature. Keep display formatting distinct from stored identity. A name rendered in uppercase for a heading should not silently replace the original bytes used as its persistent key.

Acceptance tests and recovery

Test ASCII, multibyte scalars, combining sequences, supplementary characters, embedded NUL, truncated sequences, invalid continuation bytes, overlong encodings, surrogates, and values beyond the scalar range. Test whitespace pairs and non-ASCII decimal digits separately from decoding tests.

For malformed input, choose an explicit rejection or replacement policy at the boundary. Preserve a safe error offset and format context without printing private content. Do not attempt recovery by advancing an unsafe decoder one byte at a time. Once text is validated, retain its byte boundaries and select the correct layer for classification, string transformation, or grapheme processing.

Related:

Sources: