silk/unicode_tables

Generated canonical-normalization data for the silk.unicode public façade.

When to use

Application code should normally call silk.unicode rather than this backing module. These declarations are public so the ordinary Silk normalization implementation can read the pinned data without compiler privilege.

Details

The tables come from Unicode 17.0.0, are packed as big-endian byte strings, and are searched with ordinary Silk functions. Regeneration may change their representation and record layout; the compiler gives neither this module nor its declaration names special treatment.

Gotchas

A zero lookup result is a sentinel, not Unicode scalar U+0000. Hangul decomposition and composition are algorithmic and are not present in these tables.

Examples

Read the canonical decomposition of é

import silk.unicode_tables as UnicodeTables

pub fn main() -> i32 {
  let decomposition = UnicodeTables.canonicalDecomposition(233)
  if decomposition.first != 101 || decomposition.second != 769 {
    return 0
  }
  return 42
}

Import as UnicodeTables with import silk.unicode_tables.

Public declarations: 7.

Decomposition

pub struct Decomposition

One canonical decomposition with zero sentinels for an absent mapping or second scalar.

Field first

pub first: u32

The first mapped scalar, or zero when the input has no table mapping.

Field second

pub second: u32

The second mapped scalar, or zero when the mapping contains one scalar.

PairRecord

pub struct PairRecord

One two-scalar canonical decomposition and its composed scalar.

Field composed

pub composed: u32

The scalar whose canonical mapping is this pair.

Field first

pub first: u32

First scalar of the canonical mapping.

Field second

pub second: u32

Second scalar of the canonical mapping.

dataVersion

pub fn dataVersion() -> string

Returns the Unicode data version that defines all lookup results in this module.

maximumDecompositionLength

pub fn maximumDecompositionLength() -> usize

Returns the longest full canonical decomposition of one scalar in these tables.

combiningClass

pub fn combiningClass(scalar: u32) -> u32

Returns a scalar's canonical combining class, or zero for a starter or unknown value.

canonicalDecomposition

pub fn canonicalDecomposition(scalar: u32) -> Decomposition

Returns the canonical decomposition of a scalar, or both components zero when it has none. Hangul syllables decompose algorithmically and are absent from this table by design.

canonicalComposition

pub fn canonicalComposition(first: u32, second: u32) -> u32

Composes a starter and a following scalar, or returns zero when the pair does not compose.

Details

Excluded composition pairs are absent. A nonzero result is always a primary composite.

Gotchas

Hangul composition is algorithmic and is not included in this lookup.