silk/unicode

Explicit Unicode canonical normalization backed by the pinned Unicode 17.0.0 data set.

When to use

Normalize text at a boundary where canonically equivalent spellings must compare alike. Use normalizeNfc for ordinary storage and comparison, or normalizeNfd when a decomposed, canonically ordered sequence is the desired representation.

Details

Normalization is opt-in: string equality continues to compare exact UTF-8 bytes. Both forms allocate a fresh owned String, handle Hangul composition algorithmically, and use the same generated tables on every target. dataVersion reports which Unicode database defines those results.

Gotchas

This module implements canonical NFC and NFD, not compatibility normalization (NFKC or NFKD), locale-sensitive comparison, grapheme segmentation, or case folding.

Examples

Make canonically equivalent text compare equal

import silk.allocator { Allocator }

import silk.effect { Effect }

import silk.string as String

import silk.unicode as Unicode

effect fn normalize() -> i32
! Allocator.OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let composing = Unicode.normalizeNfc("e\u{301}")
    |> Effect.provideMut<Allocator>(&mut allocator)
  let composed = run composing
  let decomposing = Unicode.normalizeNfd("é")
    |> Effect.provideMut<Allocator>(&mut allocator)
  let decomposed = run decomposing
  if String.view(&composed) != "é" {
    return 0
  }
  if String.view(&decomposed) != "e\u{301}" {
    return 0
  }
  return 42
}

effect fn recover(error: Allocator.OutOfMemoryError) -> i32 {
  return 0
}

pub fn main() -> i32 {
  return run Effect.catchAll(normalize(), recover)
}

Import as Unicode with import silk.unicode.

Public declarations: 5.

dataVersion

pub fn dataVersion() -> string

Returns the Unicode version that defines this module's normalization results.

Details

The version is data, not identity: a later database changes what this returns and what the tables contain, and changes neither a compiler type nor a target ABI.

longestDecomposition

pub fn longestDecomposition() -> usize

Returns the longest full canonical decomposition of one scalar in the active Unicode data.

canonicalCombiningClass

pub fn canonicalCombiningClass(scalar: u32) -> u32

Returns a Unicode scalar's canonical combining class, or zero for a starter or unknown value.

Parameter scalar

scalar: u32

Unicode scalar value represented as its unsigned code point.

normalizeNfd

pub effect fn normalizeNfd(value: string) -> String ! OutOfMemoryError ? &mut Allocator

Returns the Normalization Form D of text: fully decomposed, in canonical order.

When to use

Use this function when consumers require decomposed scalars in canonical combining-class order. Use normalizeNfc for ordinary normalized storage and comparison.

Details

The function does not change value. It returns freshly owned UTF-8 text and can allocate. Canonically equivalent inputs produce equal NFD text under the same Unicode data version.

normalizeNfc

pub effect fn normalizeNfc(value: string) -> String ! OutOfMemoryError ? &mut Allocator

Returns the Normalization Form C of text: decomposed, canonically ordered, then recomposed.

When to use

Use this function for ordinary normalized storage and canonical-equivalence comparison. Use normalizeNfd when a consumer requires decomposed scalars.

Details

The function does not change value. It returns freshly owned UTF-8 text and can allocate. Canonically equivalent inputs produce equal NFC text under the same Unicode data version.