Lexical form

Silk source is a byte sequence that the lexer divides into identifiers, keywords, literals, punctuation, comments, whitespace, and invalid regions. This page defines the source spellings that are meaningful before parsing begins. The syntactic meaning of those tokens belongs to the other reference pages.

LEXICAL-001 — Identifiers use the ASCII identifier alphabet

Status: Confirmed

An identifier starts with an ASCII letter or _ and continues with ASCII letters, decimal digits, and _. Matching is case-sensitive. _ is an identifier token; it gains its wildcard meaning only in a pattern position.

fn valid_name2(_value: i32) -> i32 {
  return _value
}

Boundary: Non-ASCII letters are not identifier characters. A source byte that begins no valid token belongs to one maximal unsupported-byte region.

Diagnostics: An unsupported byte region reports LEX0001 and covers the entire adjacent region rather than emitting one error per byte.

Evidence: lexer specification, identifier byte classes, lexer tests.

LEXICAL-002 — The keyword vocabulary is closed

Status: Confirmed

The following complete identifiers are keywords:

as break const continue drop effect else enum fail false fn for if impl import
interface let match move mut once pub return role run service struct true unsafe while

Keyword recognition applies only to a complete identifier. letter, matcher, and services are identifiers, not a keyword followed by a suffix.

Some grammar positions also give a contextual meaning to an ordinary identifier, such as where in generic constraints. A contextual word remains an identifier token and is not added to the closed lexical keyword vocabulary.

Boundary: Keyword spelling is lowercase and case-sensitive. A reserved keyword cannot be used as an identifier merely because the surrounding grammar would otherwise make its role clear.

Diagnostics: Keyword misuse is a syntax error at the position where the grammar requires a different token. There is no separate lexical diagnostic for a correctly spelled keyword.

Evidence: keyword table, token catalog, complete-identifier tests.

LEXICAL-003 — Line comments end at the physical line boundary

Status: Confirmed

// begins an ordinary comment. /// begins a documentation comment attached to the following declaration, and //! begins documentation for the containing module. Consecutive documentation lines retain their source order.

//! Describes this module.

/// Returns the supplied value.
fn identity(value: i32) -> i32 {
  return value
}

A blank line or an intervening ordinary comment separates a declaration from a preceding /// block. Documentation attachment and which declarations may receive comments are defined by the doc comment style guide.

Boundary: Silk has no block-comment token. Comment markers inside a static literal are literal content, and quote characters inside a comment do not begin literals.

Diagnostics: Comments themselves do not produce a diagnostic. Documentation comments that do not attach at a supported declaration position remain trivia and do not become API documentation.

Evidence: lexer specification, documentation attachment tests, lexer tests.

LEXICAL-004 — Numeric literals have explicit base, separator, and sign rules

Status: Confirmed

An integer literal is decimal by default. Prefixes 0b, 0o, and 0x select binary, octal, and hexadecimal digits; the base letter may be uppercase. _ may separate digits only when a valid digit of the same run appears on both sides.

A floating literal contains a decimal point, an exponent, or both. Its exponent uses e or E and may begin with + or -. Decimal digit separators follow the same between-digits rule.

fn values() -> f64 {
  let decimal = 1_000_000
  let binary = 0b1010_0000
  let octal = 0o777
  let hexadecimal = 0xff_ff
  let scaled = 1_000.5e-2
  return scaled
}

The leading - of a negative value is a prefix operator, not part of the literal token. Literal type selection and range checking are defined by values and types.

Boundary: A base prefix must be followed by a digit of that base. _1, 1_, 1__0, 0x_ff, and an exponent with no digits are invalid numeric spellings.

Diagnostics: A base prefix without digits reports LEX0004; an invalid separator reports LEX0005; and an exponent without digits reports LEX0006.

Evidence: numeric lexer specification, integer lexer tests, floating-point tests.

LEXICAL-005 — Static text and byte literals have a closed form vocabulary

Status: Confirmed

Silk recognizes six quote-delimited text and byte forms. A modifier must touch its opening delimiter.

The triple delimiter permits physical line endings in the body. The single delimiter ends at a physical line ending when no closing quote appears first. Raw text treats every backslash as an ordinary backslash; it still must contain valid UTF-8. Escaped text and byte bodies recognize \n, \r, \t, \0, \", \\, \xNN, and \u{...}. \x requires exactly two hexadecimal digits, while \u{...} must denote one Unicode scalar.

fn escaped() -> string {
  return "line one\nline two"
}

fn raw() -> string {
  return r"line one\nline two"
}

fn bytes() -> &[u8] {
  return b"Silk\x00"
}

The literal types, text equality, ownership, and conversion rules are defined by values and types.

Boundary: Literal modifiers form a closed vocabulary. An identifier-like modifier adjacent to a quote, such as future"value", is reserved and invalid rather than an identifier followed by a text literal. There is no raw byte-string form.

Diagnostics: An unknown modifier reports LEX0002; an absent closing delimiter reports LEX0003; malformed escapes and invalid decoded literal data report the corresponding static-data diagnostic without publishing a partial value.

Evidence: literal-form catalog, literal-form tests, raw-string acceptance tests, static text specification.

LEXICAL-006 — A character literal denotes exactly one Unicode scalar

Status: Confirmed

A character literal is delimited by ' and uses the escaped-body policy. Its decoded body must contain exactly one Unicode scalar, irrespective of that scalar's UTF-8 byte width.

const latinSmallE: char = 'é'
const snowman: char = '\u{2603}'
const apostrophe: char = '\''

Character literals recognize the escaped text vocabulary plus \' for their delimiter. Their value and type behavior is defined by CHAR-001.

Boundary: '', 'ab', an invalid scalar escape, and a character body that reaches the line ending without a closing apostrophe are invalid. b'a' is not a byte-character form: b is an identifier followed by a character literal.

Diagnostics: Zero or multiple decoded scalars report LEX0007. An absent closing delimiter reports LEX0003; malformed escapes receive their literal diagnostic.

Evidence: literal-form catalog, character scalar tests, literal-form tests.

LEXICAL-007 — Tokenization is longest and lossless

Status: Confirmed

At each byte position, the lexer recognizes the longest committed token introduction. Compound punctuation such as ==, !=, <=, >=, &&, ||, |>, =>, ->, and .. is therefore one token rather than two adjacent tokens. Whitespace and comments remain explicit trivia so tools can reconstruct the original source.

Boundary: Longest recognition does not invent a token outside the closed vocabulary. Invalid bytes remain covered by explicit invalid tokens, and unknown literal modifiers remain invalid even when their prefix resembles a supported modifier.

Diagnostics: Unsupported byte regions report LEX0001. Other malformed token introductions use their specific lexical diagnostic.

Evidence: lexer implementation, lossless token model, lexer tests.