Spec feedback

Issues, ambiguities, and inconsistencies found in the TSON spec while building this implementation. See CLAUDE.md for why this file exists and when to add to it. Spec quotes below are from 2026 Revision 33 — Part 1 (https://tson.io/raw/2026/33/tson-part1-data.md) unless noted otherwise.

Format per entry: spec section, the problem, the interpretation this implementation chose, and a suggested resolution where there is one.

This register holds what is open against the current revision, and it renumbers from #1 each time a revision closes. It is an input to the next revision’s adjudication, so its numbering is the numbering that revision’s change log will answer against — a stable index of the open set, not an archive of everything ever raised.

The seventeen below are what Revision 33 leaves open, renumbered from #1; the 55 raised against Revision 32 that it resolved are gone from here, because the spec now carries their rules and that is where the answer belongs. This file is the as-built record, not a pointer to one: where an entry proposes a design this implementation has built, the entry states the design, what is running, and what is not, so that a reviewer editing the spec needs nothing beside it. Cite the spec, not the argument that got it there: docs/ and the Javadoc name the section that requires a behaviour, and a SPEC-FEEDBACK.md #N citation is for an entry below, where there is no section to point at yet. When an entry closes, its citations become spec citations and the entry is deleted — nothing here is an archive.

1. Does !duration accept ISO 8601’s PnW week form, or only PnYnMnDTnHnMnS?

Section: §5.4.

Problem: §5.4’s table gives !duration’s format as “ISO 8601 duration (PnYnMnDTnHnMnS)” — a parenthetical showing one specific designator sequence. ISO 8601-1:2019 (the spec duration_type itself pins to, per meta.tn’s spec field) also defines a second, mutually-exclusive alternative form for expressing a duration in whole weeks: PnW (e.g. P3W for three weeks), which cannot be combined with the Y/M/D/H/M/S designators in the same value. §5.4’s parenthetical doesn’t mention W anywhere, and nothing in the surrounding prose says whether that’s because the week form is deliberately excluded from the schemaless !duration atom, or because the parenthetical is a representative example of the ISO 8601 duration format rather than an exhaustive grammar (the same way, elsewhere in the document, a parenthetical sometimes illustrates rather than fully specifies). Both readings are defensible: excluding W would be consistent with !duration’s host value being modeled as year/month/day/hour/minute/second components (a week doesn’t decompose uniquely into those without picking a day-length, though P3W itself carries no such ambiguity on its own terms); including it would be consistent with simply deferring to “the ISO 8601 duration format” as a whole, of which PnW is a normal part.

Interpretation chosen: DurationType’s parser accepts only P followed optionally by Y/M/D designators, optionally followed by T and H/M/S designators, matching §5.4’s parenthetical literally — P3W is rejected as a parse error, not specially recognized. This was the more conservative reading available (implementing a format the annotation’s own table doesn’t show would be a bigger leap than declining to implement one it might have intended by reference), but it’s a real coin flip, not a confident call.

Suggested resolution: State explicitly whether PnW is part of !duration’s accepted format or not. If it is, the table’s parenthetical should show it (PnYnMnDTnHnMnS / PnW) the same way §5.6’s table spells out multiple accepted grammar forms per numeric atom explicitly rather than by implication.

Status against Revision 33: open, carried deliberately. The change log records it as “likely revisited in a later revision”, and §5.4’s table is unchanged — !duration still reads ISO 8601 duration (PnYnMnDTnHnMnS) with no mention of W. DurationParser still rejects P3W.

2. Content-hash pinning rides in the URI query (?sha256=), where a hash is neither a request parameter nor part of identity — external review suggests a fragment, or a structured { url, sha256 } directive, instead

Section: Part 1 §2.2.1 (canonical identity / hash-pinned references), §10.2 (per-identity verification).

Problem: The spec pins a reference’s integrity by appending a query parameter to its URI — !!import:"…/core.tn?sha256=<hex>" — and then defines canonical identity by stripping that query, so a pinned and a plain reference name the same identity. Two objections from external review:

1. Query is the wrong URI component for a hash. By URI semantics (RFC 3986 §3.4) a query is part of the request — data conveyed to the origin to identify/produce the resource — whereas a content hash is verification metadata about the retrieved bytes, evaluated entirely client-side and never meaningfully sent to a server. A fragment (#sha256=<hex>, §3.5) is the component that actually matches: it isn’t sent in the request, is interpreted by the client, and is already outside what a server sees. That the spec must special-case stripping the query to recover identity is itself a symptom that the hash sits in a component whose native semantics it doesn’t share.

2. Integrity arguably shouldn’t be in the URI at all. A second reviewer proposes separating the locator from the integrity outright — a structured directive value rather than a hash smuggled into a string:

   !!schema: { url: "https://example.com/people.tn"  sha256: "c4d5e6f7…a2b3c4d5" }

   This drops URI-parsing of hash parameters, the canonical-identity stripping rule, and the “only hash-algorithm query parameters permitted, everything else rejected” special case; makes the algorithm an explicit, extensible field rather than a magic query key; and mirrors how lockfiles / package managers / Subresource Integrity separate where from what it must hash to. It is the larger change: directives currently take a bare URI string, so this is a directive-grammar change, and !!id (which today carries its own pin on its own line, excluded from the hash) would need an equivalent structured form.

Interpretation chosen: This implementation follows the spec as written — the query form. TsonContentHash parses ?sha256=<hex> off a reference (rejecting any other/unrecognized query parameter or malformed hex), CanonicalIdentity/TsonSchemaRegistry strip the query to key everything by identity, tson hash <file> stamps ?sha256= onto the !!id line, and the bundled chain (meta.tn pins meta-kernel, core.tn pins meta.tn) is pinned end-to-end this way. No change made — flagging the design, not diverging from it.

Suggested resolution: Choose among three. (a) Keep the query form — simplest, but semantically stretched and dependent on the identity-stripping rule. (b) Move the pin to a fragment (#sha256=<hex>) — better matches URI semantics, and identity can still ignore it by dropping the fragment; here a small, localized change (parse # rather than ? in TsonContentHash, and in tson hash). (c) Lift integrity out of the URI into a structured directive ({ url, sha256 }) — the cleanest separation of locator from integrity, at the cost of a directive-grammar change plus an !!id equivalent; here it touches the directive grammar, TsonContentHash, canonical identity, tson hash, and every bundled .tn’s pin lines. If the query form stays, the spec should at least justify why a hash lives in the query and name the identity-stripping as its deliberate consequence.

Status against Revision 33: open, carried deliberately. §2.2.1 keeps the query form and now states the surrounding discipline more firmly — the pin is “verification metadata, not identity”, a query MUST consist solely of hash parameters, and an unrecognised parameter name is an error rather than something silently retained. The placement question itself is untouched.

3. §9.4 has nowhere to attach: the kernel types every name token, token carries no contract, and the grammar governs it four different ways

Section: [TSON-DATA] §9.4 (Confusable Characters), §7.1 (UAX #31 profile), §2.5 (field-name identity), §7.2.1 (NFC normalization); [TSON-SCHEMA] §2.2.3 (!!import name disjointness), §5.2, §5.4.

Problem: §9.4 is one sentence of advice:

Implementations processing untrusted TSON input SHOULD consider Unicode confusable detection (UTS #39) when field name identity is security-relevant.

Four things make it unactionable, and the fourth is the root cause of the other three.

1. It names the one UTS #39 mechanism that needs context the series has not defined. “Confusable detection” is the skeleton() mapping (UTS #39 §4): two strings are confusable iff their skeletons are equal. That is a relation between strings, so it says nothing about a single name — it needs a comparison set, and §9.4 names none. The mechanisms decidable on one name alone go unmentioned: Identifier_Status (UTS #39 §3.1, a per-character Allowed/Restricted partition) and the restriction levels of §5.2, which are properties of one identifier computed from script.

2. The strictness is inverted relative to the risk. §7.2.1 makes NFC a MUST, so two canonically equivalent names are the same name. Confusability gets SHOULD-consider, so two visually identical names are different names. The series takes a firm, testable position on the convenience case and none on the attack.

3. The prescribed workaround reopens the surface it closes. §7.1 excludes ZWNJ/ZWJ and says “names whose orthography requires them MUST be quoted”. But §7.1 constrains unquoted tokens only, and §2.5 makes a quoted name an ordinary field name (field-name = token ; unquoted or quoted, and “name and "name" are the same field name”). So the sanctioned route for a name needing ZWNJ is an unconstrained route for every character the profile excludes: the hardening is bypassed by the mechanism the same sentence prescribes.

4. The root cause: the name layer exists and is empty. §7.1 defines a UAX #31 profile over unquoted tokens — a lexical class that in TSON is also how ordinary values are written ({ name: Alice }) — so §9.4 cannot attach to §7.1 without constraining values too. The place it should attach to is the kernel’s token, which types every name in the series and holds no constraint at all, deferring to a grammar that governs its positions with four different productions. That is why §9.4 ended up in Security Considerations as advice: the surface exists, but nothing has ever been said on it. It is also why point 3’s bypass exists, and why the bypass is confined to field names. The proposal below is mostly a matter of filling in a slot the kernel already reserved.

The data-cost objection, measured rather than assumed. An earlier draft of this entry argued that any normative UTS #39 requirement obliges every implementation to ship UCD data, and treated that as one cost. It is three, and they differ by two orders of magnitude:

Only the third is a materially larger ask. The first costs nothing and catches the case §9.4 opens with.

Interpretation chosen: none of UTS #39 is implemented. NFC normalization of unquoted tokens (§7.2.1) is, and §7.1’s identifier profile is now exact (see #14), which removed the invisible-character half of the problem — no Cf character, bidi control included, can appear in an unquoted token. Nothing detects a confusable pair, a mixed-script name, or a Restricted character, at either layer, including at !!import merge, where disjointness is exact string equality and a confusable pair passes it by construction.

Proposal: define an identifier profile, then use the mechanism that has a comparison set

Step 1 — the identifier profile already exists in the kernel, as token, and it carries no contract.

This is the foundation, and it is worth adopting even if no mechanism below it ever is. It is also smaller than it first appears, because nothing needs inventing: the meta-kernel already types every name position in the series with one atom.

token      => !unit {}        ; the lexical-identifier primitive
type_name  => token
field_name => token
param_name => token

Every naming position in the resolved model is typed by it, directly or through one of those three roles — type_ref.name, record_field.name, field_group.members, record.supertypes, type_definition.parameters/supertypes/subtypes, enum.members (via token_set => !set { element_type: token }), and the keys of schema => {type_name => type_definition}. §8.3 flattens the roles away, so resolved output states the type uniformly and keeps the role as an alias:

!record_field { name: name  type: @alias:field_name token }
body: !map { key_type: @alias:type_name token  value_type: type_definition }

A rule stated on token therefore reaches every name in the series without enumerating positions, and diagnostics can still say which kind of name failed, because the alias survives.

Critically, token is not text. token is a unit instance with no constraint vocabulary; text is a text_type instance with min_length, pattern and the rest. They are separate types, so they can carry separate policies — which is exactly the affordance an identifier profile needs, and it is already built. The thing that has never been decided is what token’s policy is.

The kernel says so itself, and points at the grammar:

The implementation-internal constraints on what constitutes a well-formed token are fixed by the grammar; no constraint vocabulary is needed at the type level.

That deferral is where it breaks, because the grammar does not fix one thing. Four productions govern name positions and three of them disagree:

token in the grammar is unquoted-token / single-line-token / multi-line-token, so field-name is the one name position in either part that admits a quoted spelling. Everything else the kernel types token is grammatically unquoted-token. One kernel type, four grammar rules. Enum members are a fifth treatment again: the model types them token, and the grammar has them ride core-value as ordinary data.

Three consequences follow, and they are the whole of why §9.4 is stuck:

1. Names cannot be constrained without constraining values. The only place the series constrains characters is §7.1’s UAX #31 profile, which is defined over unquoted-token — the class that also carries values. In { name: Alice } both name and Alice are unquoted tokens, so any rule added there lands on the value too. token is where a name rule belongs, and it holds none.
2. Quoting escapes the rules, at the one position that allows it, and §7.1 prescribes quoting as the remedy. §7.1 excludes ZWNJ/ZWJ and says names needing them “MUST be quoted” — but §7.1 governs unquoted tokens, and "аdmin" is an ordinary field name (§2.5). The hardening is bypassed by the mechanism the same sentence recommends. That the bypass is confined to field names is an accident of the inconsistency above, not a design.
3. §9.4 has nowhere to attach. It is a rule about names; token is the type of every name and says nothing; so §9.4 became a SHOULD-consider sentence in Security Considerations. Meanwhile [TSON-SCHEMA] §5.10 reaches across parts to cite §9.4 for a schema naming hazard, with no shared surface between them.

So the ask is two small things, not a new concept:

• Give token a stated contract — in the kernel’s own terms, or in §7.1 beside the token profile, saying that constraints on names apply to a name however it is spelled. Map keys stay out: §2.6 makes them data values, not restricted to strings, and they are typed accordingly.
• Make the grammar reference one production for it. Whichever way it is settled, the four rows above should be one row. Settling on token (admitting quoted spellings everywhere) is the more consistent direction and is only safe after the first bullet, since today the inconsistency is what accidentally limits the bypass; settling on unquoted-token closes the bypass by grammar but makes a name that needs an out-of-profile character unspellable, which is the situation §7.1’s ZWNJ advice was written to avoid.

A useful side effect: the comparison scopes Step 2 needs are then expressible in the model rather than only in prose. “The declared names of one schema” is the key set of schema => {type_name => type_definition}.

Step 1a — “token” is doing three jobs, and only one of them is lexical. The word names a lexer output, a name, and an enum member, and the three have different rules:

1. A lexer token — a unit of token-stream = *( ws / single-line-token / multi-line-token / unquoted-token / structural-delimiter / … ) (§7.4). Its constraints are genuinely lexical: an unquoted token cannot contain a space, because a space ends it; a quoted token can contain spaces anywhere, because its delimiters say where it ends. This is the only sense in which “what may appear in a token” is a question about tokenising.
2. A name — field-name, type-name, param-name. Not a lexer output at all: it is the decoded text of one, after unquoting, escape processing and NFC. Its constraints are semantic, and the grammar currently states them by writing field-name = token, which is the category error at the root of this entry. A name may not contain a space anywhere, including the middle, and that has nothing to do with whether a space would have ended the token that spelled it.
3. An enum member — also typed token by the kernel (§9: “token — admits NFC-canonical lexemes. Used for identifier types (type_name, field_name, param_name) and enum members”). Today that makes a member an uninterpreted lexeme whose base type resolves per member, which is why §5.4 can call [true false] boolean-class. Step 1d proposes closing this job rather than naming it: members become identifiers, as they are in every comparable schema language, and the third sense of “token” disappears along with the entry that carried it. That constrains how a member is written and leaves §5.4 alone — the class is still read off each member’s own token by §4.

The practical consequence for the kernel: type_name, field_name and param_name alias a new entry — identifier — carrying the contract in Step 1b, and enum.members joins them (Step 1d), which leaves token with no remaining use. One rule, stated once, reaching exactly the positions that hold names.

On the word. identifier is what every programming language calls this, and §7.1 already reaches for it: “TSON’s unquoted tokens are a declared profile of Unicode identifiers per UAX #31 requirement R1.” That sentence attaches the word to the wrong class — an unquoted token is a spelling, and UAX #31’s profile is about the thing spelled — so adopting identifier for the thing itself makes §7.1’s own sentence true where it currently is not. The sentence should be reworded to match: the profile constrains identifiers, of which an unquoted token is one spelling and a quoted token another.

And the spec already has the mechanism this needs; it is used for numbers. §7.6:

The number grammar applies to the complete text of a token; it is not part of the token-stream grammar … a candidate token is first produced by the lexer, then matched — in full — against it.

That is exactly the two-layer shape an identifier profile wants: the lexer emits a token, and its decoded text is then matched against a separate production that is not part of the token-stream grammar. Numbers already work this way; names should, and the note explaining why can be copied nearly verbatim. It also disposes of the objection that a name rule would complicate lexing — it does not touch the lexer, for the same reason the number grammar does not.

One correction worth carrying into the wording, because it decides where each rule lives. Not everything excluded from names is excluded for lexical reasons:

• Genuinely lexical: whitespace, and U+200E/U+200F (LRM/RLM) — the latter because [UAX31-R3a-1] item 2 makes them ignorable format controls admitted only where a token boundary already exists, so ab<LRM>c is a lexer error rather than a name question at all (#16).
• Not lexical at all: ZWNJ, ZWJ, BOM, soft hyphen, word joiner, the bidi overrides. Every one of these is in XID_Continue or was admitted by the JDK predicate this implementation used; the lexer absorbs them into a token happily (it did until #14). Excluding them is a policy decision, and stating it in §7.1 — which governs unquoted tokens — is why the exclusion never reached quoted names.

Step 1b — what the identifier contract should say. Form and policy stay orthogonal: the grammar decides which spellings a position admits — the @ and ! positions are unquoted-only for adjacency reasons unrelated to confusability, and stay so (Step 1c) — while the policy decides which identifiers exist, and is the same at every position. A position may admit fewer spellings without admitting a different set of identifiers.

Stated as a profile beside §7.1’s own, the contract is almost the token profile already — it differs exactly where the number grammar forced the token profile’s hand:

Token   Start = XID_Start ∪ Nd ∪ { - + . }     Continue = XID_Continue ∪ { - + . }
Ident.  Start = XID_Start                      Continue = XID_Continue ∪ { - }

Three differences, each with its own reason:

• Identifier-Start drops Nd, -, +, . — §7.1 adds those four to Start for one stated purpose: “every extension character is required by a production of the number grammar (§7.6): Nd for digits, -/+ for signs and exponent signs, . for the decimal point and .inf/.infinity/.nan”. They exist so a number can be an unquoted token, and they leak into names only because names and values share one profile. Dropping them is what makes { 42: 1 } stop being a record with a numeric field name, and it subsumes [TSON-SCHEMA] §12.1’s existing rule that “numbers are not declarable names” — every spelling the number grammar admits begins with a digit, a sign, or a dot, so the general rule and the type-name rule become one rule. Continue keeps - (my-field), which no number-grammar concern touches.
• Identifier-Start does not add _, deliberately. It is tempting to: §7.1 keeps _ out of token-Start for a purely lexical reason — the bare token _ is the absent sentinel (§2.9) — and _id is an ordinary field name. But admitting it buys only !_id and @_note, since _id is already spellable as a quoted field name ({ "_id": 1 } reads today), and it costs either a lexer change to tell _ from _x or an identifier that its own unquoted spelling cannot express. Leaving _ out changes nothing that works today. §7.1’s “names with a leading underscore MUST be quoted” then stays true and becomes a statement about spelling, which is what it always was.
• + and . are dropped from Continue as well. + is needed only for exponents. . is dropped to reserve it: a dot is the near-universal identifier separator — qualified names, path access, namespacing — and admitting it into identifiers now would foreclose using it as one later. Nothing in the three bundled schemas declares a name containing a dot, so the reservation costs nothing today and keeps person.name and ns.type available as future syntax rather than as two identifiers that happen to contain dots.

A property worth stating, because it makes the unquoted-only positions free. With _ left out, every part of the identifier profile is inside the token profile: XID_Start ⊂ XID_Start ∪ Nd ∪ { - + . } and XID_Continue ∪ { - } ⊂ XID_Continue ∪ { - + . }. So every identifier is a well-formed unquoted token, and no identifier ever needs quoting to be written. That is what makes type-ref and annotation — which admit no quoted form — lose nothing, and it is the invariant to preserve if the profile is ever widened: adding anything to identifier-Start that token-Start lacks (_ being the obvious candidate) breaks it, and creates identifiers that cannot be spelled at two of the four name positions.

Everything else falls out of XID membership rather than needing a clause: whitespace, C0/C1 controls, Cf format characters, emoji and unassigned code points are none of them XID_Continue, so a one-line profile excludes the lot. Two rules do have to be stated on top:

1. NFC — and here the series already has a rule, which this entry would change rather than clarify. §7.2.1 does not merely define identity by NFC; it says what a processor does about it: “quoted tokens that occupy identifier positions … are NFC-normalised by the resolver before identity comparison. String-typed positions are not normalised.” So a non-NFC name is normalised, not refused, and the section already draws this entry’s own axis — position, not quoting — several sections before the identifier layer this entry proposes naming. That is worth saying plainly: §7.2.1’s second paragraph is the closest the series comes to having the concept already.

   The open question is only whether normalising is the right treatment or whether a name should be required to be NFC, as an unquoted token already is. Requiring the form is the simpler rule — it is checkable on one name, it keeps the document’s bytes authoritative for names as the first paragraph keeps them for unquoted tokens, and it removes the one place a conforming processor alters text. Against it: it rejects documents §7.2.1 accepts, and a quoted name is exactly where an author has least control over the form their editor produced. Recommendation: keep §7.2.1’s normalisation, and treat the requirement version as a tightening to weigh separately rather than as part of the identifier profile. Either way the profile’s other clauses are unaffected.

2. ZWNJ and ZWJ are constrained contextually, not excluded. See below — this is the one place the proposal contradicts §7.1 outright.

Mixed script stays permitted; a restriction level remains available as an opt-in profile, default off (Step 4). At scope level rather than per name, no two names in one scope may share a skeleton() (Step 2).

Step 1c — what the grammars become, and where the constraint is actually enforced. Part 1 gains an identifier production and uses it at the two positions that are already unquoted-only:

identifier      = identifier-start *identifier-continue
identifier-start    = XID_Start
identifier-continue = XID_Continue / "-"
                ; the profile of Step 1b; a name's decoded text is matched
                ; against this in full, as §7.6 matches a number's

type-ref        = "!" identifier
annotation      = "@" identifier [ ":" data-value ]
field-name      = unquoted-token / single-line-token

field-name deliberately stays lexical, and that is the move that makes the whole thing work. A Class 1 document has no schema, so nothing there knows which tokens are meant as identifiers; {"first name": 1} reads as an ordinary record and JSON compatibility is untouched. The identifier constraint is stated once, in Part 2, on declared names — and the data side then conforms by construction: a schema’s field names are identifiers, a data field name is valid only if it matches a declared one, so a field name that is not an identifier matches nothing and is already an UNRECOGNIZED_FIELD. No second check at the data layer, and no new failure mode — a document that would have been rejected is rejected for the reason it was always going to be.

This is the shape the series already uses for values, one layer down. { x: text } accepts the token true as the string "true", because §7.3 gives true no special status under a schema: the lexer produces a token, and the layer that knows the type decides what it is. Names work the same way — the lexer produces a token, and the schema decides whether it was an identifier.

One interaction this creates, verified against the current implementation:

• field-name drops multi-line-token. It is token today, which includes the triple-quoted form, so a multi-line string is currently a legal field name. Dropping it is a tightening with no cost anyone will notice, but it is a change and should be stated as one.

And one consequence for the security story, which should not be left implicit. Because Class 1 field names stay unconstrained, the identifier profile protects schema-governed documents only. Schemaless documents keep their protection from Step 2 instead — skeleton distinctness within a record’s own field set, which needs no schema, since a record’s fields are a closed scope on their own. The two mechanisms cover the two classes, and neither covers both; saying so is what keeps the layering honest.

Step 1d — the kernel edit, including enum.members. The three identifier roles move, and enum members move with them:

identifier => !unit {}
type_name  => identifier
field_name => identifier
param_name => identifier

enum_set   => !set { element_type: identifier  min_items: 1 }
enum       => ~atom & { members: enum_set }

token and token_set are then unused and go. The word that was doing three jobs ends up doing none, and the one real job has its own name.

Why identifier and not text. text would leave members as arbitrary strings and change nothing; identifier constrains how a member may be spelled and does so for the same reasons every other name is constrained. Three of them:

1. Every comparable schema language already does it. GraphQL is the closest — “enum values are represented as unquoted names (ex. MOBILE_WEB)”, “it is recommended that Enum values be ‘all caps’”, and “there must be at least one and they must have unique names”. Protobuf, Rust, C#, Java and TypeScript agree. An author arriving from any of them finds what they expect, which for a format aimed at generated output matters more than for one aimed at hand-editing.
2. Members inherit Step 1b’s contract at no cost — NFC, no whitespace, no Cf, no controls — protection that would otherwise need restating for this one position. !enum [ACTIVE ACTIVE<ZWNJ>] stops being expressible, which is the class of member confusion nothing currently prevents.
3. The cases it removes have better homes. !enum [1 2 3] is an !integer ^ { min: 1 max: 3 } or a choice, and modelling it as an enum was a way of saying “small integer” in the wrong vocabulary. !enum ["in progress"] becomes IN_PROGRESS with the display string mapped at the boundary — exactly what GraphQL and Protobuf require, and not a limitation anyone reports as one. Every enum in the three bundled schemas is already identifier-shaped, so nothing in the series moves.

No carve-out for true, false and null, unlike GraphQL — and the reason is instructive. The kernel defines the boolean primitive as an enum:

boolean => !enum [true false]

so excluding those tokens would leave the kernel unable to state its own scalar types. GraphQL can carve them out because Boolean is a built-in scalar there and enums are a user-facing construct layered above it; in TSON enum is a kernel constructor that defines primitives, and the carve-out does not transfer.

Nor is one needed, which is worth being precise about because it is easy to talk oneself into. Typing members as identifier constrains how a member is written; it does not change what §4 makes of that token on the wire. An enum’s discrimination class (§5.4) is computed by running base-type resolution over each member’s own text, so boolean stays boolean-class, [INDEX NAMED] stays string-class, and a mixed enum still yields none. §5.4 is untouched by this change, choice disjointness is unaffected, and there is no special case anywhere: the enum reader compares the token it was given against the member list, which is the same thing it does for INDEX.

One rule the change does bring: min_items: 1. An enum with no members is uninhabited, and GraphQL requires “at least one” for the same reason. Stating it on the set is one field, and it means !enum [] fails at schema load with a size violation rather than at first read with nothing to match.

Uniqueness continues to come from set (unique_items: = true), which is why this stays a set rather than becoming [identifier]: !enum [OPEN OPEN] must remain the error it is today, and GraphQL requires it too. Confusable members — !enum [ACTIVE АCTIVE] with a Cyrillic А — still need Step 2, since both are valid, distinct identifiers; §5.4’s member list is already one of the scopes it covers.

Precedent, and one place the obvious precedent is a warning rather than a model. Java’s identifier rules line up with Step 1b’s on the two decisions that were live here: an identifier may not start with a digit, and _ is a legal start character. Java adds $, which TSON has no reason to copy — UAX #31 cites exactly that as its example of a profile extension (“Java and C++ identifiers include ’$’, which is a Pattern_Syntax character”). Two Java rules TSON should state rather than inherit silently:

• Case sensitivity. Java identifiers are case-sensitive; so are TSON names, and §2.5’s NFC-based identity says so only by omission. One clause.
• Reserved words. Java excludes its keywords. TSON’s equivalent is exactly one character — _, which §7.1 reserves at token-initial position for the absent sentinel — and it is reserved at the level of spelling, not of names. Under the split that resolves cleanly and is worth stating as the worked example: { "_": 1 } is a record with a field named _, while { _: 1 } is not a record at all. The identifier is legal; one of its two spellings is taken. Nothing else in TSON needs a keyword list, because name collisions with declared types are a namespace question (§3.3.1) rather than a lexical one.

The warning: Java identifiers are not the ASCII set they are often summarised as. int café, int Ω, int 名前 all compile, because the rule is Character.isJavaIdentifierStart/Part — and that predicate carries the same defect as isUnicodeIdentifierPart (#14): it admits every identifier-ignorable character, so isJavaIdentifierPart returns true for U+200C, U+FEFF and U+202E. A bidi override is a legal character inside a Java identifier, which is the Trojan Source class of vulnerability in the language that defines the predicate. So Java is a good precedent for the shape of an identifier rule and a poor one for its character set — and it is a second, independent instance of the exact trap this register is asking §7.1 to warn implementers about.

Interior whitespace, and why JSON compatibility survives it. XID_Continue excludes whitespace, so first name is not an identifier. Under Step 1c that costs JSON nothing: field-name stays lexical, so { "first name": 1 } is still an ordinary Class 1 record and §7.1’s claim that “the comma separators and quoted keys required by JSON are accepted by the TSON grammar” stays true as written. What changes is only what happens when such a document is validated against a schema: no declared field can be named first name, so the field is unrecognised — an ordinary schema mismatch, reported where the mismatch is, rather than a lexical rejection of the document. That is the outcome to want, and it is the reason the constraint belongs on declarations rather than on the field-name production.

ZWNJ and ZWJ: §7.1’s exclusion is over-broad, its remedy is the attack, and Unicode has moved. §7.1 excludes both, reasoning “they are invisible, which makes them confusable and spoofing surface (§9.4); names whose orthography requires them MUST be quoted”, and notes that “UAX #31 permits them in restricted contexts”.

Three things are wrong with that, and they compound:

1. The characters are not optional decoration in the scripts that use them. ZWNJ breaks a cursive connection; ZWJ forces one. In Persian it is ordinary spelling — می‌رود (“he goes”) and کتاب‌ها (“books”) are written with ZWNJ separating the morpheme, and without it the word is misspelled. Indic scripts use both to control conjunct formation. Excluding them does not inconvenience an author; it makes a class of correct words unspellable as names.
2. They are invisible only where they do nothing. In a cursive script a ZWNJ has a visible effect — the letters render disconnected, which is the whole point of writing it. It is invisible precisely when inserted where it has no shaping role, i.e. into Latin: ad<ZWNJ>min renders exactly as admin. So the dangerous case and the legitimate case are distinguishable by context, mechanically.
3. UAX #31 no longer says what §7.1 cites it as saying. The “restricted contexts” wording matches requirement R1a, which has been removed: “The characters that were added when meeting this requirement are now part of the default; the contextual checks required by this requirement remain as part of the General Security Profile in [UTS #39].” That is why ZWNJ/ZWJ are in XID_Continue today (see #14) — they are default identifier characters now, and the safety rule moved to UTS #39 §3.1.1.1, Limited Contexts for Joining Controls, which states conditions A1/A2/B on the neighbouring characters’ Joining_Type, under two global conditions — the sequence must be single-script (ignoring Common/Inherited) and in NFC — and which UTS #39 itself describes as “simple enough to be easily implemented with standard mechanisms such as regular expressions”.

So §7.1 has it exactly backwards. The blanket exclusion blocks the legitimate use — a Persian field name — while the escape hatch it offers in the same sentence (“MUST be quoted”) is precisely the route by which "ad<ZWNJ>min" reaches a Latin name today, because §7.1 governs unquoted tokens only. It forbids the safe case and permits the attack.

Recommendation: replace the exclusion with UTS #39 §3.1.1.1’s contextual rule, applied to identifiers, and leave §7.1’s set algebra alone — XID_Continue should keep meaning XID_Continue. The rule is total and two-valued, permits کتاب‌ها, refuses ad<ZWNJ>min, and composes with the rest of Step 1b rather than sitting beside it: its two global conditions are NFC and single-script, and NFC is already clause 1 above. §7.1’s ZWNJ paragraph then states a rule instead of a prohibition plus a remedy that does not work.

The token profile moves with it. Because the joiners are XID_Continue, the lexer’s unquoted-token profile admits them too — which it must, or the subset invariant above breaks and an identifier containing a ZWNJ becomes unspellable at the ! and @ positions. So one rule governs both layers, and the contextual check is the only thing standing between کتاب‌ها and ad<ZWNJ>min. It follows that the lexer change and the contextual check are a single step: admitting the joiners first would reopen the hole #14’s fix closed.

If the contextual rule is judged too much machinery, the fallback is to keep the exclusion and delete the remedy — say plainly that such names cannot be expressed, rather than directing authors to a spelling that also lets the attack through. What should not survive is the current pairing.

Step 2 — require skeleton distinctness within each named scope. No two names in the same scope may have equal UTS #39 skeleton()s. The scopes are the closed sets the series already defines:

• the field names of one record (§2.5 already defines their identity and a duplicate rule)
• the members of one enum, the variants of one choice (§5.4)
• the declared names of one schema
• the merged namespace at !!import — the sharpest, since §2.2.3 already requires imported names be “disjoint from each other and from local entries”, and disjointness there is exact equality, which a confusable pair passes by construction: two entries a reviewer reads as one name are, to the resolver, two names

Why this and not the restriction level, which is free. An earlier draft of this entry recommended the opposite, ordering the mechanisms by the data an implementation must ship. That is the wrong cost to sort on. Sorted by what each rejects, they invert — measured, both prototyped:

The restriction level is a property of one name, so it must guess from the name alone, and it guesses wrong on ordinary names. These are all rejected by it:

REJECTED  id_пользователя      REJECTED  url_адрес      REJECTED  api_ключ      REJECTED  χ_index
accepted  日本語id

A Latin abbreviation beside a Cyrillic or Greek word is an ordinary way to name things, and the last line is the tell: the same mixing is permitted for Japanese and refused for Russian. That is principled — Han is not confusable with Latin and Cyrillic is — but it lands as a tax on precisely the authors the rule exists to protect, while a Latin-script author never encounters it.

And “ordinary” understates it. Software development is conducted in an English-oriented system: keywords, library names, protocol terms and the conventional abbreviations (id, url, api, http, json) are all Latin, and they appear inside identifiers written in every other script. For a developer working in Cyrillic, Greek, Devanagari or Arabic, mixing Latin into a name is not an edge case to be traded away — it is the common case, and id_пользователя is a more natural name than the all-Cyrillic alternative, not a worse one. So the restriction level’s false-positive rate is not “occasional in some documents”: it is close to routine for one part of the userbase and exactly zero for another. A rule with that distribution is not a strict default with a documented cost; it is a rule that will be switched off wherever it actually applies, and protect only the people who were never going to trigger it.

skeleton() distinctness has no such failure mode: it is a relation, so it fires only when two names in the same set actually collide. Over 32 names of the kind a real schema declares together — including all four rejected above — it reported zero collisions, while catching every homograph pair tested, the whole-script aec/аес included. What it does flag is genuine: l/I, O/0, rn/m.

The reason the free mechanism looks attractive and is not. Restriction levels are what browsers use for IDN, and browsers are right to: a browser cannot enumerate the comparison set — it has no way to know which domains a user might confuse this one with — so it must judge a name in isolation, and accepts false positives as the price. TSON is in the opposite position: every scope above is closed, small, and known at the moment the check would run. Borrowing the browsers’ answer without borrowing their constraint imports a false-positive rate that TSON has no reason to pay.

Step 3 — where it binds. A MUST at the !!import merge and over one schema’s declared names; a MUST for the field names of one record and the members of one enum, which are the same closed-set check one level down. Data documents then conform by construction under a schema, since a data field name is valid only if it matches a declared one. For schemaless data (Class 1) the record-scope check still applies — it needs no schema, only the record’s own field names.

The 706 KB falls on implementers, once, and can be a generated table; the alternative puts a recurring cost on authors, in one script community, forever. If shipping the table is judged too much, the honest outcome is to keep §9.4 advisory rather than to adopt the cheap mechanism as a substitute — it is not one.

Step 4 — the restriction level: Highly Restrictive by default, with per-segment as the first relaxation. An earlier draft of this entry recommended opt-in and default off, because §5.2’s levels are stated over a whole identifier and every one of them rejects id_пользователя, url_адрес and api_ключ — the Latin abbreviations that appear inside identifiers written in every other script. The objection was sound and its conclusion was not: it assumed the only alternative to a strict rule is no rule. Applying the same level to each _/- delimited segment is a narrower rule, not an absent one, and it is what makes a strict default safe to ship.

Apply Highly Restrictive to each _/- delimited segment rather than to the whole name. Programming identifiers are compounds, and their separators are exactly the boundaries at which a script change is ordinary rather than suspicious. Measured over the cases that decided the earlier recommendation:

                       whole name      per segment
аdmin  pаssword  usеr   REJECT          REJECT        within-word homographs — the attack
id_аdmin                REJECT          REJECT        one bad segment is still one bad segment
id_пользователя         REJECT          accept        the common case for a non-Latin developer
url_адрес  api_ключ     REJECT          accept
alpha_α  χ_index        REJECT          accept
日本語id  order_id       accept          accept

Every rejection the rule exists for survives; every false positive that made it undeployable disappears. Nothing is given up to the attacker, because a homograph has to sit inside a word to read as that word: spoofing admin needs аdmin, which is one segment mixing scripts, and spoofing id_admin needs id_аdmin, whose second segment does. What passes the script rule is a segment that is wholly one script and merely looks Latin — id_аdмin — and that is the whole-script case Step 2’s skeleton distinctness catches. The two mechanisms compose: the script rule refuses mixing inside a word, the skeleton refuses lookalikes across a scope, and neither covers the other’s case.

So it can be a MUST rather than an option, which is the better default for a security rule and removes the awkwardness of a normative document specifying something it expects to be switched off.

What it still costs, and who pays it. A name with no separator that legitimately mixes scripts is refused, and the author writes the separator: ид_HTTP rather than идHTTP. That is much smaller than the narrowing it replaces, but it is not symmetric, and the asymmetry is inherited rather than introduced.

Highly Restrictive admits three augmented script sets — Latin+Han+Hiragana+Katakana, Latin+Han+Bopomofo, Latin+Hangul+Han — so 日本語id is {Latin, Han} and passes with or without a separator, while пользовательid is {Latin, Cyrillic} and passes only as пользователь_id. Unicode draws that line for a reason: Han, Kana and Hangul share no confusable characters with Latin, and Cyrillic and Greek are full of them (а/a, е/e, о/o, р/p, с/c; α/a, ο/o, ρ/p). So the rule is not arbitrary — but a Japanese author may omit the separator and a Russian one may not, and that is the residue of the objection segmentation otherwise removes. It is worth saying out loud in the spec rather than leaving an author to infer it from a rejection, because the fix is one character and the confusion otherwise is total.

Step 4’s configuration surface — two dimensions, not a ladder. It is tempting to expose one ordered knob from strictest to loosest. That is wrong, because per-segment Highly Restrictive and Moderately Restrictive are incomparable: the first admits id_пользователя (Latin and Cyrillic, but never inside one word) and refuses id_हिन्दी; the second does the opposite (Latin plus any one Recommended script “except Cyrillic, Greek”, anywhere in the name). Neither contains the other, so they are two axes:

• level — which script combinations a unit may contain. UTS #39 §5.2’s own six, unchanged, so the default is a named standard rather than a TSON invention and two implementations agree on it without reading this document.
• unit — whether the level is applied to the whole name or to each _/- delimited segment. A no-op at ASCII-Only and at levels 5–6; it changes levels 2–4.

Recommended default: Highly Restrictive, whole name. Strictest of the practically deployable levels, and a level UTS #39 names — which matters more than it first appears. A default that is TSON’s own refinement would mean a document validating under one implementation and not another that implemented only the standard; a standard default interoperates, and the refinement is something a deployment opts into knowingly.

And the relaxation to reach for first is the unit, not the level. Moving to per-segment keeps every within-word homograph refused — аdmin, pаssword, usеr, id_аdmin — while admitting the compound names that made the whole-name form undeployable. That is the property that makes a strict default safe to ship: the pressure a strict rule creates is answered by a narrower rule rather than by switching the rule off, which is the failure mode an earlier draft of this entry worried about and the reason it recommended defaulting to off.

The two “off” positions are different, and the difference matters. §5.2 is explicit: Minimally Restrictive drops the script restriction while “characters in the string must also be in the identifier profile”; Unrestricted additionally allows characters “outside of the identifier profile” — so it discards Identifier_Status too (Step 5), and with it the obsolete and technical characters and the joiners. A deployment that means “stop checking scripts” wants Minimally Restrictive; Unrestricted is a diagnostic tool, and §5.2 says so. A configuration surface offering one “off” would silently give the second.

A third axis worth offering, narrower than any of the above: an additional permitted script set. A deployment that knows what it is — a Russian shop, a Greek-language corpus — can say so precisely (Latin + Cyrillic) instead of dropping a level and losing the rule everywhere else. It is the same mechanism §5.2 already uses for Latn+Jpan and its siblings, extended by configuration rather than by standard, and it is the most targeted relaxation available.

What should not be configurable, and stating it is part of the design. Skeleton distinctness has no false positives — it fires only on a colliding pair, so there is nothing for a deployment to be relieved of. Identifier_Status is not a separate switch: it is what level 6 turns off, and offering it twice would let a configuration hold two contradictory answers. And no severity knob: the levels are the severity, each one a conforming position, where a report-but-accept mode would make the processor non-conforming while looking like a setting.

Step 4b — the same levels belong on values, and there the restriction level is the only mechanism. Everything above constrains names. A value is unconstrained beyond §7.1’s token profile if unquoted, and entirely unconstrained if quoted — so аdmin with a Cyrillic а can be a value wherever it cannot be a name. That matters: an application comparing a value against a blocklist, or rendering it to a user, faces the same spoofing surface §9.4 raises for names, and nothing in the series speaks to it.

Why the mechanism has to be the restriction level here, having been the wrong choice for names. Step 2 prefers skeleton distinctness because it is a relation and TSON can name the sets it holds over. Values have no such sets: two values in one array are not required to be distinguishable, and two values in different documents cannot be compared at all. So the argument that made the restriction level second-best for names — that a better, set-based rule was available — does not apply, and the per-string mechanism is what remains. This is the same reasoning that makes restriction levels right for a browser judging a domain name (#3’s own comparison), applied to the one surface in TSON that is likewise setless.

The default must be the opposite of the names default. A value is data, and data may legitimately be anything: { note: "Ωmega" }, a Greek quotation, a Cyrillic display name. So Unrestricted for values, Highly Restrictive for names — the same ladder, opposite ends, for the same reason in each case. A service that renders untrusted values, or matches them against a list, raises the value level knowingly; nothing is imposed on a format that mostly carries ordinary data.

Three consequences worth stating. Levels 5 and 6 collapse on the token surface — §5.2 says so directly (“where there is no such identifier profile, Levels 5 and 6 are identical”) — and Identifier_Status is a name rule, so a token that is not a name has no identifier profile to drop. The default costs nothing at read time: at Unrestricted no scan runs, so a format carrying ordinary data pays for none of this. And because the token check runs before anything knows which tokens are names, a token policy stricter than the identifier policy simply subsumes it — a name is a token, so it has already cleared the stricter rule by the time the name rule looks at it. That is a property of where the checks sit, not a special case, and the naming below is chosen to make it visible.

A suggested configuration surface. Two checks at two layers, which is what the naming should say:

// The defaults, written out. Tson.builder().build() gives exactly this.
Tson tson = Tson.builder()
        .tokenPolicy(UnicodePolicy.unrestricted())            // every token off the stream
        .identifierPolicy(UnicodePolicy.highlyRestrictive())  // additionally, at naming positions
        .build();

// Relaxing identifiers: reach for the unit before the level. Per-segment still refuses
// every within-word homograph, and admits id_пользователя and alpha_α.
        .identifierPolicy(UnicodePolicy.highlyRestrictive().perSegment())

// Narrower still, where a deployment knows what it is:
        .identifierPolicy(UnicodePolicy.highlyRestrictive().permitting(LATIN, CYRILLIC))

// The two "off" positions, named apart so neither is reached by accident:
        .identifierPolicy(UnicodePolicy.scriptsUnchecked())   // level 5 — identifier profile kept
        .identifierPolicy(UnicodePolicy.unrestricted())       // level 6 — profile dropped too

// Tightening tokens, for a service that renders or matches the values it reads:
        .tokenPolicy(UnicodePolicy.moderatelyRestrictive())
        .tokenPolicy(UnicodePolicy.asciiOnly())

Why token and not value, and why that is the honest name. The check runs where tokens leave the stream, before anything knows which of them is a name — so it constrains names as well as values, and the effective constraint on a name is the stricter of the two policies. Calling the setter valuePolicy would hide that; calling it tokenPolicy states it, and matches where the check actually sits. The consequence is worth having in the Javadoc rather than discovered: a deployment that sets tokenPolicy(asciiOnly()) has made its identifiers ASCII-only too, whatever identifierPolicy says, and that is the right answer rather than a wrinkle — a name is a token.

UnicodePolicy, not ScriptPolicy. The policy is script-based today, but it already carries a unit, and its loosest rung reaches Identifier_Status rather than any script rule (§5.2 level 6). Naming a configuration type after the mechanism it happens to use now would age into a lie the first time it carries anything else.

perSegment() belongs on the identifier policy only. _ and - are word separators by convention in an identifier; in a value they are ordinary characters. UTS #39’s own Minimally Restrictive example is Toys-Я-Us, which per-segment would accept and which is exactly the spoof a strict token policy exists to refuse.

Every method here is a code path, never an environment variable, for the reasons below.

On how an implementation should let it be relaxed, since a normative rule with no escape hatch invites worse ones: not through the environment. A security policy read from an environment variable is ambient authority — a CI config, a container image, a shell wrapper or a dependency calling setenv changes it with no code change, no diff and nothing in review; it is invisible at the call site, process-global, so an embedding library cannot hold its own policy, and it appears in no artifact anyone reads. An opt-out expressed in code has the opposite properties: greppable, diffable, attributable, and scoped to the processor instance that holds it. With the rule strict by default, the code path is a weakening, which is the right thing to make expensive and visible — and it should be named for what it permits rather than for the check it disables.

Step 5 — Identifier_Status, cheaply, on the same profile. Requiring name characters to be Identifier_Status=Allowed (556 ranges, 47 KB) subsumes §7.1’s hand-picked ZWNJ/ZWJ exclusion — currently a single instance of a rule UTS #39 states generally — and removes obsolete and technical characters that have no business in a name. It belongs on the name profile, not the token profile, for the reason Step 1 gives: an unquoted value in a historic script should stay legal. Unlike the restriction level, it is a per-character rule with no cross-script judgement in it, so it does not reject url_адрес.

What a processor does on detection. State it: §2.6 already chose “MUST reject” for duplicate keys, and a confusable name pair is that same defect wearing a disguise. §2.6 is also the precedent for the diagnostic — at the repeated occurrence’s position.

Vectors become possible. All of the above is decidable from the document plus a fixed table, so the shared suite can carry it: a record declaring admin and аdmin rejected, one declaring id_пользователя alone accepted, l beside I rejected. That is the first part of this topic that could ever have been conformance-tested.

Open for discussion: which of these is a conformance requirement, and which is deployment policy

Steps 1–5 say what the rules should be. None says what kind of requirement each is, and building all four made the question unavoidable: they do not have the same character, and treating them alike costs something real either way. Two properties separate them.

Does the verdict survive a Unicode upgrade? §7.1 makes a promise the later steps quietly break:

Growth is monotone — characters that were lexer errors become token characters, and valid documents remain valid under later versions.

That rests on Unicode’s stability guarantee for XID_Start/XID_Continue/Nd/Pattern_White_Space/ Pattern_Syntax, which §7.1 cites directly. UTS #39 makes the opposite statement about both datasets Steps 2 and 5 need:

Stability is never guaranteed between versions, although it is maintained where feasible. In particular, an updated version of confusable mapping data may use a mapping for a particular character that is different from the mapping used for that character in an earlier version. Thus there may be cases where X → Y in Version N, and X → Z in Version N+1, where Z may or may not have mapped to Y in Version N.

The Identifier_Status does not have stability guarantees (such as “Once a character is Allowed, it will not become Restricted in future versions”), because the data is changing over time as we find out more about character usage.

So making Step 2 or Step 5 a validity rule costs §7.1’s monotonicity claim, and costs it in the worst available place: §2.2.1 content-addressed identity. A schema pinned by ?sha256= is the same bytes forever; if its validity depends on data Unicode declines to freeze, those same bytes can change verdict under an implementation’s routine UCD refresh. Making the bytes decide is the entire purpose of the pin. Step 1’s profile has no such problem — it is built on the frozen properties, which is why §7.1 can say what it says.

Does the rule compose? Step 2 is a relation over a set, and §2.2.3 makes that set span !!import. The consequence is demonstrable rather than theoretical — two schemas, each accepted alone, both pure Latin, both well inside every other rule here:

a.tn declares  list_item                      accepted alone
b.tn declares  Iist_item   (capital I)        accepted alone
b.tn importing a.tn                           REFUSED — the two share a skeleton

Neither author wrote a bad schema, and the fix is to rename a name in a document the importing author may neither control nor republish. As a validity rule this makes a schema importing two independently published, independently pinned schemas impossible to write; as a policy rule its operator relaxes the check and proceeds. Steps 1, 4 and 5 are properties of a single name and compose freely — Step 2 alone has this shape.

And Step 2 is not free of false positives, including in pure ASCII. UTS #39 maps m → rn (confusables.txt: 006D ; 0072 006E ; MA). Over the 234,289 pure-ASCII words in the macOS dictionary, 58 skeleton clusters collide, every one of them through that mapping: comer/corner, comet/cornet, homer/horner, yam/yarn. Two innocently chosen ASCII names can therefore collide with no attacker anywhere. The rate is low, and a namespace holding both members of such a pair is unlikely — but “unlikely” is a sound basis for a default and an unsound one for a validity rule. Step 3 contrasts the closed-scope skeleton check favourably against the restriction level’s false-positive rate; that contrast holds for the mixed-script names it was measured on, and not for this case.

Recommendation — two layers, and the line between them is not where the steps put it.

Layer 1 — the identifier grammar. MUST, and it is validity. Step 1’s profile: XID_Start-initial, XID_Continue ∪ { - }, NFC. Determinate, local, and built on properties Unicode has frozen, so every implementation at every Unicode version returns the same answer. This is the only layer a pinned schema’s verdict can safely rest on, and the one that must be identical everywhere for a schema to mean one thing.

Layer 2 — name hygiene. Implemented everywhere, enforced by default, and never validity. Steps 2, 4 and 5. A conforming implementation:

• MUST implement all three;
• MUST enforce Steps 2 and 5 by default, and SHOULD default Step 4 to Highly Restrictive;
• MUST report a Layer 2 refusal distinguishably from a validity error;
• MUST allow a deployment to relax any of them through the implementation’s own configuration, and MUST NOT allow that relaxation to be silent;
• MUST name the UTS #39 data version in a Layer 2 refusal — two implementations can legitimately disagree, and the version is the only thing that explains it.

A schema failing Layer 2 is refused by this processor, not invalid. That one distinction buys all four of: safe by default everywhere, portable verdicts where §2.2.1 needs them, an escape hatch for the import diamond above, and no caveat on §7.1’s monotonicity.

Step 4 is the one whose setting the spec should not fix, only its vocabulary: the level and the unit are both deployment choices with no single right answer, and the spec’s job is to define them precisely enough that two implementations configure comparably. Mandating the knob while leaving its position to the deployment is the useful half.

This revises Step 3. Step 3 asks for “a MUST at the !!import merge and over one schema’s declared names”. The mechanism is right and the modality is not: the import merge is exactly where the compositional hazard bites, so it is the last place a hard validity rule belongs.

Four things the spec must settle either way, each an underspecification this implementation had to guess at:

1. Do profile extension characters participate in Layer 2? - is in the identifier profile, is not XID_Continue, and has no Identifier_Status. This implementation exempts it. Every implementation otherwise guesses, and they will not guess alike.
2. Does the namespace scope span !!import? This implementation checks the merged namespace, which is what produces the demonstration above. The answer should be yes — the diamond is precisely where a confusable pair is dangerous — and that is the strongest single argument for Layer 2 being policy rather than validity.
3. Choice variants are not a scope, for the reason recorded below.
4. Whether a Layer 2 refusal is reportable as such. It must be, per the recommendation — and this implementation does not yet do it consistently.

What the spec should not do: pin a Unicode version (that freezes the format to a UCD release and makes every later script a breaking change), or permit a relaxation that leaves no trace.

Open questions before this is drafted: whether Layer 2 belongs in the normative body at all or in Security Considerations carrying normative language, given it would be the first place either part states a MUST that is not about validity; and whether “MUST implement, MAY disable” is too heavy for a constrained implementation, given Step 2’s 706 KB — Step 3 argues the implementer should pay that once, and the argument is unchanged, but it was made when the rule was going to be validity.

Status against Revision 33: open, and Steps 1–1d, 2, 3 and 5 are built. The kernel carries identifier (XID_Start-initial, XID_Continue ∪ { - }, NFC) with type_name/field_name/param_name aliasing it and enum_set => !set { element_type: identifier min_items: 1 } feeding enum.members; token/token_set are gone. IdentifierParser enforces the contract at every naming position (below), and DefinitionResolver asserts it for declared field names, which the resolver builds directly rather than reading back as data. Skeleton distinctness runs over the merged namespace, each record’s field names and each enum’s members, reported as CONFUSABLE_NAMES; a Class 1 record’s own fields are checked by the schemaless reader, which is the one scope with no declaration behind it. Identifier_Status=Allowed is on the profile. §9.4 itself is unchanged — one SHOULD-consider sentence, no comparison scopes, no stated action on detection — so what is built is this entry’s proposal, not the spec’s text.

One scope in Step 2’s list turned out not to be one. A choice’s variants are references to declared names, so two confusable variants are two confusable entries in the namespace and are reported there; a check over variants could never fire, and the list above should drop them. Field names and enum members are genuine scopes because neither is a declared name.

Step 4 is built on the identifier surface and not on the token surface. TsonUnicodePolicy carries §5.2’s six levels, the unit, and additional permitted script sets; TsonConfig.identifierPolicy reaches TsonSchemaLinker through the compiled meta registry, and the level is applied in the same pass as the confusable check — over the namespace, each record’s field names and each enum’s members. The default is Highly Restrictive over a whole name, so an ordinary compound like id_пользователя is refused until a caller reaches for perSegment().

Step 4b is built too, so both surfaces now carry a policy. TsonConfig.tokenPolicy defaults to Unrestricted and reaches both facades, with withTokenPolicy as the per-reader axis — it has to be a reader axis rather than a registry one, since the standalone schemaless constructors hold no registry and a Class 1 read is where a value arrives least constrained. TsonReadContext.of takes the policy as a required parameter and installs the check itself, so no context can exist whose events went unchecked and the low-level API cannot drop the policy by saying nothing — naming unrestricted() is a fine answer, and is what the synthetic internal sites give, but it is not one a caller gives by accident. The check is a decorator on the event source so each token is judged exactly once despite the context’s rewinding, and nothing is installed at all when the policy checks nothing. Violations report as RESTRICTED_TOKEN, located at the token and carrying no path — there is none yet where the check runs. A per-segment policy is refused on this surface rather than ignored. The subsumption predicted above holds and is pinned by a test: a field name is a token, so tokenPolicy(asciiOnly()) constrains names whatever identifierPolicy says.

Every naming position matches the profile, and the check sits where the production is parsed. §7.6 is the model and says so for numbers — “the number grammar applies to the complete text of a token; it is not part of the token-stream grammar” — so a name’s decoded text is matched the same way, against the profile rather than against the lexical class it was spelled in:

• type-ref = "!" identifier and annotation = "@" identifier in data. TsonDataStream matches each name once adjacency is settled, so !42x and @x.y:1 are syntax errors rather than a reference to a type nobody declared and an annotation carrying a name the format reserves. It has to be the profile and not the token class, token-Start carrying Nd/-/+/. only so a number can be an unquoted token.
• type-name = identifier in the schema grammar, covering declared names, type parameters, referenced names and the ! constructor head. This replaces [TSON-SCHEMA] §12.1’s separate “numbers are not declarable names” rather than joining it, which is Step 1b’s subsumption claim built: identifier-Start is XID_Start, every number spelling begins with a digit, a sign or a dot, and the one rule also catches 42x and -foo, which merely begin like a number and which §12.1’s rule admitted.
• field-name = unquoted-token / single-line-token, narrowed from all three token forms. A map key keeps all three, being a value and not a name (§2.6), and the two constructs are told apart by : versus =>.

One part of Step 1 is deliberately not built, and reads apart from that narrowing: field-name stays lexical, so Class 1 field names are unconstrained exactly as Step 1c intends — the identifier contract is stated once on declarations, and data conforms by construction, a field name that is no identifier matching nothing and already being an UNRECOGNIZED_FIELD. {"first name": 1} therefore still reads as an ordinary record and JSON compatibility is untouched. The joiners’ contextual rule is built — Lexer no longer subtracts ZWNJ/ZWJ, and JoiningControls applies UTS #39 §3.1.1.1’s three contexts to a name, which is the layering this step always described (#14).

Layer 2 is not yet reportable as such here. Diagnostic.Code.CONFUSABLE_NAMES exists and is emitted for exactly one scope — a Class 1 record’s own field names, checked by SchemalessTreeReader because no declaration stands behind them. Every schema-side Layer 2 failure, the confusable check and the restriction level alike, goes through TsonSchemaLinker’s report, which builds a SCHEMA_ERROR. So one defect carries two codes depending on whether a schema governs the document, and on the schema side it is indistinguishable from an ordinary validity error. Under the recommendation above that is the first gap to close, and it is worth closing whatever the spec decides — the inconsistency is this implementation’s own.

4. A type argument’s literal is called a bare token and typed value, and §8.2 identity depends on which

Section: Part 2 §5.10 (type arguments), §8.1 (type_argument), §8.2 (instantiation identity); Part 1 §4 (base type resolution), §7.6 (number). Related: change log #43, and D6 of the structure-templates CR, now folded into Revision 33 as its baseline.

Problem: three statements about the same slot, which do not agree.

§5.10 and §8.1’s prose describe a type argument’s literal value as a bare token — “never annotated, never typed, never a container”. A bare token is text plus the form that produced it, unresolved.

meta-kernel types the slot as value:

type_argument => { ( name: type_ref | value: value ) }

and value is the escape-hatch primitive whose whole contract is that [TSON-DATA] §4 base type resolution has been applied to it. A token and the value it denotes are not the same thing, and this slot is declared as both.

§8.2 then keys an instantiation entry on “structural equality of the flattened, fully-bound application recorded in source”. Structural equality of what? The two readings differ, and §4 is where they part: 255 and 0xFF are the same number, so

• as tokens, vector<float32, 255> and vector<float32, 0xFF> are different applications, hence two entries;
• as values, they are one application and one entry.

This is not hypothetical. Reading the token, the two produce entries whose bodies are byte-identical:

vector<float32, 255>   → array_float32_255_255_5c5f53f7    ArrayBody[minItems=255, maxItems=255]
vector<float32, 0xFF>  → array_float32_0xFF_0xFF_462d33b7  ArrayBody[minItems=255, maxItems=255]

The bodies agree because min_items is declared integer and decodes through that atom; the identities disagree because they are derived from the argument’s token text. So the same schema holds two entries for one type, and §8.2’s one-entry-per-application rule is satisfied only under a reading of “application” that §4 contradicts.

Note this is not an artifact of how the argument reaches the resolver. A declaration-level application (a => vector<float32, 255>) never goes near a wire form and splits identically, because the split is in what identity compares, not in how the argument was parsed.

And the cost is a wrong verdict, not a redundant entry — which is what moved this from a recorded consequence to a fixed one. §5.4 requires a choice’s variants to resolve to distinct types, and it can only ask that of entry names, so two names for one type pass a check two spellings of one name fail:

u => ( [float32; 255] | [float32; 0xFF] )     accepted, disjoint=false
u => ( [float32; 255] | [float32; 255]  )     "'u' lists the variant 'array_float32_255_255_…' twice
                                               -- §5.4 requires each variant to resolve to a distinct type"

The accepted one is worse than untidy: a choice between two structurally identical variants, correctly derived non-disjoint, that no untagged read can ever discriminate. Under the value reading a conforming resolver must refuse it, and under the token reading it must not — one schema, opposite verdicts, decided by a spelling §4 spends a paragraph making irrelevant.

Interpretation chosen: option 3 below. The slot stays a Token — §5.10’s “bare token” is the only prose that speaks to what is recorded, and resolved output still shows the author’s spelling — and §4.3’s equivalence is applied where identity is derived, at both naming sites (NumericIdentity, consumed by SchemaDesugarer’s lift and TemplateMaterialiser’s instantiation). This is a change of position: the split was previously accepted and recorded here on the grounds that normalising would be inventing a rule. The §5.4 evidence above is what settles it — leaving the two apart is not neutrality between the readings, it is the token reading, and it is the one that admits a schema no reader can use.

The equivalence applied is exactly the one §4.3 states, and no wider. Radix, digit separators and a redundant sign fall away (255/0xFF/0b1111_1111/0o377/+255); a float’s written scale does too (.5/0.5, 1.0/1.00/1e0); .inf and .infinity are one value. What does not fall away is the base type: §4 resolves 1 to an integer and 1.0 to a float, so those stay two arguments even though one magnitude covers both. A spec adopting option 3 should say that boundary out loud, since “the same number” alone does not decide it.

Suggested resolution: say which, in §8.2, and make §8.1 agree with it.

1. Identity compares token text and form. Then §5.10’s “bare token” is the operative reading, the kernel should type the slot token rather than value, and §8.2 should say plainly that two spellings of one number are two types — surprising enough to be worth stating, since §4 spends a paragraph making them one value.
2. Identity compares the values the arguments denote. Then the kernel’s value typing is right, §5.10’s “bare token” is about what may be written rather than what is recorded, and §8.1’s type_argument holds a resolved value — which also settles what an implementation binds it to.
3. Keep the token in the model and normalise before comparing — the slot stays a token so the original spelling survives into §8 output, and identity applies §4 first. This is the only option that keeps both the written form and the equivalence, and it is the one that most needs saying out loud, because no implementation will arrive at it from the current text.

Option 2 is the smaller edit; option 3 is the better answer if resolver output is meant to round-trip what the author wrote.

Status against Revision 33: open on the spec’s side; option 3 is built here, so the recommendation is a report rather than a proposal. §5.3 now says an unquoted token argument “is classified against the applied signature’s parameter kinds”, which settles what kind of thing an argument is but not the identity question this entry asks: whether <255> and <0xFF> are one application or two. This implementation says one, and refuses the §5.4 choice above accordingly. §8.2 still does not say, and until it does an implementation reading §5.10’s “bare token” at face value gets a different entry set and the opposite verdict on that choice — which is the interoperability cost of leaving it unstated.

Coupled to #5’s D6 merge, which is not obvious and is easy to decide by accident. D6 says eagerly-lifted synthetics that become “structurally identical under resolution” merge into one entry. Re-deriving a synthetic’s name from its resolved record — the natural implementation — normalises the value channel as a side effect and settles this entry in favour of option 2/3 without anyone choosing it. The two splits live in different channels of the same derived name (a reference argument that is itself an application, versus a value argument’s spelling), so an implementation that wants D6 without prejudging this one must re-derive from resolved references while leaving value tokens as written.

5. §5.10’s collection-slot boundary refuses what the kernel’s own vocabulary licenses, and it excludes the sum-typed result envelope

Section: Part 2 §5.10 (the two parameter kinds, and the collection boundary), §8.1 (template_argument, type_ref, record_field.value_param), §5.3 (the lift rule), §5.10.1 (regularity), §12.1 (instance-template). Supersedes the item declined at Revision 33 as #53. Read with #7, which widens reference.target — the one open body this design could not otherwise spell, and what makes its “every open entry is a constructor application” true rather than nearly true.

Problem: §5.10 says, plainly:

Collection-valued slots are not parameterizable — a parameter inside a collection-typed slot (an enum’s member list, a choice’s variants, a tuple’s elements, a record’s fields) has no open representation, and a declaration writing one is a resolver error at the declaration; this is a deliberate boundary of this revision, and nesting goes through a second named template instead.

That is not ambiguous, and this entry is not an ambiguity report. What is inconsistent is that the kernel’s own vocabulary licenses exactly what the prose refuses. type_ref’s @doc in meta-kernel says name is “the referenced type — or, within a template body, a parameter of either kind, read against the enclosing definition’s parameters list”, and choice is declared

choice => ~sum & { variants: [type_ref] }

so every variant position is already a channel licensed to hold a parameter, at any depth. !choice { variants: [T error] } is spellable in the vocabulary that describes resolved schemas and forbidden by the prose that describes resolution.

The refusal traces to one place, and it is not the constructor vocabulary. template_argument is ( param: param_name | value: value | type_ref: type_ref ) with no collection case, and §5.10’s uniformity clause requires every open instance body to be an instance_template. So a body the reference channel could have carried unchanged must instead be re-expressed in a vocabulary that cannot hold it. The boundary is a property of the chosen open representation, not of the problem being represented.

The asymmetry underneath it is worth stating on its own, because it decides how much mechanism the problem actually needs: a slot that holds names can hold a parameter for free, because a parameter is a name; a slot that holds an immediate value cannot. Type slots ride type_ref and never needed a spelling. Only immediate value slots did — min_items: N, format: F — and record_field.value_param is that spelling, already shipped, for exactly one node. §8.1’s stated reason for quoting type slots anyway (a parameter in a type slot “would have two spellings and body identity would depend on the choice”) buys a property that is obtainable by construction — make the body form a function of the body, and no entry has two spellings — at the price of the boundary.

The cost is not theoretical. result => <T> ( T | error ), the sum-typed result envelope, is the likeliest headline use of generic schemas and is inexpressible by rule. §5.10’s declared workaround does not reach it: there is no second template to name when the parameter is a variant, so the sum must be monomorphised by hand at every use.

Interpretation chosen: implemented as written first — SchemaDesugarer refused a parameter in a collection-valued slot at the declaration, classified as a schema-author error rather than a library gap, the verdict being one that does not change as this implementation improves. That refusal is now replaced by the design below, built here and running: result => <T> ( T | error ) resolves, closing to an ordinary choice body over [text, error]. This is a deliberate divergence from Revision 33, offered as evidence rather than as a conformance claim.

The alternative weighed and rejected was the other coherent completion: grow the typed quotation until it covers what the constructors can express — a collection case on template_argument, and a shadow spelling for every slot kind a parameter can reach. It is rejected on proportionality, and the reason generalises: every per-channel mechanism in the shipped design is compensation for binding too early, so completing the quotation adds a spelling per constructor form in perpetuity, where holding removes the need for any. The one property uniform quotation buys — body identity not depending on which spelling an author chose (§8.1’s stated rationale) — is obtainable by construction instead: make the body form a function of the body, and no entry has two spellings. What that obligation becomes under holding is the one-spelling rule stated near the end of this entry, which is a requirement on producers rather than a vocabulary.

Suggested resolution: hold an open body rather than quoting it. What follows is the design as built, so that what is proposed and what is known to work are the same thing.

1. An open entry’s body is the constructor application as written, held and unread until materialisation substitutes its parameters away. Not a typed quotation of the constructor vocabulary. Substitution is then one rule at any depth — rewrite a token whose text resolves into the entry’s parameters (§8.1’s shadowing rule) — uniform across type slots, value slots, collection elements and nesting alike, because nothing has been classified by slot kind. Materialisation binds against constructor vocabulary exactly once, at the only moment binding is decidable.
   • A held token needs no channel label, and that is what removes the boundary. template_argument needs param because a bare token in a value slot is otherwise always a literal; a held body is not read as that vocabulary until the parameters are gone, so a parameter in variants is a token inside an array like any other. Quoting is no part of it — a token’s form is a schemaless-data concern (§4.4), which is why the rule is on the token’s text.
   • The cost is shadowing’s usual one: inside a template, a literal spelled like a live parameter is unreachable.
2. §12.1’s instance production takes a parameter list, and the instance-template / template-def / template-bind productions delete with the vocabulary that motivated them. Open and closed share one production and one payload grammar, which is what admits a collection payload. A parameterized atom-refinement remains no form at all, as §12.1 already has it.
   • What no payload can spell is an application, in either form: !array { element_type: box<text> } does not parse, box<text> being schema grammar where instance takes a core-value. The line falls where the grammars already divide — a type position takes box<text> directly, while !C value takes data, so an application inside one is written in type_ref’s own record form, which is what the sugar expands to anyway. Revision 33 gave the open form a spelling the closed form never had; losing the asymmetry is part of the point.
3. Lifting is unchanged, and open synthetic entries remain a category. This is worth stating because the opposite is the obvious guess and it does not work: a template’s body cannot simply hold everything nested inside it. A type_ref slot names a type and nothing else, so a sugar form inside a template body — <T> { a: [T] } — must still lift to an entry, and a lifted form naming a parameter lifts open. §5.3’s lift rule therefore stands as written, and §8.2’s identity-up-to-consistent-renaming of parameters is still required, since two open synthetics alike up to renaming must land on one entry.
   • What changes is only what an open synthetic’s body is: held, not quoted. <T> { a: [T] } still injects array_p0_… => <p0> !array { element_type: p0 } and the field still reads array_p0_…<T>.
   • Applications inside a held body close before its entry is named. Desugar lifts innermost-first, so a form it writes already names the entry its inner form became; a form closed at materialisation must agree, or [[pixel; 3]; 3] written out and grid<pixel, 3> closed land on two entries for one type.
   • Not built here, and a spec question rather than an oversight. The two lift channels hash different things: a closed lift hashes the binding record at desugar, before its inner applications are rewritten, where the open lift hashes the closed record at materialisation. So [box<text>] written directly and [box<T>] closed with T := text land on two entries for one type in this implementation. The resolution is already in §8.2’s own D6 — “eagerly-lifted synthetics that become structurally identical under resolution merge into one entry” — a merge pass at the end of resolution that re-derives each synthetic’s name from its resolved record. It was never needed before, every form lifted closed having been concrete at desugar, and holding is what makes it reachable. §5.10 should say that the merge is required rather than incidental, because an implementation reading D6 as an optimisation will skip it and get a second entry for the same type.
4. The resolved form of an open entry is its declaration, not a type_definition value — which could not carry it in any case, the kernel declaring body: top REQUIRED with no top an open body could be. This keeps resolved output a valid schema document under §12.1, so it stays re-resolvable, and it needs no new kernel vocabulary: no new primitive, no ( body | template ) field group. §1.3 is unaffected, a conforming consumer of resolved output meeting only closed entries and instantiations.
   • It does, however, retire three declarations, which is a subtraction rather than an addition and is the other half of adopting this design. instance_template and template_argument exist only to quote an open body slot by slot; a held body has no producer for either. record_field’s ( value | value_param )? group narrows to value? with them — a routed parameter rides value, told from a literal by §8.1’s shadowing rule, and a held body is not read as this vocabulary until its parameters are gone, so the label has nothing left to disambiguate. All three are gone from the kernel shipped here, and §5.10/§8.1 should drop them when the design lands.
5. Checking splits, and §5.10 should say where. Two questions are answered at the declaration from the binding record’s own field names, needing no stand-in values and so unable to fabricate a verdict: that each name is a field the constructor declares, and that every REQUIRED-without-default field is bound. §5.10’s unreferenced-parameter rule is answered there too, from the tokens the held body names. Everything value-shaped waits for materialisation, where the whole body binds through the constructor’s own reader. An unapplied template is checked no further and gets no verdict — not a warning, no verdict.
   • Checking an unapplied template by substituting stand-ins should be ruled out explicitly, because it manufactures false errors on exactly the slots this mechanism exists for: <N> !integer ^ { min: N max: 3 } is correct for every argument anyone passes and fails under a stand-in of 10.
   • A materialisation diagnostic must be located at the declaration whose text wrote the offending name, with the application as context. Deferred checking is survivable only if the author is sent to the line they can edit. This is a requirement rather than a report — what the design owes an author in exchange for deferring the checks — and it runs here.
     • The rule is not “the template”. box => <T> { v: T w: no_such_type } applied by holder belongs to /box: holder is correct, does not contain the name, and would be blamed once per applier, each under a different subject ('box<text>', 'box<int32>'). But box => <T> { v: T } applied as box<3> belongs to /holder, which wrote the 3. A blanket “locate at the template” sends that author to a declaration with no 3 in it, so the spec should state the rule over the name, not over the entry.
     • What that costs an implementation is one lookup, not bookkeeping through the minting phase. The offending name is the evidence: the declaration to blame is the open one whose held body mentions it, and there is none when the name arrived in the argument list. Tracking derived-entry lineage instead gets the alias case wrong — half => <B> pair<no_such_type, B> closes to an entry sourced on pair, which is faultless — so the name is both the cheaper and the more accurate key.
     • One defect earns one diagnostic, however many declarations apply the template; otherwise the count an author sees is a property of the schema’s callers rather than of the mistake.

What holding costs, and where §5.10 should say so. A held body has no slot types — that is what it is for — so every check keyed on which slot a thing sits in waits for materialisation, and one of them does not survive the trip. §5.10’s argument-kind rule (“a reference argument binds a type parameter, a literal binds a value parameter”) is enforced today by record_field.value_param, whose presence is what says this slot expects a value; where a parameter stands in an ordinary value slot, a type name substituted there is a token like any other. §5.10 should say which of its checks are declaration-time and which are materialisation-time under an open form, rather than leaving an implementation to discover that one of them is neither.

Built out, the loss is half the rule, and the wrong half to worry about. A literal applied where the body uses the parameter as a type is still refused, because the substituted token stands in a type position and nothing declares a type called 3 — the verdict arrives as an unresolved reference rather than as a kind error, but it arrives. Only the converse escapes: a type name applied where the body routes the parameter into a field’s value, which value (§4’s escape-hatch atom) accepts.

And the converse is better closed by a rule §5.2 already states than by the kind rule. meta-kernel’s own @doc on value says record_field.value holds “the type of fixed/default values, which must be the field’s declared type — a dependency the schema language does not express directly”. Enforce that and retry => <N> { attempts: int32 ~ N } applied as retry<text> fails because text is not an int32 — whether a parameter put it there or the author wrote it literally, and with no notion of a parameter’s kind involved. What it would not catch is a type name applied into a text-typed value slot, which is a value slot holding a valid value: no error to give. So the recommendation is to drop the argument-kind rule rather than find a home for the slot’s expectation, and to state the value-conformance dependency §8.1 currently describes as inexpressible. That removes the one check holding cannot carry, and removes it by strengthening a rule the format wanted anyway.

• The value-conformance rule is a report, not a recommendation: it is built and running here. The linker checks a field’s ~/= value against the field’s own resolved type, so { first: int32 ~ "nope" } is refused at the declaration that wrote it, and retry => <N> { attempts: int32 ~ N } applied as retry<text> is refused identically — one rule, no notion of a parameter’s kind, the same verdict whether a parameter or the author put the value there. The check runs the field’s own reader parser, so it accepts a value exactly when a read would accept the same token in that position and cannot drift from the atom contracts §5 defines.
  • Its boundary today is the field’s type kind, not the parameter. A field typed by an atom or an enum is checked; one typed by a record, container, tuple or choice is not, because checking a value against those needs the compiled reader and compilation runs after linking. That boundary is this implementation’s, not the rule’s: nothing in §5.2’s dependency is atom-specific.
  • So §5.10 can drop the argument-kind rule, provided §5.2 states the value-conformance dependency normatively — a resolver that drops the one and does not add the other loses both.

One position the loss does reach, and §5.10 should decide it. §5.10 settles a parameter’s kind from its use, and there is a use no channel recognises: an enum member. e => <M> !enum { members: [a b M] } applied as e<c> fails, because §12.1’s type-arg rule sends an unquoted non-numeric argument down the reference channel and c is a member name, not a type. It is the same root as the argument-kind loss above — a held body has no slot types, so nothing says members is a value channel — but it is the case where the loss produces a wrong verdict rather than a late one: the author is told c is an unresolved reference when c is not meant to be a reference at all. The two want settling together, and the choice is between naming enum.members a value channel in §5.10’s own kind table (so an argument reaching it is read as a literal), or saying that a parameter in a member list requires the quoted spelling e<"c">. Either is a sentence; leaving it unsaid means every implementation that adopts holding hits this on its first parametric enum.

Scope of what is built. Every template shape holds its body, and one process closes them all: the sugar forms, the explicit <T> !C { … }, record templates, and composition and refinement templates. §5.2 already says a bare record body denotes !record { fields: [ … ] }, so <T> { x: T } is normalised to that and closes the way <T> [T] does. A composition or refinement is resolved against its namespace first and the flattened record is held — a §5.7 tightening entry states a modifier and no type-ref, so it is not a record_field until the inherited field supplies one — which is the one reason those two are normalised a phase later than the rest.

And what is not built, gathered here so a reviewer has the boundary in one place rather than in footnotes: a parametric enum member (above), locating a held-body defect at the template’s declaration (above), and the D6 merge that would make the two lift channels agree (above). Value conformance of a field’s ~/= against its declared type — this entry’s proposed replacement for the argument-kind rule — is not among them: it is built, and #8 carries the one question it raises that this entry does not. A parameterized atom-refinement is deliberately not on that list: it remains no form at all, as §12.1 already has it, and holding does not change that. Nothing on the list is load-bearing for the design — each is a check or a location, not a shape the held form cannot express — but a reviewer adopting this should know which claims here are running code and which are recommendations.

So record_field.value_param has no producer left, and §5.10 can retire the channel along with instance_template and template_argument. A routed parameter rides the ordinary value slot with §8.1’s shadowing rule to tell it from a literal, and §5.7’s fixation moves to materialisation — where §5.7 already says it belongs (“fixation happens downstream, where values are concrete”). The kernel’s record_field group narrows from ( value | value_param )? to value?.

One implementation note the spec should absorb, because it is a property of the design and not of this codebase: the open form needs one spelling, however many phases produce it. An open body is read by later phases as wire form, and an entry’s derived name is a hash of what is written, so two spellings of one form are two entries for one type — and worse, a serialiser that states a no-argument type_ref in the explicit record form where the sugar table states it positionally makes type_argument and type_ref indistinguishable to a walk that reads neither against a vocabulary. The trap is concrete: an ordinary canonical-explicit object writer is exactly the wrong producer, and not only for that reason — it quotes every token, where a held body’s whole parameter mechanism keys on a token being unquoted, so a written-out body references no parameters at all. Two of the four shapes here genuinely cannot be normalised syntactically (composition and refinement need a namespace to flatten against), so “produce it in one phase” is not achievable; “produce it in one spelling” is, and is what §5.10 should require.

If Revision 34 wants a smaller edit than all of that, one scoped change resolves the flagship case against Revision 33 as shipped: restate §5.10’s uniformity rule so that an open entry carries an ordinary constructor body whenever every parameter occurrence sits at a type_ref position, requiring instance_template only where a value slot is parameter-bound, and narrowing the collection error to parameters at value positions inside collections. Choice, tuple, [T] and {K => V} templates fall out immediately. Note it is not free for implementations that shipped Revision 33: it changes the resolved output of templates that already work, <T> { v: [T] } ceasing to lift an instance_template.

Status against Revision 33: open, new against this revision. The same gap was raised against Revision 32 as #53 and declined, §5.10 gaining the explicit boundary sentence and §8.1 the uniform-quotation rationale in response. The design above is implemented here and passing: the flagship result => <T> ( T | error ), <T> [T, text], <T> { v: (T | text) } and nested sized forms all resolve, every template shape holds its body, value_param has no producer left, and every schema that resolved before produces the same entries it did. What is not built is listed under “And what is not built” above, and every recommendation this entry makes that is a proposal rather than a report is marked as one where it is made.

6. Every schema that writes a container sugar form inside a template mints its own copy of the same few templates

Section: Part 2 §5.3 (the lift rule), §8.2 (synthetic entry identity and content-derived naming), §9 (what the kernel declares). Related: #5, whose held-body proposal does not change this either way.

Problem: a sugar form inside a template body lifts to an open synthetic entry — <T> { a: [T] } mints

array_p0_358380cd => <p0> !array { element_type: p0 }

and box’s field references it as array_p0_358380cd<T>. That entry is the same entry in every schema that writes [T] inside a template, up to a content-derived name §8.2 already declares non-normative. The lift rule mints it per schema because it has nowhere else to put it, so a fixed, tiny set of templates is re-derived by every author who uses generics over a container.

The kernel already takes the other route one level down: rather than have every schema inline !set { element_type: token }, §9 declares token_set once and enum.members references it. The same argument applies to the open forms, and nothing but availability decides it.

Interpretation chosen: mint per schema, as §5.3 specifies. SchemaDesugarer injects the lifted declaration into the document being desugared, with positionalNames/rename alpha-normalising the parameters so that two spellings of one form land on one entry within that document.

Suggested resolution: consider declaring the fixed-arity open forms in the kernel — <T> !array { element_type: T }, its state: OPTIONAL sibling, and the map pair — so that §5.3’s lift targets a declared name rather than an injection. Two things to weigh, both real:

1. Only part of the family is fixed-arity. The size specifier’s variants differ by which bounds are present ([T; 3], [T; 1..], [T; 1..2] are three shapes, since an absent max_items is not a defaulted one), and tuple and choice are variadic, so [T, U] and ( T | error ) have no fixed-arity template at all. A kernel set would cover the commonest case and leave the lift rule in place for the rest, which is a smaller win than “declare them once” suggests.
2. Availability is the hard part. A schema’s type-name namespace is its own declarations plus its !!imports (§3.3.1, §2.2.3); it does not include the namespace of the schema its !!meta names. So a kernel-declared array_of is not in scope for a schema that has not imported the kernel, and a lift targeting it would make desugaring — a phase whose whole virtue is being syntactic, consulting no governing meta and no namespace — depend on the import set. Either §5.3 would have to name these as always-available regardless of import (a new category of name), or they would have to live somewhere every schema already reaches.

Note this is not a proposal to re-parameterize array/set/map: those stay de-parameterized constructors with element_type as an ordinary field, and what is proposed here is named templates over them, which is the layer a user’s own box => <T> { ... } lives in.

Status against Revision 33: open, new against this revision. This implementation mints per schema and ContainerSugarEndToEndTest pins the resulting entry sets.

7. An alias to an application cannot state the arguments it binds, so reference is the one open body that is not a constructor application

Section: Part 2 §8.1 (reference), §5.10 (partial application), §8.2 (identity keyed on source), §8.3 (use-site flattening), §9 (a slot holding a type reference). Read with #5, whose held-body design this completes: an alias is the one open entry that could not be written as a constructor application until target widened, so #5’s uniformity claim rests on this entry being adopted with it.

Problem: §5.10’s partial application is an alias to an application:

uuid_pair => <B> pair<text, B>

The kernel gives it nowhere to say so:

reference => top & {
  target: type_name      ← a bare name; `pair<text, B>` does not fit
}

So an implementation has to keep the argument list somewhere else, and the only place available is the entry’s own source — which §8.2 keys instantiation identity on. That gives one component two jobs depending on whether the entry is open, and it makes an alias with no recoverable target a representable state: nothing in the vocabulary says a reference entry must carry a source, so nothing prevents one that cannot say what it aliased.

It is also inconsistent with §9’s own guidance for extension meta-schemas — “a slot holding a type reference MUST be typed type_ref” — which reference.target plainly is and plainly is not.

The deeper cost is uniformity. Every other open entry is a constructor application, <params> !C core-value, which is what lets substitution be one walk: rewrite the parameter tokens in a held core-value, then read the result through the named constructor. A partial application is the one shape that cannot be written that way, because !reference { target: pair<text, B> } is not spellable while target is a name.

Interpretation chosen: widened to target: type_ref here, and the alias states its own arguments. That deletes the source double-duty, the guard against an alias with no recoverable target, and the special case that kept a name-only body in step with source whenever materialisation rewrote it.

Suggested resolution: declare reference => top & { target: type_ref }.

• A closed alias never carries arguments, so resolved output is unaffected in practice: materialisation rewrites text_box => box<text> to name the entry it minted. An argument-bearing target appears only where an application is still open — inside a template.
• §8.3 needs one sentence. A use site is flattened past a REFERENCE entry; this slot is not, because the chain must stay walkable and an alias records where it points. The walk additionally stops at an argument-bearing target: that is an application, not a hop to another entry, and there is no entry at the end of it until materialisation mints one.
• What it unlocks, and what is built here, is writing the partial application as <B> !reference { target: pair<text, B> }. That brings the last template shape onto §12.1’s one open-form production: every open entry is now [type-params] "!" type-name ws core-value with a held core-value body, so substitution is one token walk for all of them and a resolver tells the cases apart by the constructor head — record closes to the instantiation entry, reference composes and mints nothing (§5.10’s “no intermediate entry per alias hop”), everything else closes to a synthetic.
• Two kernel facts make reference a dispatched head rather than an ordinary one, and §5.10 should say so if it adopts this. reference is deliberately not a ~ constructor — it describes no value — so the generic “!C value requires a constructor” rule refuses it; and §4.1 gives an alias kind: REFERENCE, which is a type_kind with no base kind in the composition hierarchy to supply it. Neither is a property of the alias form; both are the kernel’s own, and an implementation has to special-case the head either way.

Status against Revision 33: open, new against this revision. Implemented here, which makes it the first change to the bundled spec/m/ artifacts’ content — the three digests move with it.

8. §5.2 makes a field’s fixed or default value a value of the field’s declared type, but does not say which declared types can have one

Section: Part 2 §5.2 (record_field.value, the six field-state spellings), §5.6 (positional form), §12.1 (field-modifier). Part 1 §7.4 (form is not meaning).

Problem: §12.1 admits only a bare token after ~/= — writing ~ [ ... ] or ~ { ... } is a syntax error, not another value — and meta-kernel’s own @doc on record_field.value says the slot holds “the type of fixed/default values, which must be the field’s declared type — a dependency the schema language does not express directly”. Put together, those settle every scalar case. They do not settle whether a field whose declared type is a record or a choice may carry one, because a bare token can legitimately reach both:

point => { n: int32 }        rec => { p: point ~ 3 }        # §5.6 positional form
ch    => ( int32 | text )    rec => { c: ch ~ oops }        # discriminates to the text variant

Both read cleanly if a resolver admits them: §5.6 fills a record with exactly one bare REQUIRED field from a bare value, and a choice discriminates a token by its §4 base-type class to a variant that accepts it. Nothing in §5.2 says whether the “must be the field’s declared type” test is satisfied by a value the type admits or by a type a token denotes directly, and the two answers differ for exactly these two kinds.

What this implementation does: refuses both. A fixed or default value is available on a field typed by an atom or an enum and nowhere else — TsonSchemaLinker resolves the field’s type, and a body that is not a scalar is a resolver error at the declaration, whatever token stands beside it. void, unknown and extern fall out of the same rule for their own reasons: the type with no value, the universe of types, and a mechanism with no token shape.

Why, and it is a cost worth naming. Admitting the two cases makes “may this field have a default?” depend on the referenced type’s field count (exactly one bare REQUIRED) and on its variant list and their discrimination classes. That is a rule an author computes rather than remembers, and it is computed against a different declaration than the one they are editing — adding a second field to point would silently invalidate a default written on a field somewhere else. The refusal costs two spellings that would have worked and buys a rule that fits in one line. §5.6 is a spelling rule for data values; reading it as a claim that a record is a token is what would carry it into a schema’s own field modifiers, and it does not say that.

Recommendation: §5.2 should state which declared types may carry a fixed or default value, in one sentence, rather than leaving it to be derived from §12.1’s token-only production plus §5.6. Either answer is implementable; what costs an implementation is that the question is not asked. If the answer is the permissive one, §5.2 should also say that a record’s positional-form eligibility is part of its contract — because a default written against it then breaks when an unrelated field is added to that record, which authors will not expect from a change that is otherwise backward-compatible.

Status against Revision 33: open, new against this revision. The restrictive reading is what is built and running here (FieldValueConformanceTest); the permissive one is what a resolver gets by deferring to its reader, which is the shape an implementation falls into by accident.

9. time_type/datetime_type declare precision and require_timezone, and no prose anywhere says what either means

Section: Part 2 §9 (the bundled meta.tn artifact), §5.5/§5.7 (constraint vocabularies and tightening). Part 1 §5.4 (the temporal atoms), §5.2 (an atom’s parse/validate split).

Problem: meta.tn declares both facets, normatively:

time_type => ~atom & atom_specification & {
  spec:              = "https://www.rfc-editor.org/rfc/rfc3339"
  min:               value?
  max:               value?
  precision:         integer?
  require_timezone:  boolean?
}

datetime_type is identical. Neither field carries a @doc, and neither name appears anywhere in Part 1 or Part 2’s prose — the only mention of time_type/datetime_type outside the artifact itself is the §9 table listing them as constructors. So the vocabulary a conforming schema may write is defined, and what writing it means is not. An implementation must pick, and every pick is a different accept/reject set.

precision, four ways to read it, and they are not equivalent:

1. Exactly N fractional digits, or at most N? precision: 3 against 12:00:00.12 — accept or reject?
2. Does it constrain the written token or the value? 12:00:00.100 and 12:00:00.1 are the same instant with different digit counts. A token-level reading separates them; a value-level one does not.
3. Is it a validation constraint or a truncation instruction? Part 1 §5.6 sets the precedent that the approximate atoms round onto their grid and “loss of precision is expected, not an error”, so a reader could reasonably expect precision to truncate rather than reject. The temporal atoms are not on that split, which leaves the question open rather than answered.
4. What is precision: 0? No fractional part admitted, or unconstrained?

The choice is not cosmetic, because it decides whether the facet can participate in refinement at all (§5.7). Under “at most N”, a smaller N is a narrowing and precision behaves like every other bound. Under “exactly N”, two different values are disjoint rather than ordered, and a refinement tightening precision: 6 to precision: 3 is neither a narrowing nor a widening — a shape this vocabulary has nowhere else, and one an implementation’s narrowing check cannot express.

require_timezone is stranger, because the pinned spec appears to make it unusable. Part 1 §5.4 maps !datetime to RFC 3339 date-time and !time to full-time, and both productions make the offset mandatory. So require_timezone: true constrains nothing that the atom’s own format does not already require, and require_timezone: false can only mean accepting values the named format does not produce — partial-time, or a local date-time. That is a facet that widens an atom, against a spec field the same record fixes to RFC 3339. Every other facet in this vocabulary narrows. Either the field means something narrower than it reads, or it is the one place a constraint vocabulary relaxes its own specification pin, and §5.5 should say which.

What this implementation does: neither facet is enforced, and a schema setting either is accepted at load and refused at the first read of that type, with a message that names the ambiguity rather than resolving it — 'datetime' does not enforce 'precision' yet … the spec does not say whether it bounds the fractional-second digits exactly or at most, and this implementation will not guess. That is deliberate: the alternative is to pick silently, and a schema author who wrote precision: 3 would then get whichever of the four readings this implementation happened to choose, with no way to tell. Two of the six remaining read-time gaps in this implementation are these two facets, and they are the only ones whose cause is a question rather than unwritten code.

Recommendation: state both, in §5.5, beside the family they belong to. For precision, the useful answer is almost certainly “at most N fractional digits, at the token level, a validation constraint” — that is the only reading under which the facet orders, refines, and composes like the rest of the vocabulary. For require_timezone, either delete it (RFC 3339 already requires the offset, so the facet is vacuous) or say explicitly that false relaxes the atom to partial-time and reconcile that with spec being FIXED. Deleting it is the smaller change and loses nothing a schema can currently express.

Status against Revision 33: open, new against this revision. Both facets are declared in the bundled meta.tn this repo packages, so a schema can write them today and no implementation can agree with another about what happens next.

10. §8.2 defers “family coherence rules whose operands were parameters — min_items ≤ max_items (§5.3) and their kin” without saying what the kin are

Section: Part 2 §8.2 (materialisation), §5.3 (size specifier), §5.5 (constraint vocabularies).

Problem: §8.2’s sentence is normative and its subject is a set it does not enumerate:

Materialisation also runs the value-level checks that open bindings deferred: family coherence rules whose operands were parameters — min_items ≤ max_items (§5.3) and their kin — and the typing of every substituted value (§5.10) are verified once concrete, and a violation is a resolver error reported at the materialising application.

One rule is named and the rest are gestured at. An implementer can build the named one and has no way to know whether they have finished — “and their kin” is not a set anything can be checked against, and the obvious reading (that it means the other bound pairs) understates it, since a family may state a coherence rule that is not a bound pair at all (min_prefix/max_prefix against the address family’s own width).

What this implementation does, and it needs no list. Every constraint family already owns its coherence rule for the body an author writes literally — that is where min_length: 10 max_length: 3 is refused, and where §7.2 puts it (“family coherence between bindings (e.g. min ≤ max) is a compilation and ingest concern”). The rule an application closes onto is the same rule over the same facets, so materialisation does not need its own set: it asks every family the question it already answers. One call, at the phase that sees every entry exactly once, covers min_items ≤ max_items and its kin together, and a family that gains a rule later is covered without anything being added.

This is not hypothetical for the atom families. §12.1 refuses a parameterized ^ refinement, but the constructor spelling is open, so an atom’s own bounds reach materialisation as readily as a container’s:

b => <N> !integer_type { min: N  max: 3 }      r => { x: b<10> }
b => <N> !text_type { min_length: N  max_length: 3 }   r => { x: b<10> }
b => <N> !cidr4_type { min_prefix: N  max_prefix: 8 }  r => { x: b<40> }

All three describe a type nothing can satisfy, none of them is min_items ≤ max_items, and all three are the kin the sentence has in mind.

Recommendation: replace “and their kin” with the general rule, which is shorter than a list and cannot go stale: every family coherence rule §5.3 and §5.5 state applies again at materialisation, over the operands that were parameters. If a list is preferred, it must include the non-pair rules, which the current phrasing reads past.

One thing §8.2 gets exactly right and is worth keeping: “a violation is a resolver error reported at the materialising application”. That is the only location an author can act on — the entry itself is minted, content-named, and appears nowhere in their file — and it is also where §5.10’s substituted-value typing has to land for the same reason.

Status against Revision 33: open, new against this revision. The general rule is what is built and running here (ContainerBoundCoherenceTest), across both base kinds and every family.

11. Part 1 §2.8 resolves an empty brace to “the empty container of that type”; Part 2 §7.7 enumerates two containers, and the series has four

Section: Part 1 §2.8 (brace disambiguation and empty braces), Part 2 §7.7 (resolver behaviours at typed positions), §5.3 (the container sugar forms and their size specifiers).

Problem: the two parts state the same rule at different widths, and the wider one is the one that reads like the general statement.

§2.8 defers an empty brace to the resolver and closes with: “In the absence of declared type information, an empty-brace resolves to an empty record. When a higher part supplies an expected type ([TSON-SCHEMA]), it resolves to the empty container of that type.” Nothing there is limited to two kinds; “that type” is whatever the position declares, and §5.3 gives the series four container kinds — record, map, array, tuple.

§7.7’s “Empty braces” paragraph is the higher part supplying it, and it enumerates instead: “the resolver transforms an empty-brace value into an empty record or empty map per the expected type, defaulting to an empty record when the position is untyped.” An array position is not in the list, and §7.7 does not say whether that is a deliberate exclusion or an enumeration of the two cases the author had in mind.

Both readings are defensible and they differ observably on a one-line document. Under holder => { tags: [text] }, the data !holder { tags: {} } is either an empty array (§2.8 read generally) or a type error (§7.7 read as exhaustive). Neither part says which.

Three things make the narrow reading the more likely intent, none of them decisive:

1. {} is brace-shaped and an array is bracket-shaped. Every other empty container has a spelling of its own — [] for an array and a tuple — so nothing is unspellable under the narrow reading, where an empty map genuinely needs {} because it has no other empty form. The rule earns its keep only for the two kinds that share the {...} form and cannot be told apart when empty.
2. §5.4’s discrimination classes already treat {} as brace-class, ambiguous between a record and a map and distinct from bracket. Admitting {} at an array position would make a value of brace class conform to a bracket-class type, which is a wrinkle in a table the spec is otherwise careful to keep total.
3. A tuple would follow an array, and there §7.7’s silence is louder: a tuple’s arity is fixed and exact (§5.3), so {} at a two-slot tuple would resolve to “the empty container of that type” and then fail arity — a two-step verdict for what the narrow reading calls one type error.

What the narrow reading costs is that §2.8’s sentence is then wrong as written, in the part that defines the concept, and an implementor reading Part 1 first will build the general rule.

Interpretation chosen: §7.7’s enumeration, treated as exhaustive. {} reads as an empty record at a record position and an empty map at a map position (facing min_items/max_items there like any other map — the count is validated in MapAbstractReader.expectMapShape, the one funnel every map reader passes through); at an array or tuple position it is a TYPE_MISMATCH, reported as “expected an array for ‘[text]’, found {}”. Schemaless, and at any untyped position, it is an empty record, per §2.8’s own first sentence.

Suggested resolution: make the two parts agree, in whichever direction, and say it in Part 1 as well as Part 2 — the sentence an implementor builds from is §2.8’s, and it is the one that currently overstates.

If the narrow reading is intended, §2.8’s closing sentence wants replacing with something that does not generalise past it: “When a higher part supplies an expected type ([TSON-SCHEMA]), it resolves to the empty record or the empty map according to that type” — and §7.7 wants one clause saying an empty brace at any other container position is a validation error, so the exclusion is stated rather than inferred from a list.

If the general reading is intended, §7.7’s list wants replacing with §2.8’s own phrasing, and two consequences want stating outright, because both are places an implementation would otherwise diverge silently: an empty brace at an array position is a zero-element array and faces the size constraints ([text; 1..] rejects it, exactly as {text => text; 1..} rejects it today); and an empty brace at a tuple position resolves to a zero-element tuple that then fails §5.3’s exact-arity rule for any tuple with slots — unless §5.4’s brace/bracket classes are meant to exclude tuples and arrays from the rule after all, which would be the third possible answer and is currently unstated.

Status against Revision 33: open, new against this revision. The narrow reading is what is built and running here, across all four container positions.

12. §12.1’s grammar admits {K => V?}, §5.3’s prose forbids it, and map has no state field to bind it to

Section: Part 2 §5.3 (the container sugar forms), §7.6 (the absent sentinel under a schema), §12.1 (the map-type and element-type productions), §9 (the kernel’s map constructor); Part 1 §2.9.

Problem: four passages — three of prose and one of grammar — and they do not agree about whether _ means anything at a map entry value.

§5.3 refuses the marker and gives its reason:

Neither side of => admits ?: the kernel’s map has no state field, and absence has no defined meaning for map values (an absent key is already a resolver error, [TSON-DATA] §2.9).

§7.6’s table gives that exact position a defined meaning, and the only unconditional permission in the table:

| Map entry value (schema in scope) | yes | Entry present with an absent value ([TSON-DATA] §2.9); map carries no element-state facet, so the permission is not schema-conditional |

Part 1 §2.9 agrees with §7.6, twice and normatively: “_ may occupy any data-value position: record field values, map entry values, array elements, and the document’s top-level value”, and “A field or entry set to _ is present with an absent value — distinct from not appearing at all.”

So §5.3’s justification is contradicted by the section that states the rule and by the Part 1 clause both cite. Taken together the two live passages produce an author-visible incoherence: the value is optional, and the author has no way to say it is not.

And §12.1’s grammar is a fourth passage, which sides against §5.3. The map-type production draws its value from element-type, and element-type carries the marker:

map-type     = "{" ws map-key ws "=>" ws element-type
               [ ws ";" ws size-spec ] ws "}"

map-key      = type-name [ "<" type-args ">" ]
element-type = type-ref [ "?" ]

So the ABNF already admits {K => V?} and already forbids {K? => V} — map-key has no ? to write. §5.3’s prose contradicts the production directly on the value half, and merely restates it on the key half. That makes the defect statable at its sharpest: the grammar produces a marker the model has no field to bind. A conforming parser built from §12.1 alone accepts {K => V?}, reaches the desugar table, and finds map has no state slot to put it in; only §5.3’s prose stops it, and an implementation that reads the grammar first will not find that prose until it has already built the node. Two implementations of Revision 33 therefore disagree about whether the document parses, which is the practical cost of leaving this open.

It also settles what the fix costs: §12.1 needs no change. The recommendation below is one kernel field plus two prose edits, with the grammar already saying what the kernel would then be able to express.

Two further problems with §5.3’s argument, independent of which way the contradiction is settled. It reasons from the artifact to the rule — “the kernel’s map has no state field” — where the kernel is this series’ own bundled document and its field list is the thing in question, not evidence about it. And its parenthetical concerns absent keys, which is §2.9’s own unconditional rule and says nothing about values; the sugar’s key side needs no ? for a reason that has never been in doubt.

Interpretation chosen: the recommendation below, built. map carries a state field here, so _ at a map entry value reads as an entry present with an absent value where the declaration wrote {K => V?}, and is FIELD_REQUIRED under the default — the two answers an array element already gets. Either way the entry counts toward min_items/max_items (§2.9 has higher parts count all slots): the refusal costs the value its verdict, not the entry its place. In tree mode a permitted absence is a TsonAbsent; in bind mode the bound Map holds the key against a null, which is as close as Java comes to the distinction and still tells it from a key never stated.

This is a deliberate divergence from the published Revision 33 kernel, the third this implementation carries, and it resolves the contradiction rather than picking a side of it: §7.6’s permission survives, now sayable; §5.3’s refusal survives for the key, where it was never in doubt. Before it, both halves were implemented as written and the incoherence was author-visible — the value could not be marked optional and was optional anyway.

Recommendation — give map the state field, and let the sugar spell it. This is the reading that makes the container family uniform, and it is a subtraction from the prose rather than an addition:

map => ~product & {
  access_pattern:  product_access_type = NAMED
  size_type:       product_size_type = VARIABLE
  key_type:        type_ref
  value_type:      type_ref
  state:           element_state ~ REQUIRED
  min_items:       integer?
  max_items:       integer?
}

element_state ~ REQUIRED already appears three times in the kernel — array, tuple_element, field_group — so this reuses the enum and the default rather than introducing either. Then:

• §5.3’s table gains one row, {K => V?}, beside the [T?] row it copies; the “neither side admits ?” sentence keeps only its key half, where it was never in question.
• §7.6’s map-entry-value row becomes conditional and reads identically to its array-element row, and the clause explaining why the map row is the exception comes out. The table stops having an exception.
• Naming needs no new word. A map key can never be optional (§2.9), so state on map can only govern the value, exactly as state on array governs the element.

The two prose edits in full, so adoption needs nothing beside this entry. §5.3’s table gains a row directly under the [T?] row it copies:

| `{K => V?}`, `{K => V?; …}`  | the corresponding form with `state: OPTIONAL` bound directly                   |

and §5.3’s map paragraph replaces one sentence. Currently:

Neither side of => admits ?: the kernel’s map has no state field, and absence has no defined meaning for map values (an absent key is already a resolver error, [TSON-DATA] §2.9).

becomes:

The value side admits ?, marking the value OPTIONAL exactly as [T?] marks an array element (§12.1’s map-type already draws its value from element-type). The key side does not: an absent key is a resolver error ([TSON-DATA] §2.9), and map-key has no ? to write.

§7.6’s map-entry-value row becomes its array-element row with the nouns changed. Currently:

| Map entry value (schema in scope) | yes | Entry present with an absent value ([TSON-DATA] §2.9); map carries no element-state facet, so the permission is not schema-conditional |

becomes:

| Map entry value (schema in scope) | conditional | Permitted only when the map type’s value state is OPTIONAL, written {K => V?} (§5.3); the entry is then present with an absent value ([TSON-DATA] §2.9) |

The key row of §7.6 is unchanged, and so is every other row.

Adopting this invalidates data that Revision 33 accepts, which is worth stating plainly rather than leaving an editor to notice: a document writing _ at a map value validates today against any map type and validates afterwards only where the schema wrote ?. Revision 33 is a working draft with no compatibility guarantee between revisions, so this is permitted; it is not, however, a pure clarification, and it is the one part of this proposal with a cost outside the specification text. The direction of the break is the conservative one — documents become invalid rather than silently changing meaning, and the fix in every case is one character in the schema.

The reason to prefer it over the status quo is that the default flipped. Every other container defaults strict and is loosened with ? — [T] is REQUIRED, a tuple position is REQUIRED, a record field is REQUIRED. {K => V} alone defaults permissive, and there is no marker to tighten it. So a schema author can forbid an absent array element and cannot forbid an absent map value, which is not a distinction any of the four passages sets out to draw; it falls out of a missing field. For a format whose stated use is validation feedback, “this map has no absent values” is an ordinary thing to want to say and currently unsayable.

The third option is closed, and worth recording as closed. Reading §5.3’s sentence at face value — absence genuinely has no meaning at a map value, so _ is refused there outright — would be coherent, and it is what this implementation did before the §7.6 reading was applied. It requires amending Part 1 §2.9, which lists map entry values explicitly and states the present-with-an-absent-value distinction for entries as well as fields. Part 1 is frozen, so this option is not available without reopening it; that is a reason to rule it out rather than merely to disfavour it.

Status against Revision 33: open, new against this revision. The state field, the sugar’s ? on the value side, and the reader’s two answers are built and running here; the spec still says otherwise in both places, which is what this entry asks the next revision to settle.

13. atom-refinement takes a record-def, which cannot express a constraint binding — every atom refinement in the spec’s own core.tn fails to parse

Section: Part 2 §12.1 (atom-refinement, refined-def, record-def, field-def), §5.5 (atom refinement), §5.6 (canonical form), §9 (the bundled core.tn).

Problem: §12.1 gives the two ^ forms the same payload production:

refined-def     = type-name [ws "<" type-args ">"] ws "^" ws record-def
atom-refinement = "!" type-name ws "^" ws record-def

For refined-def — §5.7 record refinement — record-def is right. production => config ^ { host: = "prod.example.com" } binds a modifier onto a field the source record already declares, which is exactly field-def’s third alternative.

For atom-refinement it is wrong, because the body is not a record of field declarations. It is a record of values filling the constructor’s constraint vocabulary. record-def expands to record-entry, to field-def:

field-def  = *annotation field-name ws ":" ws
             ( field-type field-modifier / field-type / field-modifier )
field-type = type-ref ["?"]

so whatever follows name: must be a type-ref — and type-ref is paren-type / bracket-type / map-type / type-name ["<" type-args ">"], with type-name = unquoted-token. A constraint value is none of those.

This is not a theoretical mismatch: the spec’s own core type library does not parse. core.tn declares 17 atom refinements, and every one of them fails or misreads under this production:

• 12 carry a nested record — int8 => !integer ^ { size: { bits: 8 signed: true } }, and the eleven other sized integer families. A braced record is not a type-ref at all: map-type is the only brace alternative and it requires =>, while a bare record body is explicitly unspellable at a type position (§5.2). There is no reading under which this parses. Hard failure.
• 5 carry a bare number — positive_integer => !integer ^ { min: 1 }, negative_integer => !integer ^ { max: -1 }, non_empty_text => !text ^ { min_length: 1 }, and the rest. Here the production shapes the text but assigns it the wrong meaning: min: 1 becomes a field named min whose declared type is 1. Whether that is also a parse error turns on whether type-name’s numeric restriction (“a declaration name whose text matches the number production … is a parse error”) reaches a reference position or only a declaration; either way the resolver is handed a field declaration where the author wrote a constraint.

§5.5’s own worked examples are in the second group (!integer ^ { min: 0 max: 150 }), so the production does not parse the examples of the section it implements. A vocabulary with a quoted-string facet — a pattern, a spec — would land in the first group, since a quoted token is not an unquoted-token.

The one alternative of field-def that carries a value is field-modifier, and it means something else. field-name ":" field-modifier admits min: ~ 0 or min: = 0 — a field with a default or a fixed value and no declared type. No example anywhere writes a refinement that way, and it would not mean what a refinement means: §5.5’s body binds a value into the constructor’s vocabulary, where ~/= declare how a field of a record under construction behaves.

§5.6 makes the contradiction exact, because it prints the two forms side by side and they are the same characters:

!integer ^ { min: 0 max: 150 } → !integer_type { min: 0 max: 150 }

The right-hand side is instance, whose payload production is core-value. So one production calls { min: 0 max: 150 } a record of field declarations and the other calls the identical text a data record of bindings, and §5.6 says the desugar between them is a retargeting of the head — the body is carried across untouched.

§12.1’s own prose already half-concedes it. Its opening paragraph says core-value “appears at exactly one point — the constructor-application payload (instance, §5.5–§5.6)”, citing the atom-refinement section for the production that does not use it, and then states the mismatch as though it were a design choice: “an atom-refinement body is a braced record-def (§5.5)”.

Interpretation chosen: the body is a core-value, restricted to the braced form. TsonSchemaParser requires a { and then parses the payload with the data grammar — the same parseCoreValue() the instance branch calls one block below — and hands the resolver a DataValue. DefinitionResolver then merges it over the target instance’s own bound values per §5.6 and binds the result through the constructor’s compiled reader, which is what makes !integer ^ { size: { bits: 8 signed: true } } work at all. The brace requirement is enforced as its own diagnostic (“an atom refinement’s body is a braced record of constraint bindings (§5.5), never a bare value, a second type-ref or an annotation”) rather than by falling through to instance, since ^ has already committed the production.

This was not a deliberate divergence — the AST node’s own Javadoc has always read atom-refinement = "!" type-name ws "^" ws data-value, three lines below a comment quoting the production as record-def. The grammar could not have been implemented as written; nothing that reads core.tn can.

Suggested resolution — change one token in the production, and the prose that describes it:

atom-refinement = "!" type-name ws "^" ws core-value
                ; atom refinement (§5.5): the constructor's own
                ; constraint bindings, the same payload `instance`
                ; takes; the target MUST resolve to an atom-family
                ; instance (§3.3.1)

§5.5 already supplies the restriction the production then needs — “the body MUST be a braced record of constraint bindings” — so the positional form (§5.6) is excluded by prose that exists, and no second production is required. refined-def keeps record-def unchanged; it was always the right payload there, and the fix does not touch §5.7.

§12.1’s opening paragraph then needs its count corrected: core-value appears at two points, the constructor-application payload (instance, §5.6) and the atom-refinement body (§5.5) — which is the same statement §5.6 already makes when it desugars one into the other.

This is an error, not an open design question. Unlike the entries above it, there is nothing here for a revision to weigh: the production cannot parse the bundled artifact the same document publishes, and one token fixes it. It is filed with them only because that is where findings against Revision 33 live.

Status against Revision 33: open, new against this revision. The core-value reading is what is built and running here, and is the only reading under which core.tn loads.

14. §7.1 excludes ZWNJ and ZWJ from the profile in prose, and includes them in the set algebra — XID_Continue contains both

Section: Part 1 §7.1 (the UAX #31 profile; the ZWNJ/ZWJ paragraph; the byte-order-mark paragraph), §9.4 (confusables and bidi).

Problem: §7.1 states the profile as set algebra,

Start    = XID_Start ∪ Nd ∪ { - + . }
Continue = XID_Continue ∪ { - + . }

and three paragraphs later states an exclusion in prose:

The format-control characters ZWNJ (U+200C) and ZWJ (U+200D) are deliberately excluded from the profile, although UAX #31 permits them in restricted contexts and some languages admit them. They are invisible, which makes them confusable and spoofing surface (§9.4); names whose orthography requires them MUST be quoted.

U+200C and U+200D are in XID_Continue — and are so by design, not by accident. UAX #31’s requirement R1a, whose wording §7.1’s “restricted contexts” parenthetical echoes, has been removed: “The characters that were added when meeting this requirement are now part of the default; the contextual checks required by this requirement remain as part of the General Security Profile in [UTS #39].” So the joiners are ordinary identifier characters now, with their safety rule relocated to UTS #39 §3.1.1.1 rather than withdrawn. §7.1’s prose is describing a version of UAX #31 that no longer exists. Unicode 16.0, DerivedCoreProperties.txt:

200C..200D    ; ID_Continue  # Cf   [2] ZERO WIDTH NON-JOINER..ZERO WIDTH JOINER
200C..200D    ; XID_Continue # Cf   [2] ZERO WIDTH NON-JOINER..ZERO WIDTH JOINER

So Continue = XID_Continue ∪ { - + . } admits them and the prose forbids them, and the two statements are in the same section. An implementation that computes the profile from the named property — which is what the section tells it to do, and what an implementation with real XID tables will naturally do — accepts ab<ZWNJ>c as one token. One that follows the prose rejects it. Both are reading §7.1.

This is not the harmless kind of disagreement. The prose is the reading with the security argument behind it, it cites §9.4, and it imposes a MUST on authors (“names whose orthography requires them MUST be quoted”) that the algebra silently lifts.

Interpretation chosen: the prose, for now, and it is the half that should change. Lexer implements Continue as XID_Continue ∖ { U+200C, U+200D }, so ab<ZWNJ>c is a lexer error today. The exclusion is not a special case in the code — the same subtraction that removes the format characters generally removes these two — but it is a deliberate departure from the stated algebra, called out at the predicate so it is not “cleaned up” by someone checking the code against the formula. The intended end state is the opposite: the lexer admits the joiners, because they are XID_Continue, and the contextual rule below refuses the cases that are actually invisible. That change waits on the contextual rule existing, for the reason given under the resolution.

Suggested resolution — keep the algebra, replace the prose. The set algebra is correct as written and should not move: XID_Continue contains the joiners because UAX #31 put them in the default when it removed R1a, and a profile stated over a UCD property should mean that property. What goes is the blanket exclusion, which is over-broad in one direction and bypassable in the other (#3 has the full argument: it forbids the Persian names that need ZWNJ while its “MUST be quoted” remedy is how the attack reaches a Latin name).

In its place, the contextual rule the removal of R1a relocated to UTS #39 §3.1.1.1 — conditions A1/A2/B on the neighbouring characters’ Joining_Type, under two global conditions, single-script and NFC. It belongs on the identifier layer #3 proposes, not on the token profile.

What it costs, measured — and correcting an earlier draft of this entry. It said the rule “needs Joining_Type and Script only: both UCD core properties, neither requiring a UTS #39 table”. No UTS #39 table is needed, which was the point being made, but the rest understates it. The three conditions between them read four properties, and a mainstream runtime exposes two:

The data is small, though, because these code points are contiguous. Merged, the four sets are 48, 58, 148 and 143 ranges — 397 in all, not the ~1,400 lines the source files spend on them. As a delta-encoded table that is about 1.9 KB, and under 1 KB gzipped; as plain JSON arrays, 4.3 KB. So the honest summary is not “another confusables.txt” but “one kilobyte and three tables an implementation would not otherwise need”.

And how much is free depends sharply on the host. JavaScript’s regex property escapes cover \p{XID_Start}, \p{XID_Continue}, \p{Script=…}, \p{gc=…} and even \p{Join_Control} natively — so a browser implementation ships no table for the identifier profile itself, and the joiner rule is the only thing that makes it ship any UCD data at all. The JDK is the opposite: it exposes Script and General_Category, has no Joining_Type, Canonical_Combining_Class or Indic_Syllabic_Category API, and its identifier predicates are not the XID properties (#14), so it ships tables for both.

Worth the spec saying so where it quotes UTS #39 calling the rule “simple enough to be easily implemented with standard mechanisms such as regular expressions”: that sentence is true of the matching and silent about the data behind the character classes, and which side of that line a platform falls on varies by an order of magnitude.

The two must land together, and that is a real constraint rather than a tidiness point. A lexer whose token profile is exactly XID_Continue admits ab<ZWNJ>c; if the contextual check does not yet exist, that is the invisible-character hole reopened. So the profile correction, the contextual rule, and any conformance vector asserting the current behaviour move in one step — not the algebra first.

A second, smaller ask in the same section: say that no format character is admitted, not only that U+FEFF and ZWNJ/ZWJ are not. §7.1 currently names exactly three invisible characters — U+FEFF, in a paragraph of its own with a precise rule, and the ZWNJ/ZWJ pair. Everything else invisible is excluded only by not being in XID_Continue: the soft hyphen, the word joiner, the Arabic letter mark, and the bidi formatting controls U+202A–U+202E and U+2066–U+2069, which are the ones §9.4 exists for. A reader checking “does this document handle invisible characters?” finds two precise rules, concludes the topic is covered, and never learns that the dangerous members are handled by property membership alone.

That matters because the property is easy to approximate wrongly in exactly this direction. Every host identifier predicate this implementation’s authors reached for is a superset of XID_Continue that adds the format characters: Java’s Character.isUnicodeIdentifierPart is ID_Continue unioned with everything Character.isIdentifierIgnorable covers — all of Cf plus the non-whitespace C0/C1 controls — and JavaScript offers \p{ID_Continue} and \p{XID_Continue} one letter apart. This implementation used the Java predicate and accepted a bidi override inside a field name until a second implementation, a TypeScript port with real XID tables, reported the difference. Every ASCII test passed throughout.

Suggested wording, beside the byte-order-mark paragraph:

U+FEFF is not special in this respect. No character with General_Category Cf, and no control character, is in XID_Continue; none may appear in an unquoted token. This includes the bidi formatting controls (U+202A–U+202E, U+2066–U+2069, U+061C), on which §9.4 depends. Host-language identifier predicates frequently compute ID_* rather than XID_* and frequently admit the identifier-ignorable characters, and are not substitutes for the properties named here.

Not a finding: an earlier draft of this entry reported that UAX #31 R1a was unaddressed. It is not — the ZWNJ/ZWJ paragraph quoted above addresses it directly and gives its reason. The paragraph is the answer; the defect is that the algebra above it says the opposite.

Status against Revision 33: open, new against this revision. The recommendation is now what is built: the lexer follows the set algebra, so a joiner continues an unquoted token, and the exclusion the prose wants is applied where it belongs — to a name, by IdentifierParser through JoiningControls, which implements UTS #39 §3.1.1.1’s contexts A1, A2 and B under both of its global conditions. A joiner is admitted where it has a shaping effect (Persian کتاب<ZWNJ>ها, §3.1.1.1’s own Malayalam conjunct, a Sinhala ZWJ) and refused where it is invisible, which is every Latin position — so ad<ZWNJ>min is still rejected, and now for the right reason rather than by a blanket rule that also cost Persian its spelling.

All four properties ship here (Joining_Type, Canonical_Combining_Class, Indic_Syllabic_Category= Vowel_Dependent, and the General_Category default for Transparent), the JDK exposing none of the first three; that is 531 ranges, of which the Transparent default costs ten — ArabicShaping.txt states the rule in prose, so only the exceptions to it need storing. A1 was not implemented alone: the Arabic condition without the two Indic ones admits Persian and refuses Malayalam, which is the shape of failure this register rejects the restriction level for elsewhere.

The conformance vector moved with the change: lexer/invalid/zwnj-inside-unquoted-token is now lexer/valid/…, and its sidecar states the layering rather than the contradiction. The §3.1.1.1 rule itself has no vector — it is a name rule, so Part 2, and the suite has no Part 2 layer to put one in.

15. §5.11’s labelled-sum example does not parse: a datetime cannot be an unquoted token, and §7.1’s always-quote list omits the kind

Section: Part 2 §5.11 (the labelled-sum pattern); Part 1 §7.1 (the UAX #31 profile and its “Profile boundaries” paragraph), §7.2.4, §5.4/§5.5 (the temporal atoms).

Problem: §5.11 illustrates the labelled-sum pattern with

timestamps => { ( created: timestamp | modified: timestamp | accessed: timestamp ) }

An instance is { modified: 2026-05-21T13:05:00Z } …

and that instance is not a well-formed TSON document. : is Pattern_Syntax and outside §7.1’s unquoted profile — deliberately, since it is the record field separator — so the token ends at the first colon. The lexer produces 2026-05-21T13, then a :, and the record grammar has a value where it expects a separator or }. This implementation reports “adjacent values must be separated by whitespace, a comma, or both”; the diagnostic varies but the rejection does not. The example works only quoted:

{ modified: "2026-05-21T13:05:00Z" }

The near miss is what makes it worth reporting. §7.1 gives 2025-03-13 among its unquoted examples, and that is correct — a date is entirely inside the profile, Nd and - both being members. The profile covers dates and stops at times, because a time needs :. So the two look alike, one is spellable bare and the other never is, and nothing in §7.1 says so.

§7.1’s own “quote by kind” list is where it should say it. That paragraph exists for exactly this purpose:

Content kinds the profile cannot cover totally (paths, URIs, monetary amounts, rationals, networks, percentages, ranges) are excluded entirely, so their quoting rule is always, never a per-character scan.

Times and datetimes belong in that list and are missing from it — which is presumably how the §5.11 example came to be written. The omission is the more consequential half of this entry: the list is what a generator implements, and a generator built from it will emit 2026-05-21T13:05:00Z bare and produce documents no reader accepts. The one-word fix is to add the kinds; the sentence’s own construction (“cannot cover totally”) already justifies it.

Two details worth getting right in the wording, because the boundary is not where a reader might guess:

• It is not the whole temporal family. date is fully coverable and should stay spellable bare — the entry is time and datetime, and any duration carrying a time part (PT1H30M is fine; the colon forms are not).
• It is not about the T or the Z. Both are XID_Continue. The single character that ends the token is the colon, which is why 2026-05-21T13 survives as a token and the rest does not.

Interpretation chosen: none was available — the example is rejected by the ordinary lexer, and this implementation neither special-cases it nor could. 2025-03-13 lexes as one unquoted token; 2026-05-21T13:05:00Z does not; 13:05:00 does not. The temporal atoms parse quoted content in all three cases, so nothing about the type vocabulary is affected: this is purely which spellings reach it.

Suggested resolution: fix the §5.11 instance to quote its value, and add times and datetimes to §7.1’s excluded-kinds list so the next example is written correctly by construction rather than by review.

Status against Revision 33: open, new against this revision. §5.11’s example is unchanged and §7.1’s list does not mention the temporal kinds.

16. §7.2 rule 1 treats LRM and RLM as horizontal space; UAX #31 makes them ignorable format controls whose insertion “shall have no effect on the meaning of the program”, and §9.5 documents the resulting hazard instead of removing it

Section: Part 1 §7.2 rule 1 (the whitespace rule and its immutable eleven-character set), §9.5 (bidirectional formatting characters); UAX #31 §4.1 (Pattern_White_Space, requirement R3a) and §4.1.3 (contexts for ignorable format controls), both already normative references of Part 1.

Problem: §7.2 rule 1 gives Pattern_White_Space as one flat set, every member doing one job:

Whitespace — Characters with the Pattern_White_Space property are consumed and not emitted as tokens. The set is immutable: U+0009 (TAB), U+000A (LF), U+000B (VT), U+000C (FF), U+000D (CR), U+0020 (SPACE), U+0085 (NEL), U+200E (LRM), U+200F (RLM), U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR).

UAX #31 does not give that property one job; it gives it three. R3a-1, verbatim:

[UAX31-R3a-1]. Use Pattern_White_Space characters as the set of characters interpreted as whitespace in parsing, as follows:

1. A sequence of one or more of any of the following characters shall be interpreted as a sequence of one or more end of line: U+000A, U+000B, U+000C, U+000D, U+0085, U+2028, U+2029
2. The Pattern_White_Space characters with the property Default_Ignorable_Code_Point shall be treated as ignorable format controls; they shall be allowed in the contexts UAX31-I1, UAX31-I2, and UAX31-I3 defined in Section 4.1.3, Contexts for Ignorable Format Controls, where their insertion shall have no effect on the meaning of the program.
3. All other characters in Pattern_White_Space shall be interpreted as horizontal space.

and the note immediately under it removes any question of which characters item 2 means:

The characters to be treated as ignorable format controls under item 2 of [UAX31-R3a-1] are U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK.

So the two characters §7.2 rule 1 folds into horizontal space are precisely the two UAX #31 excludes from it. The three contexts item 2 allows them in are UAX31-I1 “adjacent to lexical horizontal space (within a sequence of lexical horizontal spaces, or at the start or end of such a sequence)”, UAX31-I2 “as optional space, that is, wherever horizontal space could be inserted without changing the meaning of the program”, and UAX31-I3 “at the start and end of a lexical line” — all three being places a boundary already exists, which is why item 2 can require the insertion to mean nothing.

§9.5 then states the consequence as a property of UAX #31 rather than as a departure from it:

Pattern_White_Space includes two bidirectional formatting marks that are not visual whitespace — U+200E (LRM) and U+200F (RLM). These are token separators per UAX #31, so a stray LRM or RLM inside what an author perceives as a single identifier silently terminates the token and can alter document structure invisibly. Implementations processing untrusted input SHOULD consider surfacing bidirectional formatting characters outside quoted tokens.

“These are token separators per UAX #31” is the sentence to change: UAX #31 says the opposite in a numbered requirement. The hazard §9.5 describes is real, but it is a hazard TSON creates by applying item 3 to item 2’s characters, and the remedy is item 2, not a SHOULD to surface the damage afterwards.

The damage is narrower than §9.5 suggests, and worse where it lands. Under the flat reading a split token is usually caught by the grammar a moment later — a record key, a field value or a type-ref that splits in two produces a parse error, so the document is refused and nothing is silent. The exception is every position where juxtaposition is itself the separator, which is where a split yields a document that is still valid and says something else:

An array field declared [int32] therefore gains an element that no one wrote and no one can see, and the reader is behaving correctly while it happens — the bytes and the rendering simply disagree, which is the class of defect §9.5 exists to name.

The other ten bidirectional formatting characters need no rule at all, which is worth saying because §9.5’s remedy is worded over all of them. U+061C, U+202A–U+202E and U+2066–U+2069 are Cf and are not in Pattern_White_Space; §7.1’s profile already excludes them, so they are a lexer error outside a quoted token today with no rule of their own — the existing lexer/invalid/bidi-override-inside-unquoted-token vector pins one. The entire question is LRM and RLM, and R3a-1 answers it.

Interpretation chosen: UAX31-R3a-1, built. Lexer implements the three groups as three groups. Item 1 and item 3’s characters are unchanged. An LRM or RLM is consumed and contributes nothing, and a run of them holding no real horizontal space is refused when the code points on either side of it would have continued a single token — which is I1 and I2 decided by looking at two characters, and is R3a’s own suggested strategy:

Since these characters are allowed only where a boundary would, in their absence, exist between lexical elements, an implementation could ignore them when lexing, and then consider as illegal any lexical element that contains them.

So [1<LRM>2] and ad<LRM>min are lexer errors naming the character, the pair it stands between and its position; { a:<LRM>1 }, [<LRM>1 2], [1<LRM> 2], [1 <LRM> 2] and an LRM at either end of a line are accepted and change nothing, I2/I1/I3 respectively; and inside a quoted token an LRM stays content, R3a’s own carve-out and the remedy §7.1 already prescribes for a name that needs one. The one carve-out the two-character test needs is ..: . is an unquoted-continuation character, so 1<LRM>.. would otherwise look interior when §7.2 rule 3 puts a token boundary there regardless.

This is a refusal, not a repair. ad<LRM>min could in principle be read as the single name admin, which is what “ignore them when lexing” says on its own — but R3a’s second clause is the operative one for a format that has to be safe to review, and a name whose bytes and rendering differ is exactly what must not resolve silently. It is also the same position §7.1 already takes for ZWNJ and ZWJ by a different mechanism (#14): the character is admitted as content where it is visible work and refused where it is invisible.

Suggested resolution — split §7.2 rule 1 the way R3a-1 splits the property, and retire §9.5’s remedy. Rule 1 becomes three clauses rather than one immutable list: the seven line terminators, the two ignorable format controls (allowed only where a token boundary already exists, with no effect on meaning), and the two horizontal spaces. §9.5 then has nothing left to warn about for LRM and RLM and no need to say anything about the other ten, which the profile already excludes; the section can go, or shrink to a note that the whitespace rule is where the bidi question is answered. The SHOULD should not survive in any form — it asks an implementation to report a meaning-changing insertion that the requirement it cites forbids from changing meaning at all.

UTS #55 is worth citing in the replacement text, because it supplies the half a bare prohibition would get wrong. §3.2 recommends that languages meet R3a-1 precisely so that authors may correct display: “Allowing the specified ignorable format controls between lexical elements allows the author of the program to correct its plain-text display by inserting characters where needed”. A rule that refused LRM everywhere outside quoted tokens would be simpler and would take that away — which is why the boundary/interior distinction is the rule and not a blanket ban.

Status against Revision 33: open, new against this revision. §7.2 rule 1’s list is unchanged and §9.5 still attributes the separator reading to UAX #31.

17. §2.5 and §2.6 require a duplicate to be rejected but assign it no §8.1 category, and §8.1’s own lists do not mention duplicates

Section: Part 1 §2.5 (record field uniqueness), §2.6 (map key identity), §8.1 (errors and reporting).

Problem: §8.1 is emphatic that every diagnostic the series requires falls into exactly one of four categories, and that the category is always recoverable:

Canonical phrasing. Normative rules throughout this series refer to errors using one of four canonical phrasings, each mapping unambiguously to a category: “is a lexer error”, “is a parse error”, “is a resolver error”, “is a validation error”. Where conformance language appears without an explicit category, the layer that detects the violation determines the category.

§2.5 and §2.6 use none of the four phrasings. Both say malformed:

Field names within a record MUST be unique. A record containing the same field name more than once is malformed and MUST be rejected, with the diagnostic at the repeated occurrence’s position.

Duplicate keys MUST NOT be present: a map containing two identical keys is malformed and MUST be rejected, with the diagnostic at the repeated occurrence’s position.

The fallback — “the layer that detects the violation determines the category” — does not settle it either, because §1.2 puts the detection below no layer in particular and §2.6 then puts it in two at once. §1.2 excludes it from both tiers of the structural grammar, and §2.6 makes the rule explicitly layered:

Textual identity is the parser’s minimum … A processor that decodes values compares decoded values: from base type resolution (§4) onward, different spellings of one value are one key (0xFF and 255, 1_000 and 1000), so a reader producing decoded output rejects keys the parser’s textual rule could not relate.

So one document’s duplicate is detected by the parser and another’s only by a reader, and under §8.1’s fallback the two would land in different categories for the same rule. Meanwhile §8.1’s own enumerations mention neither case: its parser bullet lists “unclosed brackets, adjacency violations, unexpected tokens, missing separators, !! without an adjacent colon form, a directive name outside the closed positional set”, and its resolver bullet lists “an absent sentinel in map key position; a built-in type annotation on a container value (§5.1); a token that a built-in atom’s parsing contract rejects (§5.2)”. A duplicate is in neither list.

This matters because a conformance corpus has to state the category: an error vector that only says “rejected” passes an implementation that rejects for the wrong reason.

What this implementation does: reports resolver for both, and for every spelling of both. The reasoning is §8.1’s own resolver bullet, which already houses the data-format rules a processor applies after the structural parser has accepted the document — the absent map key next door is exactly parallel, and §2.9 names it a resolver-layer constraint in so many words. Choosing by detecting layer instead would make { Alice => 1 "Alice" => 2 } a parse error and { 0xFF => 1 255 => 2 } a resolver error, which is one rule with two categories and an author’s diagnostic changing category with the spelling they happened to use.

Suggested resolution: state the category in §2.5 and §2.6 directly, using one of the four canonical phrasings, and add duplicates to §8.1’s matching bullet. resolver for both is the recommendation, on the parallel with the absent map key. If the intent is instead that the minimum textual duplicate is a parse error and the decoded-only duplicate a resolver error, that is a defensible reading but wants saying outright — it is not what the fallback rule produces on its own, and it makes the category depend on which spelling a document used rather than on which rule it broke.

Status against Revision 33: open, new against this revision. Both sections still say “malformed”, and §8.1’s lists are unchanged.