Research

The article introduces the concept of a proto-schema and provides context and direction for using seven structural primitives identified in the previous series as the way to develop a proto-schema from first principles.

Starting from the physical constraints of serialization, this article derives the fundamental concepts needed to describe valid data configurations: choice, sequence, multiplicity, and naming. These primitives appear independently across formal grammars and schema languages. The article explores why schemas matter and how the proto-schema’s aim is to transcend the limitations of any particular data format.

This article revisits the seven structural primitives identified in the previous series and reclassifies them through the lens of the proto-schema. By examining each primitive against the physical constraints of serialization and the needs of schema design, the seven collapse into three categories: structural primitives, semantic markers, and structural composition. The analysis reveals that most familiar data structures like records, arrays, tuples, sets, maps are variations of a single underlying Sequence primitive.

This article examines the Sequence to derive the common data structures found across data formats and programming languages. It reduces the core proto-schema types to the Tuple, Record, Array, Map, and Set. Each derived type is a Sequence that is distinguished by the properties that are active rather than by fundamentally different structure. It connects the common data structures to the underlying physical constraints of serialization. Edge cases like prefix items, flattened sequences, and conditional structures are also examined.

This article explores templates and partial types as the bridge between fully specified data and fully abstract schemas. Using the beads on twine analogy, a template is a pattern with blanks, a definition awaiting completion. Through the spectrum of completeness, the article demonstrates that data and schemas are not separate concepts but exist on a continuum of how many blanks remain. Templates provide a formal mechanism for reuse and parameterization within schemas that are resolved before validation.

This article explores structural composition as the mechanism for building new definitions from existing ones. Using the beads analogy to demonstrate physical constraints and then using the spread operator from the earlier JSON series, the operations of inclusion, override, and removal are examined. The concept of is-a relationships and sub-types demonstrates that types on a spectrum of completeness must follow narrowing rules. These constraints establish that definitions in the proto-schema are immutable, a property that ensures is-a relationships remain valid transitively across both Templates and structural composition.

What appeared to be the simplest of the proto-schema’s core properties, choice as “one of these options” turns out to be a deep and unsettled subject. This article reframes Choice as part of a family of variant mechanisms and catalogues nine such mechanisms that recur across schema systems. It then focuses on the most familiar of these, the tagged and untagged sum types, finding that both tagging and labelling of sum types are orthogonal to each other. It concludes that despite surface similarities, labelled and unlabelled tagged unions encode genuinely different concepts and both structures belong in the proto-schema. The remaining variant mechanisms are flagged for later articles.

Multiplicity defines the rules of how many values can exist, and while the idea is simple, it has been the source of significant problems in many modern data systems. This article dissects the history of "optionality" failures, from the "Zero Value" problem in Protocol Buffers to the meaning of null in JSON. By deconstructing the interaction between the Encoder, the Wire Format, and the Decoder, we derive a rigorous logic table that distinguishes the aims of the programmer from the aims of the schema. The resulting changes to the proto-schema establish solid rules for implementation.

JSON (JavaScript Object Notation) has become one of the major foundational formats for data interchange across modern software systems. Its simplicity, readability, and compatibility are key to its success, but those same strengths have exposed limitations as JSON is stretched across domains it was never designed to support. This article begins a series that will go deep (probably ridiculously deep) into every aspect of the JSON data format and investigate its use and application against real world data. By going back to first principles of data and serialization, the aim will be to uncover the gaps, provide new approaches and designs for a new data format.

JSON's number representation was designed to align with JavaScript's number model which maps directly to IEEE 754 double-precision numbers and is easily readable in web browsers. However, as JSON's application has evolved outside the browser, this simple design now creates challenges for applications requiring mathematical precision when using programming languages that support a wider range of mathematical values. This article examines JSON's numeric grammar, explores the broader mathematical concept of numbers, and investigates how the gap between mathematical ideals and computational realities affects data interchange formats and programming languages. By understanding these constraints, we can better evaluate the trade-offs inherent in any text-based numeric representation, and possibly come up with some new designs.

JSON's boolean type is one of only three atomic value types in JSON; numbers (explored in Part 2, and strings to be explored later). It is also the format's only supported enumerated type. This article examines the JSON boolean type and the broader concept of enumerated types (e.g. categorical data, and finite value sets) in data interchange formats. It explores the general concept of enum types and how they differ from string types, and investigates the potential benefits of making them distinct from strings.

Strings are the universal value container in JSON for values that are not numbers (integers and floats, see Part 2) and boolean values (see Part 3). It's also the last of the atomic value types before exploring compound types of JSON arrays and objects. This article examines JSON's string representation, from its Unicode foundations to its role as the default container for strings, dates, urls, uuids, binary data, and various other formats. It investigates the concept of sub-parsers and concludes with proposing borrowing a concept from YAML and applying it to JSON.

In Parts 2, Part 3 and Part 4, the atomic value types of JSON were explored. This article explores compound types of JSON; arrays and objects. It explores the fundamental concepts of serialization using the analogy of beads to develop a core set of structural primitives for data formats. These primitives will help in analysing and informing new data formats.

Part 6 of the deep dive into JSON series investigates the syntax of JSON. JSON inherited its syntax from JavaScript which provides a very succinct data format that itself inherits from a long lineage of programming languages. As languages and data formats evolve, can there be alignment that builds common understanding between data formats and programming languages. This article explores the early history of what syntax means and how it might affect the design of a new data format.

JSON doesn’t allow comments and has nothing like XML’s attributes. This article goes beyond JSON’s capabilities and explores the attribute & annotation structural primitive previously identified. It investigates the history and concepts around annotations before investigating a “what-if” scenario that expands JSON to include a syntax for annotations. It’s the beginning of a stronger theoretical framework for the creation of a new text format.

This article explores the final two structural primitives from the framework established in Part 5: references and structural composition. References enable pointing to previously defined values within a document, addressing JSON's inability to represent cyclic structures. Structural composition goes beyond simple references to allow combining and merging compound data structures. Through iterative design exploration, the article reveals how these primitives could be implemented through different syntax. By examining implementation challenges, the article shows how a JSON-like data format could be extended to include all seven structural primitives and create a complete theoretical framework for data interchange. The exploration concludes with the surprising discovery that structural composition sits at the precipice before a data format transforms into a computational system, blurring the traditional boundary between data representation and functional programming languages.

This article explores the unification of data schemas and data formats through the lens of prototype-based programming languages. Building on the structural primitives framework established in Part 5, it introduces the eighth and final primitive, templates and partial types. Drawing inspiration from Self and JavaScript's prototypal heritage, the article demonstrates how schemas and data can exist on a spectrum of structural completeness rather than as separate concepts. The exploration ends with a practical API example demonstrating how a potential syntax and unified approach to data and schemas could handle a real-world scenario. While acknowledging the complexity of full implementation, the article demonstrates that unification could provide more expressive and flexible data formats than traditionally provided by JSON and JSON Schema.

This article explores how constraints add another layer to structured data and templates. By examining constraints with comparisons to JSON Schema and XML Schema, we discover that meta-schemas become a self-referencing head-fuck that is barely researched but the basis for all data formats. The article follows on to further investigate how versioning can affect the ability for data and data formats to adapt and grow without becoming locked in by early standardisation. Finally, the article explores a multi-level architecture that splits structural parsing (handling syntax) and semantic parsing (interpreting values) to build an adaptable and flexible data format. The article concludes the exploration phase of this series with reflections on the theoretical framework discovered and future directions for practical implementation.