When the Test Suite Is the Spec: What Vine's JSONata Rewrite Actually Demonstrates

Simon Willison linked this week to a post from the team at Vine describing how they rewrote the JSONata engine with AI assistance in roughly a day and eliminated $500K in annual costs. The headline reads like hype, but the technical setup behind it is worth pulling apart, because the conditions that made this possible are specific and not universally present.

What JSONata Is

JSONata is a query and transformation language for JSON, designed by Andrew Coleman at IBM and open-sourced in 2016. If XPath is what you use to navigate XML trees, JSONata is the equivalent for JSON, extended with transformation semantics and a functional expression language.

A simple JSONata expression like Account.Order.Product[price > 50].description traverses a nested JSON structure, filters by predicate, and extracts a field, all in a single expression. The language goes further: it supports aggregation functions, string manipulation, conditionals, lambdas, and higher-order functions. The reference implementation is written in JavaScript, produces an AST from the input expression, and walks that tree against the input document at evaluation time. It is an interpreter in the classical sense, not a compiled or JIT’d evaluator.

JSONata is embedded in IBM App Connect, Node-RED, and a number of enterprise integration platforms. It has tens of millions of weekly npm downloads, most of them transitive from these tools. The language has grown from a query tool into the expression substrate for a significant slice of enterprise data mapping.

Why This Was a Solvable Problem in a Day

The key asset that made a one-day port credible is the JSONata conformance test suite, which exists as a separate repository specifically to enable third-party implementations to verify behavioral compatibility. This is not incidental. The test suite was designed to answer the question: does this implementation evaluate JSONata expressions the same way the reference does?

When you feed a language model the reference JavaScript implementation, the language specification, and a conformance test suite with hundreds of cases, the problem structure changes from open-ended generation to directed translation with an automated oracle. You are not asking the model to design anything. You are asking it to produce an implementation in a target language whose correctness is verifiable on every iteration. The feedback loop is mechanical: run the conformance tests, find which ones fail, give the failures back to the model, repeat.

This is fundamentally different from prompting an LLM to write a new feature from a vague description. The boundaries are sharp. The expected outputs are defined. The measure of doneness is objective. JSONata’s AST evaluation model is also relatively amenable to this kind of translation, because the core evaluation logic is a tree walk with well-scoped cases for each node type. There is no complex runtime, no garbage collector to reimplement, no concurrency model to port. The interpreter’s core is recursive descent over a data structure.

The language also has bounded scope. JSONata does not attempt to be a general-purpose language. Its function library is documented in full, its operator semantics are specified, and its edge cases (particularly around undefined propagation and the behavior of path expressions over arrays) are covered by the conformance suite. A full-featured language runtime would not have these properties.

The Economics of $500K

$500K per year is $41K per month, which sits comfortably in the range of enterprise software licensing for platforms that embed JSONata. IBM App Connect’s enterprise pricing, and similar iPaaS platforms that use JSONata as their expression language, carry subscription costs in this territory for large deployment footprints. If Vine was previously routing significant data transformation workloads through a commercial platform to get JSONata evaluation, replacing that with an in-house implementation removes the platform dependency entirely.

There is an alternative reading: if the team was running the JavaScript reference implementation at scale, the interpreter overhead for millions of evaluations per day adds up in compute costs. The JavaScript implementation is not designed for high-throughput embedded use; it parses and evaluates on every call unless you cache the compiled AST yourself. Porting to a compiled language like Go or Rust would reduce per-evaluation overhead significantly, and at sufficient volume that difference becomes material.

Either way, the $500K figure points at a real phenomenon in enterprise software: proprietary platforms that embed open-source language engines are charging for infrastructure whose core is freely available, and a team that can build a conformant implementation in-house recovers that margin. The AI rewrite compressed the time cost of doing so from weeks to a day.

Spec-Driven Porting as a Repeatable Pattern

What Vine did fits a pattern that has become more common over the past year. The conditions for it are:

1. A well-specified library or language with formal documentation
2. A conformance test suite that can serve as an automated acceptance criterion
3. A reference implementation that the model can read as a source of behavioral truth
4. A bounded scope that does not require porting a runtime, OS interface, or external dependency tree

JSONata meets all four. Many important open-source libraries meet fewer. A library with good unit tests but no conformance spec is harder, because unit tests encode implementation details as well as behavior, and distinguishing the two is non-trivial. A library with a spec but no test suite requires the model to derive correctness from documentation alone, which produces more iteration and more gaps.

The conformance test model, where the test suite is designed to be run against any implementation and not just the reference one, exists in a few other places in the ecosystem. The WebAssembly spec tests work this way. WHATWG HTML conformance tests work this way. ECMAScript has Test262. These are all targets where AI-assisted reimplementation would have similarly favorable conditions.

The pattern does not generalize to arbitrary codebases. A codebase without a conformance test suite requires the team to write the acceptance criteria themselves before the porting work begins. That is often the harder part: not implementing the behavior, but specifying it precisely enough to verify. The JSONata case succeeds in part because that specification work was done years earlier, by the original maintainers, for exactly this purpose.

What This Is Not

A one-day AI port of a bounded DSL interpreter does not demonstrate that large-scale system rewrites are now tractable in similar timeframes. The properties that make JSONata suitable for this approach, a formal spec, a conformance suite, a clean interpreter architecture, are not present in most production systems. A payments service, a storage engine, or a distributed systems component has implicit behavior, environment dependencies, and operational semantics that resist this kind of translation.

The result also still requires verification beyond the conformance suite. Conformance tests check that the implementation matches the spec. They do not check performance under production load, memory behavior at scale, or interaction with the surrounding infrastructure. Vine presumably did additional testing before routing real data through the new implementation. That work is not captured in “one day.”

But the underlying observation holds. For libraries and DSLs with formal specifications and comprehensive test suites, AI-assisted porting has become a genuinely fast path. The economics are real. The time savings are real. The preconditions are specific enough that “rewrite anything with AI in a day” is the wrong lesson, while “invest in conformance test suites” is the right one.