· 5 min read ·

The Test Suite Was the Real Hero in the JSONata Rewrite

Source: simonwillison

When a team from Vine posted about rewriting JSONata with AI in a single day and saving half a million dollars annually in cloud costs, the obvious reaction is to focus on the AI part. That framing is understandable but misses the more instructive story. The AI was the accelerant. The test suite was the foundation that made any of it trustworthy.

What JSONata Actually Is

JSONata is a query and transformation language for JSON, created by Andrew Coleman at IBM and open-sourced around 2016. If you have worked with IBM App Connect Enterprise, AWS EventBridge, or AWS Step Functions (which added native JSONata support in late 2024), you have likely encountered it. The syntax looks deceptively simple:

Account.Order.Product.Price

But JSONata is not a simple path extractor. It is a full expression language with operator precedence, lambdas, closures, partial application, a built-in function library covering strings, math, datetime, and aggregation, and a ~> chain operator that pipes values through function sequences. It also has non-obvious sequence semantics: a single value and a one-element array are not the same thing, and the runtime has to maintain that distinction correctly throughout evaluation.

The reference implementation is JavaScript, sitting at roughly 15,000 lines across the parser, evaluator, and function library. The parser is a Pratt parser, also called top-down operator precedence parsing, which handles left-recursive grammars elegantly but requires careful implementation to get operator associativity and precedence right. The evaluator supports async expressions, which the JavaScript implementation handles via promises. Porting all of this correctly is not a weekend project.

The Runtime Economics

The $500K annual saving is the number that makes people read the headline, and understanding it requires thinking about what it costs to run JavaScript at enterprise scale in cloud environments.

Node.js is not cheap to run as a hot-path compute layer. A minimal Node.js process carries roughly 30-60 MB of baseline memory just for the V8 runtime before your application code loads. In AWS Lambda, you pay for memory-time: allocate 256 MB and run for 100ms, and you pay for that full allocation. Cold start times for Node.js Lambda functions typically range from 200ms to over a second depending on bundle size and initialization work. If you are running JSONata transformations on every message in a high-throughput integration platform, those numbers compound quickly.

Port the same workload to Go or Rust, and the picture changes substantially. A Go binary has a far smaller runtime footprint, starts in milliseconds, and compiles transformation-heavy code to native machine instructions rather than running it through a JIT compiler. At sufficient request volume, the cost difference between Node.js and a compiled binary for CPU-bound transformation work can easily reach six figures annually.

This is not a new observation. Teams have been reaching for Go, Rust, or even C extensions for Python when they hit the limits of interpreted runtimes in hot paths for years. What changed is the cost of the migration itself.

Why AI Was Suited to This Problem

Not all code is equally amenable to AI-assisted porting. JSONata’s reference implementation is well-structured, well-commented, and the semantics are defined by a specification and a comprehensive test suite rather than by implicit tribal knowledge embedded in the codebase. Those properties matter.

When you ask a model to translate a well-written JavaScript function to Go, it is doing something closer to mechanical translation than creative engineering. A function that splits a string into tokens, checks operator precedence, or applies a lambda to a sequence of values has a clear, verifiable output. The model does not need to infer intent; the intent is in the code.

The hard problems in software porting are usually not in the algorithmic core. They are in the edge cases: off-by-one errors in string indexing, differences in how division handles remainders across languages, integer overflow behavior, Unicode normalization subtleties. These are exactly the kinds of problems a comprehensive test suite catches. JSONata’s test suite contains hundreds of cases specifically designed to probe edge behavior, because the JavaScript reference implementation has been in production long enough to accumulate that coverage.

The porting workflow likely followed a pattern that is becoming standard for this class of problem: translate a module with AI assistance, run the test suite, fix failures by feeding them back to the model or fixing by hand, move to the next module. The AI compresses the mechanical translation time from weeks to hours. The tests provide the correctness signal. Neither works without the other.

The Specification as Portable Truth

There is a broader principle here that applies beyond JSONata. Code that is backed by a formal or semi-formal specification, whether that is a language specification, an RFC, a wire protocol definition, or a test-as-spec suite, is dramatically more portable than code whose correctness lives only in comments and institutional memory.

The JSONata test suite functions as a portable specification. It is the contract that any implementation must satisfy, regardless of language. This is why the JSONata community has seen ports to Python, Go, and other languages: the test suite tells you unambiguously whether your port is correct, without requiring the original authors to validate every edge case manually.

Contrast this with porting a codebase where correctness is only verified by the main application’s integration tests, or where the logic is entangled with framework internals. Those ports are hard regardless of what AI tools you use, because there is no portable oracle to verify the output.

What This Pattern Generalizes To

The Vine rewrite is a good example of a category of migration work that AI genuinely accelerates: porting a well-tested, algorithmically clear implementation from a higher-overhead runtime to a lower-overhead one, where the test suite travels with the code.

Other strong candidates for this pattern include database drivers, serialization libraries, cryptographic primitives with published test vectors, parser implementations for standardized formats like JSON, YAML, or CSV, and compression algorithm implementations. These share the key properties: the semantics are externally specified, the test coverage is comprehensive, and the core logic is algorithmic rather than framework-dependent.

Projects where this pattern breaks down are ones where correctness depends on external state, where the test suite is thin relative to the behavior surface, or where the existing implementation has grown organically around language-specific idioms that do not map cleanly across. A React frontend is not a good candidate. A JSON query evaluator is.

The Actual Cost of the Migration

Saying the rewrite took a day is accurate in the narrow sense that the mechanical translation work was compressed into roughly that timeframe. But the prerequisites for that day represent years of work: a mature reference implementation, a comprehensive test suite, a specification detailed enough to resolve ambiguity, and enough production deployments to surface the edge cases that ended up in those tests.

The AI tooling made the port fast. Everything that came before made it possible to trust the result. That distinction matters for anyone trying to replicate this story elsewhere, because it tells you where to invest: not in AI tooling, which is already good enough for mechanical translation, but in test coverage and specification clarity, which remain the limiting factor for nearly every migration project I have seen.

Was this interesting?