openapi3: decoding allocates ~65x the document size, mostly re-parsing JSON it just generated

[reuvenharrison]

Summary

Loading a document allocates roughly 65x its size on disk, in the default configuration. The largest single contributor is a JSON text round trip performed in the middle of decoding: the document is decoded to map[string]any, re-serialized to JSON bytes, and parsed again. About 62% of all allocation goes on parsing JSON this process generated moments earlier.

Filing this for discussion rather than proposing a patch, since the fix has real trade-offs and the shape is worth agreeing on first.

Measurement

Real public specs, default Loader, IncludeOrigin off:

spec	size	total allocated	peak RSS	retained
github/rest-api-description api.github.com.yaml	9.3 MB	601 MB (65x)	208 MB	54 MB
APIs.guru stripe.com/2022-11-15	3.6 MB	236 MB (66x)	95 MB	19 MB

alloc_space for the GitHub spec, flat:

153 MB 25.1%  encoding/json.(*decodeState).objectInterface
 78 MB 12.8%  yaml3.(*parser).node
 56 MB  9.2%  reflect.unsafe_New
 46 MB  7.6%  reflect.mapassign_faststr0
 34 MB  5.6%  encoding/json.Unmarshal
 32 MB  5.2%  encoding/json.unquote

Cumulatively:

582 MB 95.8%  yaml.Unmarshal      (the whole decode)
377 MB 62.0%  yaml.jsonUnmarshal  (the re-parse)
361 MB 59.3%  encoding/json unmarshal

The 62% share is stable across specs; the 65x multiplier is not, and rises on documents that are structurally dense (short field names, few long descriptions).

Mechanism

The yaml decode path does:

[]byte
  -> yaml3 Decode            -> map[string]any
  -> convertToJSONableObject
  -> json.Marshal            -> JSON bytes          <-- re-serialize
  -> json.Unmarshal          -> the target struct   <-- re-parse

The round trip exists for a good reason: the openapi3 types implement UnmarshalJSON, not UnmarshalYAML, so reaching them requires JSON. But a 9.3 MB document becomes an interface tree, then JSON text, then a second full parse.

JSON input avoids all of this: unmarshal tries json.Unmarshal first and returns on success, so a JSON document is parsed exactly once. The round trip is specific to YAML input.

That fast path is also the existence proof for a fix. It shows the target types can be fully populated by a single json.Unmarshal with no intermediate representation; the open question is only how to feed them from YAML without going through text.

Why it is like this

Worth saying plainly, since none of it is an oversight.

The yaml layer is a fork of ghodss/yaml (via invopop/yaml), and the round trip is its stated purpose, not a side effect. From its own package doc:

this package first converts YAML to JSON using go-yaml and then uses json.Marshal and json.Unmarshal to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods MarshalJSON and UnmarshalJSON unlike go-yaml.

The rationale is ghodss's 2014 post. For openapi3 it is load-bearing: every type carries json tags and many implement UnmarshalJSON, so decoding with go-yaml directly would mean maintaining parallel yaml tags and UnmarshalYAML methods across the package.

The map[string]any intermediate is not a bare stepping stone either. convertToJSONableObject reconciles YAML and JSON semantics, and does it target-aware: it walks the destination reflect.Value alongside the value to decide coercions. Non-string YAML map keys become strings; whether a scalar stays a string depends on the destination field's type; levels that implement UnmarshalJSON or TextUnmarshaler are left alone.

So this is an 11-year-old design that solves a real problem, and the intermediate carries semantics. That constrains the fix rather than ruling it out, but it means "just skip the round trip" is not available.

Possible directions

1. Transcode the node tree straight to JSON bytes, skipping map[string]any. Keeps json.Unmarshal and every UnmarshalJSON hook exactly as they are, and removes the interface tree, most of reflect.mapassign_faststr0, and the decode into any. The catch is the target-aware coercion above: a purely syntactic transcode would change behaviour for YAML scalars whose interpretation depends on the destination type, so those rules have to be preserved, which means threading the destination through the transcode.
2. Decode map[string]any into the target directly, without the text round trip. Keeps the coercion exactly as it is (it already runs on this representation) and only removes the serialize/re-parse, but it has to reimplement dispatch to UnmarshalJSON, which is where the correctness risk sits.
3. Native UnmarshalYAML on the openapi3 types. One parse, no intermediate, no JSON text.

The one I would pursue: option 3

It measures fastest by a clear margin (numbers below), and unlike 1 and 2 it removes the intermediate rather than making it cheaper, so the 62% does not come back in a different shape later.

It is also the only one that fixes positions rather than working around them. A *yaml.Node already carries Line/Column, so each UnmarshalYAML can read the position off the node it is handed. That is a better answer than anything reachable from the round trip, and it applies to JSON documents too, since JSON parses as YAML.

Two things I did not expect when I tried it, both of which make the case better than I would have guessed:

The methods are about three lines each. The known field set can be read off the struct's yaml tags instead of restated. The JSON versions maintain that list by hand -- Schema.UnmarshalJSON is 91 lines, roughly 60 of them delete calls -- so a field added to a struct and forgotten in the list silently becomes an extension today. A reflection-derived set cannot drift. That also puts the ~30 methods closer to mechanical than to 30 individual design problems.

The JSON hooks are the actual cost. Responses.UnmarshalJSON re-marshals every child back to JSON and re-parses it, once per entry; SchemaRef.UnmarshalJSON parses the same bytes up to four times. Native decoding skips all of it, which is why it beats even the JSON fast path.

What it costs, stated plainly

• ~30 methods. Mechanical, but not free, and Schema is the one with real semantics in it ($ref siblings, the 3.1 conditionals). I have not ported it.
• Two decoders, or a breaking change. Keeping UnmarshalJSON means two implementations that must not drift. Dropping it is a breaking public API change, since callers json.Unmarshal these types directly.
• One behaviour difference to decide. A number inside an extension arrives as float64 from encoding/json and int from the YAML decoder. Marshaling normalises it, so comparing serialized output does not catch it -- a consumer doing Extensions["x-n"].(float64) would panic. Either normalise on the YAML path or document it.

Before anyone writes it

I have a spike covering four types (Responses, ResponseRef, Response, MediaType) with equivalence tests against the JSON path, which is where the numbers below come from. Extending it to the rest is a real chunk of work, so:

Is this a direction you would take? If option 3 is not something you want in openapi3 -- because of the two-decoders problem, the API break, or anything else -- that is a completely reasonable answer and worth saying now rather than after a large PR appears. Option 1 is the conservative alternative and I would be happy to pursue that instead.

How much is actually recoverable

Worth stating plainly, because "62% is spent re-parsing JSON" invites the inference that removing it recovers 62%. It does not, and I would rather put the number here than have someone discover it after the work.

Measured on a 2000-endpoint synthetic document (3.7 MB YAML / 5.1 MB JSON of the same content) decoded into a struct with both tag sets and no custom unmarshalers, so every decoder does the same work:

	ns/op	B/op
encoding/json, JSON input	33.4M	12.1 MB
native YAML decode into the struct	120M	84.4 MB
the round trip above	184M	143 MB

So on a struct with no custom unmarshalers, option 3 is 1.55x on time, not 2.6x: json.Unmarshal is fast, and what replaces it -- YAML struct decoding -- is not. No YAML decoder here is within 3.5x of encoding/json, on fewer bytes.

But that understates it for openapi3, and I would rather correct myself than leave the smaller number standing. I ported four real types -- Responses, ResponseRef, Response, MediaType -- to UnmarshalYAML and measured 400 responses with content and extensions:

path	ns/op	B/op	allocs/op
round trip (what happens today for YAML)	5.84M	5.33 MB	90,381
json.Unmarshal + the UnmarshalJSON hooks	3.02M	2.76 MB	48,401
native UnmarshalYAML	2.30M	2.32 MB	37,129

2.54x faster than today, and 1.31x faster than the JSON fast path. The hooks are the difference and a hook-free benchmark cannot see them: Responses.UnmarshalJSON re-marshals every child back to JSON and re-parses it, once per entry, and SchemaRef.UnmarshalJSON parses the same bytes up to four times (the ref, the extra keys, a sibling schema, the value). There are 16 nested json.Marshal calls across refs.go, maplike.go and openapi3.go. Handing the already-parsed child node to the child decoder skips all of it.

Options 1 and 2 keep json.Unmarshal and remove less, and I have not measured them.

That also answers the obvious suggestion of swapping the YAML library. I measured goccy/go-yaml, the performance-oriented alternative: 302M ns/op and 432 MB/op on the same input, slower than the round trip it would replace. UseJSONUnmarshaler did not change that. go.yaml.in/yaml/v3 is within noise of the current parser.

Happy to share the harness.

With IncludeOrigin

The same GitHub spec allocates 842 MB instead of 601 MB. The round trip is present either way; it is simply a smaller fraction of a larger number.

Origins make option 3 disproportionately attractive, though it is a general win and not an origin-only one. They currently ride inside the decoded data -- encoded into the node tree, decoded, then extracted again -- purely so they survive the JSON serialization. Under native decoding that cycle disappears, since each UnmarshalYAML is handed a node that already carries the position. Measured against the origin-enabled path, native decoding is 3.2x faster and 4.7x leaner.

Relationship to #1235

Disjoint. #1235 targets the origin maps, which exist only when IncludeOrigin is on and which are retained for the document's lifetime; it halves retained size and moves peak by 1.6%. Everything above is transient allocation in the default configuration, garbage by the time the load returns, and #1235 does not touch it. Neither blocks the other.

Reproducing

Load either spec above and compare runtime.MemStats plus peak RSS from getrusage; a heap profile with -sample_index=alloc_space gives the ranking. Happy to share the harness.