UPER: extension-addition/open-type values >= 16384 units are not fragmented; the 16384-unit case omits the required zero-length final component

[shakespark]

Summary

X.691 (02/2021) §11.9.3.8.3 (applied to the UNALIGNED variant via
§11.9.4.2) requires a UPER length determinant that takes the
general/unconstrained-length-determinant path (i.e. the length is not a
small constrained whole number) and whose value is >= 16384 to be encoded
as one or more fragments: each fragment is introduced by a single header
octet 11xxxxxx whose low 6 bits give the number of 16384-unit blocks
in that one fragment (1–4, i.e. a fragment covers 16K/32K/48K/64K
units total — the header is per-fragment, not one header per 16K
block), followed by that many blocks' worth of content. This repeats until
the remaining length is < 16384 (0..16383 inclusive, including 0); that
final remainder, however small, always gets its own explicit
length-determinant octet(s) — for OCTET STRING/open-type content, "units"
means octets; for SEQUENCE OF/SET OF, "units" means components (list
entries), not octets.

asn1tools 0.167.0's PER/UPER codec does not implement this loop for
several important callers. append_length_determinant() /
read_length_determinant() (asn1tools/codecs/per.py, ~lines 278/460)
are themselves correctly-written single-fragment primitives: for a
length >= 16384 they emit/consume exactly one fragment-header octet and
return/report the size of that one fragment — the looping is meant to
happen in the caller, via append_length_determinant_chunks() /
read_length_determinant_chunks() (~lines 300/481), which do loop
correctly (see the control-group test below). All affected callers below
consume or produce only one length component and never loop; how each
one uses the returned/read fragment size differs by call site (see the
decoder bullets):

Encoders (write one length-determinant call, then dump the entire
remaining content in one shot, regardless of how large it is):

• Every extension addition in an extensible SEQUENCE/SET (open-type
  wrapping): per.py:788, inside encode_additions() (per.py:745,
  MembersType, shared by SEQUENCE and SET).
• Every extension alternative in an extensible CHOICE (also open-type
  wrapping): per.py:1625, inside encode_additions() (per.py:1602).
• The generic UPER OpenType.encode(): asn1tools/codecs/uper.py:470
  (class at line 463). (per.py has its own, unrelated OpenType class
  around line 1932 that UPER does not use — uper.py's class is what
  matters here, not a subclass of the per.py one.)
• Extensible OCTET STRING, when the actual value falls in the extension
  region: uper.py:271, inside OctetString.encode().
• Extensible SEQUENCE OF/SET OF, when the actual value falls in the
  extension region: uper.py:122, inside ArrayType.encode().

Decoders read only one length component too, but use it differently
depending on call site — this is not uniform "read the entire remaining
content" behavior (that description is only accurate for the encoders
above):

• The generic UPER OpenType.decode() (uper.py:475) and the
  extension-region decoders for OctetString (uper.py:289) and
  ArrayType/SEQUENCE OF/SET OF (uper.py:145) call
  read_length_determinant() once and then read exactly that many
  octets/components as the field's entire value — silently truncating
  anything beyond the first fragment, with no attempt to check for or
  consume a terminator.
• For a known SEQUENCE/SET/CHOICE extension addition specifically
  (per.py:865 inside MembersType.decode_additions(), and per.py:1682
  inside Choice.decode_additions()): the outer open-type length
  determinant IS read (consuming whatever fragment-header/length byte
  comes next), but it is then not used to bound the inner decode at
  all — the addition's own inner type decoder is simply invoked and
  reads directly from the bit stream wherever it happens to be positioned
  next (see the decoder proof below for exactly how this corrupts a
  following field).
• For an addition the schema does not recognize, the outer length is
  used, but only to skip exactly that many octets
  (decoder.skip_bits(8 * open_type_length)) — i.e. still just the first
  fragment, never a loop.

Consequence: this affects every one-shot call site above whenever the
relevant length reaches 16384 units, not only exact multiples of 16384.
The 16384-unit case (discussed in most detail below, since it's the
simplest to construct and verify) is missing exactly the mandatory
zero-length terminating octet on encode; 16385 units is missing the
0x01-length octet that should introduce its 1-unit remainder; larger
values are similarly under-length-prefixed. (BIT STRING's extension-region
path raises NotImplementedError outright rather than mis-encoding — a
separate, already-known limitation, not part of this report.)

What is not affected, with one caveat:

• An unconstrained OCTET STRING/SEQUENCE OF/similar (no SIZE
  constraint at all), when it is not itself an extension addition,
  goes through
  append_length_determinant_chunks()/read_length_determinant_chunks(),
  which loops correctly (verified below).
  Caveat: if that same unconstrained field IS itself an extension
  addition, its own inner encoding is still correctly fragmented/
  terminated, but the outer open-type wrapper the extension-addition
  mechanism puts around it (per.py:788) is a separate one-shot call
  using the inner encoding's own byte count as its length — and since
  that inner byte count is itself often >= 16384 too, the outer wrapper
  hits the identical bug independently, on top of the (correct) inner
  one. Verified below: an unconstrained OCTET STRING extension addition
  with 16384 octets of content produces an inner encoding that is 16386
  bytes (0xC1 + 16384 octets + a correct 0x00 terminator — 16386 is
  not itself a multiple of 16384). The outer wrapper's spec-correct form
  is then 0xC1 (its own fragment header, covering one 16384-byte block
  of the inner encoding) + the first 16384 bytes of that inner encoding
  • 0x02 (the outer fragment's remainder-length octet, since
    16386-16384=2) + the last 2 bytes of the inner encoding. What
    asn1tools's one-shot encoder actually produces is the outer header
    followed by the entire 16386-byte inner encoding with no remainder
    marker at all — 1 byte short of spec-correct, missing specifically the
    0x02 remainder-length octet (not a zero-length terminator this
    time, since the inner byte count isn't a clean multiple of 16384).
• A length bounded by a SIZE constraint with an upper bound < 65536 is
  encoded as a constrained whole number instead of taking the
  general/unconstrained-length-determinant path this issue is about, and
  never reaches this code — but only when the type is non-extensible,
  or the actual value falls within the extension root range. A value
  in the extension region of such a type still goes through the same
  broken one-shot path regardless of its declared upper bound (see the
  uper.py:271/122 bullets above).

Reproduction (asn1tools only)

import asn1tools

SPEC_TMPL = """
OpenLenTest DEFINITIONS AUTOMATIC TAGS ::= BEGIN
Payload ::= SEQUENCE {
  before INTEGER (0..127),
  ...,
  blob OCTET STRING (SIZE(%d)) OPTIONAL
}
END
"""

def encode_blob(n):
    spec = asn1tools.compile_string(SPEC_TMPL % n, codec="uper")
    value = {"before": 5, "blob": bytes(i % 256 for i in range(n))}
    return spec, value, spec.encode("Payload", value)

for n in (16383, 16384, 16385):
    spec, value, encoded = encode_blob(n)
    print(n, len(encoded), encoded[:6].hex(), encoded[-4:].hex())

Output (asn1tools==0.167.0, Python 3.13.5):

16383 16387 8501bfff0001 fbfcfdfe   (below the fragmentation threshold; correct)
16384 16387 8501c1000102 fcfdfeff   (missing the mandatory trailing 0x00; spec-correct total is 16388)
16385 16388 8501c1000102 fdfeff00   (missing the 0x01 remainder-length octet; spec-correct total is 16389)

The full script (attached, repro_16384_analysis.py) additionally:

• checks exact byte values and positions with explicit comparisons and a
  PASS/FAIL print per check (never a bare assert, which python -O
  strips) — every check asserts the spec-correct expected behavior, so
  PASS uniformly means "matches spec" and FAIL uniformly means "bug
  detected here" (the script exits non-zero if any check fails, and is
  meant to be re-run as a regression check once fixed — FAILs are
  expected on the currently-installed version);
• includes a control group: the same 16384 and 16385 octets of content
  encoded as an unbounded OCTET STRING (not itself an extension
  addition) — correctly produces the spec-correct totals (16386 and 16387
  bytes respectively) and round-trips, confirming
  append_length_determinant_chunks() is implemented correctly in
  isolation;
• includes the extension-addition caveat described above, checked
  against the FULL expected byte sequence (not just a total-length count):
  the same 16384-octet unbounded OCTET STRING, this time itself an
  extension addition, produces an inner encoding that is correctly
  fragmented+terminated (16386 bytes) but an outer PDU that is missing
  exactly the outer fragment's 0x02 remainder-length octet — spec-correct
  is header + inner[:16384] + 0x02 + inner[16384:] (16390 bytes total),
  asn1tools's actual output is header + inner verbatim (16389 bytes,
  1 short), and the byte-level diff is at the exact position where 0x02
  should be;
• includes a receive-side decoder proof using only a legal, in-spec
  input, with every byte of that input hand-written from the spec (below)
  — not sliced or patched from asn1tools's own malformed output at all
  (patching a byte onto today's malformed layout would need to be
  rewritten, not just re-run, the moment the encoder changes at all,
  including the moment it's fixed correctly); and feeding the decoder a
  value whose length violates its own SIZE constraint would not test
  fragmentation handling at all (it just exercises "read the declared
  fixed size and stop," an unrelated and much less interesting failure
  mode). This reproduction avoids both pitfalls.

Decoder proof, step by step

SEQUENCE { before INTEGER(0..63), ..., blob OCTET STRING(SIZE(16384)) OPTIONAL, after INTEGER(0..63) OPTIONAL }, value {before: 5, blob: <16384 legal octets>, after: 42} (blob's length is exactly its declared
SIZE — fully legal per the schema):

1. Build the entire spec-compliant stream by hand from the schema and
   X.691 SS11.9.3.8.3 — nothing sliced or patched from asn1tools's own
   output: 8a 07 c1 (byte 0: before=5 + extension bit; byte 1: the
   2-addition presence/count field; byte 2: blob's open-type fragment
   header, 11+000001 = one 16384-octet block) + <16384 octets of blob content> + 0x00 (the spec-mandated terminator) + 0x01 0xa8
   (after's own open-type encoding: length=1, value=42<<2=0xa8). Call
   this valid. (Byte 2, 0xC1, is itself the exact fragment-header byte
   this issue is about — it is not taken from asn1tools's output, it is
   written out from the spec's own rule for a one-block fragment.)

2. Encoder-side check: asn1tools.encode() on tvalue does not
   equal valid — it's missing exactly the 0x00 terminator (this is
   the same encoder bug shown in PART 1/2 above, now confirmed against a
   from-scratch reference rather than just a byte count).

3. Decoder-side check: decoding valid (a fully spec-compliant
   stream — nothing about it is malformed) with asn1tools.decode()
   gives {'before': 5, 'blob': <16384 bytes>, 'after': 0} — not
   after: 42. The exact mechanism (verified against per.py's
   MembersType.decode_additions(), lines 855–884):

   a. The decoder reads valid's 0x00 (the terminator, placed right
   after blob's content) as if it were after's own open-type
   length determinant — a single octet with top bit 0, decoding as
   the plain value 0.
   b. For a known addition (after is recognized by the schema),
   decode_additions() does not use that length to bound anything —
   it just calls after's own inner INTEGER(0..63) decoder directly
   on whatever bits come next.
   c. Those next bits are the top 6 bits of what was actually after's
   real length-determinant byte, 0x01 = 00000001 — all zero,
   giving after = 0.
   d. decode_additions() then computes alignment_bits = (offset - decoder.number_of_bits) % 8 to re-align to the next octet
   boundary. Having consumed 6 bits for after, alignment_bits = 6,
   so it calls skip_bits(8 - 6) = skip_bits(2) — consuming the
   low 2 bits of that same 0x01 byte as inner open-type padding.
   Only the entire trailing 0xa8 byte (after's real encoded value,
   42) is left completely unconsumed — silently ignored as top-level
   trailing data; asn1tools.decode() raises no error about it.

Expected behavior

For a 16384-octet open-type/extension content: 0xC1 (one fragment, one
16K block) + 16384 octets of content + 0x00 (mandatory terminator). For
16385 octets: 0xC1 + 16384 octets + 0x01 (remainder length) + 1
octet. A decoder presented with either — or with any correctly-fragmented
stream of any length — should consume all fragments (including a
legitimate zero-length terminator) and continue decoding subsequent
fields normally.

Actual behavior

The encoder omits the required terminating/remainder length octet(s) in
one shot (1 byte short in both the 16384 and 16385 cases, and again for
an extension-addition-wrapped unconstrained OCTET STRING even though its
own inner encoding is correct). The relevant decoders, independently, have
no fragmentation-continuation logic at all: even given a genuinely valid,
independently-constructed spec-compliant input, a known extension
addition's inner decoder reads directly from the bit stream past the
(happens-to-decode-as-zero) outer length, corrupting the value of
whatever comes immediately after.

Impact

asn1tools's own UPER output for any open-type/extension-addition (or
extension-region OCTET STRING/SEQUENCE OF/SET OF) value reaching
16384 units is not spec-compliant. Depending on the receiving
implementation, this may be rejected outright as truncated, or the
following bytes in the stream may be misinterpreted as a continuation
length, desynchronizing the rest of the decode (demonstrated above for the
receive side, using asn1tools's own decoder on an independently
constructed, fully valid input). I confirmed the "rejected as truncated"
outcome specifically for the 16384-octet SEQUENCE-extension case against
two other independent UPER implementations
(asn1c/its forks, and a separate
third-party commercial ASN.1 toolchain) — both reject asn1tools's
16384-octet output outright; I have not separately verified their
behavior at 16385+ octets or for CHOICE/SET/extension-region
strings/arrays, though the same missing-bytes pattern applies to those by
the same code-path analysis above.

Environment

• asn1tools version: 0.167.0
• Python version: 3.13.5 (also reproduced on 3.11.2)
• OS: Linux