`generate_insar_mask()`: inputDataExceptionMask bits silently dropped from InSAR mask layer under NumPy ≥ 2.0 (uint8 scalar left-shift overflows to 0)

[xhuang-jpl]

Summary

generate_insar_mask() in python/packages/nisar/products/insar/utils.py packs the reference/secondary RSLC inputDataExceptionMask values into the InSAR mask layer by left-shifting np.uint8 scalars by 16 and 8 bits:

# utils.py, L592
ref_input_exception_mask_id = ref_input_exception_mask[int(i),int(j)] << 16
...
# utils.py, L599
sec_input_exception_mask_id = sec_input_exception_mask[sec_i,sec_j] << 8

Both masks are loaded as np.uint8 (L551), so indexing yields a np.uint8 scalar.

• With NumPy < 2.0, legacy value-based promotion widens np.uint8 scalar << python int to int64, so the shifts produce the intended values.
• With NumPy ≥ 2.0 (NEP 50 promotion rules), the result stays uint8. Shifting a uint8 by ≥ 8 bits pushes every bit out and yields 0 — silently, with no warning or error.

Consequence under NumPy ≥ 2.0: bits 8–15 (secondary exception mask) and 16–23 (reference exception mask) of the output mask dataset are always zero; only the sub-swath id in the low bits survives. This corrupts the mask layer written via InSAR_L1_writer.py (L535, L766) and everything downstream that inherits it. Note the corruption only manifests when inputDataExceptionMask is non-zero — i.e., exactly the scenes those bits exist to flag.

Affected lines (permalinks): utils.py#L592, utils.py#L599. The same code is present on current develop.

Minimal reproduction

import numpy as np

exc_mask = np.array([[3, 1], [2, 0]], dtype=np.uint8)  # as loaded by _load_exception_mask()

ref_id  = exc_mask[0, 0] << 16          # reference exception, L592 pattern
sec_id  = exc_mask[0, 1] << 8           # secondary exception, L599 pattern
mask_id = 5 | ref_id | sec_id           # subswath_mask_id | ref | sec
print(hex(int(mask_id)))

expression	NumPy 1.26.4	NumPy 2.5.1
exc_mask[0,0] << 16	196608 (int64)	0 (uint8)
exc_mask[0,1] << 8	256 (int64)	0 (uint8)
mask_id	0x30105 ✅	0x5 ❌

Root cause

NumPy 2.0 adopted NEP 50 scalar promotion (see the NumPy 2.0 migration guide). Under the old rules, a NumPy integer scalar combined with a Python int was promoted based on values (here → int64). Under NEP 50, the Python int is a "weak" scalar and the result keeps the NumPy operand's dtype (uint8), so the shift overflows to 0.

Related observation: the equivalent array expression uint8_array << 8 returns 0 under both 1.26.4 and 2.x (value-based promotion never applied to the array case). So any future vectorization of this per-pixel loop would hit the same bug even with NumPy pinned — the fix belongs at the source, not in the pin.

Environment / operational impact

• Reproduced with numpy 2.5.1; any numpy ≥ 2.0 is affected. Not reproducible with numpy 1.26.x.
• The oracle8conda runtime container pins numpy-1.26.4 (spec-file.txt#L177), so containerized production processing is currently unaffected. However, any environment that resolves numpy ≥ 2.0 (fresh conda-forge env, pip install, source builds outside the container) silently produces corrupted masks.

Suggested fix

Convert the scalar to a wide type before shifting:

ref_input_exception_mask_id = int(ref_input_exception_mask[int(i), int(j)]) << 16
...
sec_input_exception_mask_id = int(sec_input_exception_mask[sec_i, sec_j]) << 8

(np.uint32(...) << 16 works equally well; verified correct under both 1.26.4 and 2.5.1.) This matches the idiom already used correctly elsewhere in the codebase, e.g. nisar/workflows/ionosphere.py#L271:

shifted_new_bits = new_binary_mask.astype(np.uint32) << 24

Loading the exception masks as uint32 in _load_exception_mask() would also fix it, but quadruples the memory footprint of two full-swath arrays, so the per-element cast is preferable.

Follow-ups worth considering: audit other integer bit-shift/packing sites for NEP 50 sensitivity, and add a numpy ≥ 2.0 job to CI — this failure mode produces no warning, so only a regression test will catch it.

Credit

Issue first observed by @oberonia78 and handled in his PR (#333).