A support for parsing compressed data products in fprime-dp#309
A support for parsing compressed data products in fprime-dp#309kubiak-jpl wants to merge 8 commits into
Conversation
…member. Optimize type methods for numerical types
There was a problem hiding this comment.
Pull request overview
Adds support in the F Prime GDS fprime-dp decode tool to detect compressed data product records and automatically decompress them before emitting JSON, with an option to disable decompression and emit the raw compression records instead. This extends the existing ConfigManager-based DP decoder to handle the compression record format produced by dpCompressProc.
Changes:
- Added
-z/--disable-decompressionflag to thefprime-dp decodeCLI. - Extended
DataProductDecoderto detect compression records, decompress them (UNCOMPRESSED and ZLIB_DEFLATE), and re-decode the resulting byte stream. - Refactored record decoding into a reusable
decode_recordshelper.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/fprime_gds/executables/data_products.py | Adds CLI flag and threads decompression option into the decoder invocation. |
| src/fprime_gds/common/dp/decoder.py | Implements compression-record detection + decompression, and factors out record-loop logic. |
Comments suppressed due to low confidence (1)
src/fprime_gds/common/dp/decoder.py:95
DataProductDecoder.__init__now requiresdisable_decompressionas a positional argument, which is a breaking change for any existing callers importing this class. Make the new parameter optional (defaulting toFalse) and document it in the docstring so older call sites continue to work.
def __init__(self, dictionaries: Dictionaries, binary_file_path: str, disable_decompression: bool, output_json_path: Optional[str] = None):
"""Initialize the decoder.
Args:
dictionaries: Dictionaries object containing dictionary information
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| while (r_io.tell() - position_at_start) < data_size: | ||
| # Read record ID | ||
| record_id_bin = r_io.read(ConfigManager().get_type("FwDpIdType").getSize()) | ||
| record_id_obj = ConfigManager().get_type("FwDpIdType")() | ||
| record_id_obj.deserialize(record_id_bin, 0) | ||
| record_id = record_id_obj.val |
There was a problem hiding this comment.
idk about that error message but this seems like a good check to add
| if record_meta.val['algorithm'] == 'UNCOMPRESSED': | ||
| uncomp_bytes.extend(record_io.read()) | ||
| elif record_meta.val['algorithm'] == 'ZLIB_DEFLATE': | ||
| uncomp_bytes.extend(zlib.decompress(record_io.read())) |
There was a problem hiding this comment.
Thoughts on also adding an else case to log an INFO/WARNING if algorithm is neither of the two supported ones? Later we could even make it a GDS plugin
There was a problem hiding this comment.
Yes. That makes sense
| if computed_crc != dp_crc.val: | ||
| raise CRCError("Data", dp_crc.val, computed_crc) | ||
|
|
||
| if not self.disable_decompression and self.is_compression_record(results["Records"][0]): |
| if not self.disable_decompression and self.is_compression_record(results["Records"][0]): | ||
|
|
||
| # Compressed records. Decompress and re-process | ||
| uncomp_bytes = self.decompress_records(results["Records"]) | ||
| if uncomp_bytes is not None: | ||
| uncomp_io = BytesIO(uncomp_bytes) | ||
| uncomp_records = self.decode_records(uncomp_io, len(uncomp_bytes)) | ||
| if uncomp_records is not None: | ||
| results["Records"] = uncomp_records |
There was a problem hiding this comment.
Not required, but would be appreciated (AI can write the test scripts if you provide test inputs/outputs)
There was a problem hiding this comment.
Yes. I missed the test_decoder.py script. I'll add a test case for the compressed products
Change Description
The PR adds support for decoding compressed data products alongside this PR
nasa/fprime#5235
If
fprime-dp decodeencounters a compressed data product it will attempt to decompress it and output a JSON for the decompressed product. The new-z,--disable-decompressionflag will disable this behavior and output the raw byte arrays from the compressed product as a JSON.Rationale
Enables F Prime tooling to seamlessly deal with compressed data products.
Testing/Review Recommendations
Using this example data product file, generated by the TestDeployment
Dp_00002576_1780687868_00728225.fdp.txt
Running the following command will generate the proper decompressed records
Dp_00002576_1780687868_00728225.json
Adding the
-zflag will generate raw compression recordsDp_00002576_1780687868_00728225_compressed.json
Future Work
None