CM4 4deg coupled data processing - #1390
Draft
elynnwu wants to merge 6 commits into
Draft
Conversation
Path-existence decisions in create_coupled_datasets.py, the xpartition output writer, get_stats.py, and merge_stats.py now resolve the filesystem from the URL scheme via a shared fsspec helper (fs_utils.py), so gs:// inputs/outputs and local paths share one code path. Adds an end-to-end test of write_datasets_and_stats on synthetic zarrs covering staged outputs, resume-on-existing, and debug mode.
The new OM4 production ocean stores have no separate 6-hourly sea ice dataset: sea ice fields live in the ocean store at 5-daily snapshot instants, including a natively-computed ocean-relative `ocean_sea_ice_fraction`. This PR lets `compute_coupled_sea_ice` take sea surface fraction and sea ice concentration from the ocean store, selecting the source by priority: a separate sea ice dataset when configured; else the ocean store's sea ice fields when no `window_avg` is configured (the gate keeps existing no-sea-ice-dataset windowed configs, like the E3SM coupled configs, on the legacy atmosphere-sourced path unchanged); else the atmosphere's own sea ice fraction. A new `CoupledSeaIceConfig.use_atmosphere_sea_ice_fraction_fallback` flag (default True, preserving existing behavior) makes reaching the atmosphere source a loud error, so a misconfigured production run — a missing ocean sea ice variable or an accidentally-configured window average — can never silently define the coupled surface from the atmosphere field. On the ocean-sourced path the existing coherence logic (modified sea surface fraction where the ocean is wet, concentration clipped to [0, 1], sea ice and ocean fractions recomposed) is applied at 5-daily resolution and the fractions are forward-filled onto the atmosphere's 6-hourly index — no window average, which is unnecessary at the ocean cadence. Within the ocean source, the native `ocean_sea_ice_fraction` is preferred over re-deriving the clipped ratio. `hfds_total_area` is passed through from the input ocean store instead of being re-derived when it is already present. Changes: - `coupled_dataset_utils.compute_coupled_sea_ice` selects the sea-ice source by priority (sea ice dataset > ocean store when un-windowed > atmosphere) and gains the ocean-sourced 5-daily path - `coupled_dataset_utils.CoupledSeaIceConfig` gains `use_atmosphere_sea_ice_fraction_fallback` - `coupled_dataset_utils.OceanInputFieldsConfig` gains `sea_ice_fraction_name` and `ocean_sea_ice_fraction_name` - `coupled_dataset_utils.compute_coupled_ocean` passes `hfds_total_area` through when the input ocean store already provides it
compute_coupled_sea_ice selects its sea-ice source by presence priority: a separate sea ice dataset; else the ocean store when no window_avg is configured and the store carries sea ice fields; else the atmosphere's own sea ice fraction. On the ocean-sourced path the coherence logic runs at the ocean's native temporal resolution — using the ocean-relative concentration directly when present, otherwise deriving it from the full-cell sea ice fraction — and the fractions are forward-filled onto the atmosphere's time index with no window-averaging. A new use_atmosphere_sea_ice_fraction_fallback switch (default true) lets production configs make the atmosphere source a loud error. compute_coupled_ocean passes hfds_total_area through from the input ocean store when already present instead of re-deriving it. Configs with a separate sea ice dataset or a configured window_avg behave unchanged, except that compute_coupled_ocean no longer includes the atmosphere surface temperature in the coupled ocean dataset; it was previously retained by mistake when include_ts was configured.
…led-production-configs
The CM4 piControl and 1pctCO2 coupled configs now read the 2026 OM4 1-degree production stores directly on GCS and write coupled outputs and stats to gs://vcm-ml-intermediate. The sea ice fractions come from the ocean store's native 5-daily fields: no separate sea ice input, no sea-ice window average, and use_atmosphere_sea_ice_fraction_fallback is false so a misconfiguration fails loudly instead of silently using the atmosphere's own field. Time chunk sizes align with the stores' shards (atmosphere 360, ocean 365). The surface-flux window timestamps are right-labeled at the ocean stores' interval-end instants, and the 1pctCO2 timestamps follow that store's model years (0001-0141). The 1pctCO2 config's precomputed sea ice mask points at the piControl coupled ocean output, which must exist before the 1pctCO2 run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short description of why the PR is needed and how it satisfies those requirements, in sentence form.
Changes:
symbol (e.g.
fme.core.my_function) or script and concise description of changes or added featureCan group multiple related symbols on a single bullet
Tests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated
Resolves # (delete if none)