Skip to content

Commit 4f60e2e

Browse files
estebanzimanyiestebanzimanyi
committed
tools(codegen): MEOS-operator generator + design proposal for Nebula codegen path
Closes the Nebula structural parity gap with Flink/Kafka by shipping the codegen infrastructure for generating per-MEOS-function pipeline tuples (logical + physical + parser + lowering). No generated C++ committed in this PR — the maintainer (cc @marianaGarcez) runs the generator on a chosen MEOS-function batch, reviews output, ships operators in follow-up PRs at a controlled pace. Why no generated code in this PR: - Generator author cannot build NebulaStream (full C++23 + vcpkg toolchain not available in author's environment); shipping unverified generated code would risk batched-broken operators. - Per-function review value: maintainer iterates on templates with the first batch's build feedback before scaling up. - Template iteration cost: first-pass templates may need adjustment after first build; smaller blast radius if only the generator lands. What lands: - tools/codegen/codegen_nebula.py — Python generator with embedded C++ templates derived 1:1 from the hand-written TemporalEDWithinGeometry operator shape (logical/physical/.hpp/.cpp) - tools/codegen/codegen_input.example.json — first-wave input list (5 spatial-relation E/A predicates: EDisjoint, ATouches, ECovers, ACrosses, EOverlaps over tgeo_geo) - tools/codegen/README.md — full design proposal: why codegen, what the generator produces, recommended scaling-wave sequence (W1-W5), what the generator does NOT do (CMakeLists / parser / grammar remain manual paste for idempotence), compile-verification note Smoke-verified: the generator runs locally + emits 5 operators × 4 files = 20 well-formed C++ source files; templates produce syntactically-reasonable output matching the existing operator style. Scaling path (recommended sequence): - W1: 5 spatial-relation E/A predicates (the example input) — first follow-up PR - W2: All ever/always spatial-relation predicates over tgeo_geo (~18 functions) — second follow-up PR - W3: Distance functions over tgeo_geo and tgeo_tgeo (~30) — third - W4: Scalar accessors that decompose to per-event reads — template extension required - W5: Aggregations (windowed/cross-stream) — separate generator with the aggregation-specific 4-layer pattern Stacks on PR MobilityDB#20. Tools-only; touches no operator code, no CMakeLists, no parser/grammar.
1 parent 9171dbe commit 4f60e2e

4 files changed

Lines changed: 787 additions & 0 deletions

File tree

‎tools/codegen/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
__pycache__/
2+
*.pyc

‎tools/codegen/README.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# MobilityNebula MEOS-operator codegen — design + generator
2+
3+
This directory contains the design proposal and Python generator for
4+
scaling MobilityNebula's MEOS-operator surface from the current
5+
~17 hand-written operators (PRs #14, #15, #16, #17) to a larger
6+
fraction of MEOS' ~1,949 streamable public functions, mirroring the
7+
infrastructure parity that the Flink and Kafka platforms reached via
8+
their codegen + wirings stacks.
9+
10+
## Why codegen on Nebula
11+
12+
The streaming-platform parity audit
13+
([assessment](../../docs/berlinmod-streaming-forms.md)) shows:
14+
15+
| Platform | Wirable MEOS surface |
16+
|---|---:|
17+
| Flink | 2,097 / 2,097 (100%) via codegen + 5 generic wiring classes |
18+
| Kafka | 2,097 / 2,097 (100%) via codegen + 5 generic wiring classes |
19+
| **Nebula** | **~17 / 2,097 (~1%)** via hand-written 4-layer pipeline per function |
20+
21+
The Nebula gap is structural: each MEOS function on NebulaStream
22+
requires a full **4-layer pipeline tuple** — logical class, physical
23+
class, parser dispatch, lowering rule — totalling ~350–400 LOC of
24+
mostly-mechanical boilerplate per function. Hand-writing all of MEOS'
25+
streamable surface this way is multi-month engineering; codegen makes
26+
it tractable.
27+
28+
## What this codegen produces
29+
30+
For each MEOS scalar function `f` in the input list, the generator
31+
emits the four NebulaStream pipeline-layer files following the
32+
established style of the existing hand-written operators
33+
(`TemporalEDWithinGeometryLogicalFunction` etc.):
34+
35+
```
36+
nes-logical-operators/include/Functions/Meos/<NebulaName>LogicalFunction.hpp
37+
nes-logical-operators/src/Functions/Meos/<NebulaName>LogicalFunction.cpp
38+
nes-physical-operators/include/Functions/Meos/<NebulaName>PhysicalFunction.hpp
39+
nes-physical-operators/src/Functions/Meos/<NebulaName>PhysicalFunction.cpp
40+
```
41+
42+
Plus updates to:
43+
- `nes-logical-operators/src/Functions/Meos/CMakeLists.txt`
44+
- `nes-physical-operators/src/Functions/Meos/CMakeLists.txt`
45+
- Parser dispatch: a single block per generated function inserted into
46+
`nes-sql-parser/src/AntlrSQLQueryPlanCreator.cpp` (manual edit
47+
recommended; the generator emits the dispatch snippet for
48+
copy-paste)
49+
- Parser grammar: a single token per function added to
50+
`nes-sql-parser/AntlrSQL.g4` (same)
51+
52+
## Scope of this PR
53+
54+
**Generator infrastructure only.** No generated C++ committed. Reasons:
55+
56+
1. **Compile-environment constraint.** The generator's author cannot
57+
build NebulaStream (full C++23 + vcpkg toolchain). Committing
58+
unverified generated code would ship potentially broken operators.
59+
2. **Per-function review value.** Mariana (maintainer) can run the
60+
generator against a small input list (e.g. one MEOS family at a
61+
time), review the output, iterate on the templates if needed, and
62+
ship operators in follow-up PRs at a controlled pace.
63+
3. **Template iteration cost.** First-pass templates may need
64+
adjustment after the first build — better to land the generator
65+
and iterate on templates than to ship a large batch of generated
66+
operators that all have the same wrong shape.
67+
68+
## How to use the generator
69+
70+
```bash
71+
# Edit the input list to choose which MEOS functions to generate
72+
$EDITOR tools/codegen/codegen_input.example.json
73+
74+
# Run the generator
75+
python3 tools/codegen/codegen_nebula.py \
76+
--input tools/codegen/codegen_input.example.json \
77+
--output-root .
78+
79+
# Output:
80+
# nes-logical-operators/include/Functions/Meos/<NebulaName>LogicalFunction.hpp
81+
# nes-logical-operators/src/Functions/Meos/<NebulaName>LogicalFunction.cpp
82+
# nes-physical-operators/include/Functions/Meos/<NebulaName>PhysicalFunction.hpp
83+
# nes-physical-operators/src/Functions/Meos/<NebulaName>PhysicalFunction.cpp
84+
#
85+
# Plus a stderr-printed "parser snippet" per function that you paste into
86+
# nes-sql-parser/src/AntlrSQLQueryPlanCreator.cpp (the parser dispatch),
87+
# and a "grammar snippet" that you paste into AntlrSQL.g4
88+
```
89+
90+
## Input format
91+
92+
`codegen_input.example.json` is a list of MEOS-function descriptors.
93+
One descriptor per output operator:
94+
95+
```json
96+
{
97+
"operators": [
98+
{
99+
"nebula_name": "TemporalEDisjointGeometry",
100+
"sql_token": "TEMPORAL_EDISJOINT_GEOMETRY",
101+
"meos_call": "edisjoint_tgeo_geo",
102+
"args": [
103+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
104+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
105+
{"name": "timestamp","nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
106+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
107+
],
108+
"return_type": "int",
109+
"nautilus_return": "INT32",
110+
"build_temporal_point": true,
111+
"comment_one_liner": "Per-event ever-disjoint between a tgeompoint built from event fields and a static geometry."
112+
}
113+
]
114+
}
115+
```
116+
117+
Field meanings:
118+
- `nebula_name`: PascalCase NebulaStream class name (without `LogicalFunction` / `PhysicalFunction` suffix; the generator adds those)
119+
- `sql_token`: the uppercase SQL function name (Antlr lexer token)
120+
- `meos_call`: the underlying MEOS C function symbol the physical operator wraps
121+
- `args`: ordered list of per-record argument fields; the generator builds the constructor + `parameters` vector from these
122+
- `return_type` / `nautilus_return`: the MEOS function's C return type and the NebulaStream `DataType::Type` enum value
123+
- `build_temporal_point`: if true, the physical operator builds a single-instant tgeompoint from `(lon, lat, timestamp)` before calling MEOS (the common pattern for spatial predicates); if false, the operator passes args directly to MEOS
124+
- `comment_one_liner`: drops into the Javadoc-equivalent C++ doc comment
125+
126+
## Templates
127+
128+
The generator's templates are embedded in the Python source as
129+
multi-line f-strings. They mirror the exact layout of the existing
130+
hand-written operators (`TemporalEDWithinGeometryLogicalFunction` and
131+
its physical sibling are the reference; the templates were derived by
132+
1:1 inspection of those files).
133+
134+
To adjust a template (e.g. when NebulaStream's `LogicalFunctionConcept`
135+
adds a new override), edit the corresponding string in
136+
`codegen_nebula.py`; the change applies to all subsequent
137+
regenerations.
138+
139+
## Scaling path (recommended sequence)
140+
141+
| Wave | Scope | Expected output | Effort estimate |
142+
|---|---|---|---|
143+
| W1 | First batch: 5 MEOS spatial-relation E/A predicates (e.g. `TemporalEDisjoint`, `TemporalATouches`, `TemporalECovers`, `TemporalACrosses`, `TemporalAOverlaps`) | 20 generated files + 5 parser entries | Single follow-up PR after this generator lands |
144+
| W2 | All ever / always spatial-relation predicates over `tgeo_geo` (~18 functions) | 72 generated files | ~1 follow-up PR |
145+
| W3 | Distance functions over `tgeo_geo` and `tgeo_tgeo` (NAD, NAI, distance, etc.) | ~30 generated files | ~1 follow-up PR |
146+
| W4 | Scalar accessors that decompose to per-event reads | template extension required (read MEOS handle) | design decision point |
147+
| W5 | Aggregations (windowed / cross-stream) | separate generator (aggregation 4-layer pattern is different from scalar 4-layer pattern; the existing TEMPORAL_LENGTH / PAIR_MEETING / CROSS_DISTANCE shape) | full aggregation-codegen design |
148+
149+
Per-PR scope keeps the review surface small and lets each batch land
150+
with its own build verification.
151+
152+
## What the generator does NOT do (deliberately)
153+
154+
- **No build-system integration.** The CMakeLists updates are emitted
155+
as text snippets for the maintainer to apply manually. This avoids
156+
the generator silently corrupting CMakeLists on regeneration.
157+
- **No parser/grammar integration.** Same reason — the dispatch and
158+
grammar snippets are emitted to stderr for manual paste.
159+
- **No aggregation-pattern support yet.** Aggregations require a
160+
different 4-layer shape (lift/combine/lower/cleanup) that depends
161+
on per-aggregation state design. A separate generator with the
162+
aggregation-specific template is W5 in the table above.
163+
164+
## Compile-verification note
165+
166+
The generator's first output should be reviewed against an existing
167+
hand-written operator for shape parity, then `mvn compile` (or the
168+
NebulaStream `cmake --build` equivalent) should be run against a
169+
single small batch (1–2 generated functions) before scaling up. The
170+
generator's templates are derived 1:1 from the existing operator
171+
shape but have not been compile-tested in this PR (out of the
172+
generator author's environment).

‎tools/codegen/codegen_input.example.json‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
{
2+
"_comment": "Example input for codegen_nebula.py — first wave of MEOS spatial-relation E/A predicates. Each operator descriptor produces one logical .hpp/.cpp + one physical .hpp/.cpp file. Adjust the list to control which functions get generated.",
3+
"operators": [
4+
{
5+
"nebula_name": "TemporalEDisjointGeometry",
6+
"sql_token": "TEMPORAL_EDISJOINT_GEOMETRY",
7+
"meos_call": "edisjoint_tgeo_geo",
8+
"args": [
9+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
10+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
11+
{"name": "timestamp", "nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
12+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
13+
],
14+
"return_type": "int",
15+
"nautilus_return": "INT32",
16+
"build_temporal_point": true,
17+
"comment_one_liner": "Per-event ever-disjoint between a single-instant tgeompoint built from event fields and a static geometry."
18+
},
19+
{
20+
"nebula_name": "TemporalATouchesGeometry",
21+
"sql_token": "TEMPORAL_ATOUCHES_GEOMETRY",
22+
"meos_call": "atouches_tgeo_geo",
23+
"args": [
24+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
25+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
26+
{"name": "timestamp", "nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
27+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
28+
],
29+
"return_type": "int",
30+
"nautilus_return": "INT32",
31+
"build_temporal_point": true,
32+
"comment_one_liner": "Per-event always-touches between a single-instant tgeompoint and a static geometry."
33+
},
34+
{
35+
"nebula_name": "TemporalECoversGeometry",
36+
"sql_token": "TEMPORAL_ECOVERS_GEOMETRY",
37+
"meos_call": "ecovers_tgeo_geo",
38+
"args": [
39+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
40+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
41+
{"name": "timestamp", "nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
42+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
43+
],
44+
"return_type": "int",
45+
"nautilus_return": "INT32",
46+
"build_temporal_point": true,
47+
"comment_one_liner": "Per-event ever-covers between a single-instant tgeompoint and a static geometry."
48+
},
49+
{
50+
"nebula_name": "TemporalACrossesGeometry",
51+
"sql_token": "TEMPORAL_ACROSSES_GEOMETRY",
52+
"meos_call": "acrosses_tgeo_geo",
53+
"args": [
54+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
55+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
56+
{"name": "timestamp", "nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
57+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
58+
],
59+
"return_type": "int",
60+
"nautilus_return": "INT32",
61+
"build_temporal_point": true,
62+
"comment_one_liner": "Per-event always-crosses between a single-instant tgeompoint and a static geometry."
63+
},
64+
{
65+
"nebula_name": "TemporalEOverlapsGeometry",
66+
"sql_token": "TEMPORAL_EOVERLAPS_GEOMETRY",
67+
"meos_call": "eoverlaps_tgeo_geo",
68+
"args": [
69+
{"name": "lon", "nautilus_type": "double", "cpp_type": "double"},
70+
{"name": "lat", "nautilus_type": "double", "cpp_type": "double"},
71+
{"name": "timestamp", "nautilus_type": "uint64_t", "cpp_type": "uint64_t"},
72+
{"name": "geometry", "nautilus_type": "VariableSizedData", "cpp_type": "const char*"}
73+
],
74+
"return_type": "int",
75+
"nautilus_return": "INT32",
76+
"build_temporal_point": true,
77+
"comment_one_liner": "Per-event ever-overlaps between a single-instant tgeompoint and a static geometry."
78+
}
79+
]
80+
}

0 commit comments

Comments
 (0)