Add project-context.md for datastream-to-spanner#3902
Conversation
aasthabharill
force-pushed
the
datastream-to-spanner-context
branch
from
610c543 to
5595f80
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3902 +/- ##
============================================
+ Coverage 53.73% 62.04% +8.30%
+ Complexity 6743 2971 -3772
============================================
Files 1087 532 -555
Lines 66794 32141 -34653
Branches 7478 3515 -3963
============================================
- Hits 35890 19941 -15949
+ Misses 28477 11179 -17298
+ Partials 2427 1021 -1406
🚀 New features to boost your workflow:
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces documentation and architectural diagrams for the datastream-to-spanner Dataflow template. The goal is to provide a centralized source of truth regarding the pipeline's design, technical constraints, and operational guidelines to assist future development and maintenance. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds comprehensive architectural documentation for the Datastream to Spanner Dataflow template, including a Graphviz DOT file, its corresponding SVG diagram, and a detailed project context markdown file. Feedback on the architecture diagram points out a misleading label 'Write To DLQ2' and suggests renaming it to 'Write To Severe DLQ' to correctly represent that severe errors are routed to a subdirectory of the main DLQ rather than a separate resource.
| ## Technical Details | ||
|
|
||
| * **Tech Stack & Versions:** | ||
| * **Languages:** Java 17 |
There was a problem hiding this comment.
This information needs to be derived from the main readme page. How does AI agent knows to do that? Where is that instruction?
| * **Coding Standards & Best Practices:** | ||
| * Individual CEs are processed separately for parallel scaling, rather than grouping them into the original source transactions. Consistency is managed using lateness checks on the Shadow Tables. | ||
| * **Avoid Serial Processing:** Do not attempt to group events by transaction or serially order them. The approach relies on parallel workers, taking advantage of Cloud Dataflow's scale. | ||
| * **Avoid GroupBy:** Do not use `GroupByKey` or internal worker state to filter stale events before writing. It doesn't scale well and complicates state recovery. Always use Shadow Tables for the lateness check. |
There was a problem hiding this comment.
I'm not very sure about this point. I don't understand what it is trying to say.
4 participants
b/521743991