Feature/dq - #210
Merged
Merged
Conversation
* ✨ feat: Added migration for old data without metadata json file. * 🔧 chore: Update to refer to metadata.json from the blob * 🔥 chore: Update sh file to run migrate code for older countries * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: added command for migrating old metadata * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Changed the scripts path and added to infra init jobs yaml * chore: Fixed metadata path issue. * chore: Removed code for error fix in ApprovelRequest Model * chore: Removed code for error fix in FileUpload Model * chore: Removed code for error fix in ApprovalRequest Model * refactor: Change in the init jobs to add migrate as a separate job * chore: disable migrate blob job * chore: re-enable migrate blob job * refactor: migrate script modified * refactor: migrate script modified * refactor: migrate script modified * refactor: Migrate Script fixed * refactor: Migrate Script fixed * chore: make pre-commit fixes --------- Co-authored-by: nagarro <bidhan.mondal@nagarro.com>
…ew flow - remove API call from metadata continue for school-data - add review call to POST /upload/review with same payload as submit - keep submit using existing upload API - preserve polling and existing success/navigation behavior
* ✨ feat: Added migration for old data without metadata json file. * 🔧 chore: Update to refer to metadata.json from the blob * 🔥 chore: Update sh file to run migrate code for older countries * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: added command for migrating old metadata * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Changed the scripts path and added to infra init jobs yaml * chore: Fixed metadata path issue. * chore: Removed code for error fix in ApprovelRequest Model * chore: Removed code for error fix in FileUpload Model * chore: Removed code for error fix in ApprovalRequest Model * refactor: Change in the init jobs to add migrate as a separate job * chore: disable migrate blob job * chore: re-enable migrate blob job * refactor: migrate script modified * refactor: migrate script modified * refactor: migrate script modified * refactor: Migrate Script fixed * refactor: Migrate Script fixed * chore: make pre-commit fixes --------- Co-authored-by: nagarro <bidhan.mondal@nagarro.com>
* ✨ feat: Added migration for old data without metadata json file. * 🔧 chore: Update to refer to metadata.json from the blob * 🔥 chore: Update sh file to run migrate code for older countries * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: added command for migrating old metadata * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Changed the scripts path and added to infra init jobs yaml * chore: Fixed metadata path issue. * chore: Removed code for error fix in ApprovelRequest Model * chore: Removed code for error fix in FileUpload Model * chore: Removed code for error fix in ApprovalRequest Model * refactor: Change in the init jobs to add migrate as a separate job * chore: disable migrate blob job * chore: re-enable migrate blob job * refactor: migrate script modified * refactor: migrate script modified * refactor: migrate script modified * refactor: Migrate Script fixed * refactor: Migrate Script fixed * chore: make pre-commit fixes --------- Co-authored-by: nagarro <bidhan.mondal@nagarro.com>
* ✨ feat: Added migration for old data without metadata json file. * 🔧 chore: Update to refer to metadata.json from the blob * 🔥 chore: Update sh file to run migrate code for older countries * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: Migration file updated * 🔥 chore: added command for migrating old metadata * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Addressed Review Comments * chore: Changed the scripts path and added to infra init jobs yaml * chore: Fixed metadata path issue. * chore: Removed code for error fix in ApprovelRequest Model * chore: Removed code for error fix in FileUpload Model * chore: Removed code for error fix in ApprovalRequest Model * refactor: Change in the init jobs to add migrate as a separate job * chore: disable migrate blob job * chore: re-enable migrate blob job * refactor: migrate script modified * refactor: migrate script modified * refactor: migrate script modified * refactor: Migrate Script fixed * refactor: Migrate Script fixed * chore: make pre-commit fixes --------- Co-authored-by: nagarro <bidhan.mondal@nagarro.com>
bidhan-nagarro
force-pushed
the
feature/dq
branch
from
March 9, 2026 11:08
65487a3 to
51a9f21
Compare
Collaborator
There was a problem hiding this comment.
These changes should be not included since they were reverted on this PR: https://github.com/unicef/giga-data-ingestion/pull/205/changes
brianmusisi
reviewed
May 21, 2026
|
|
||
| from .base import BaseModel | ||
|
|
||
|
|
Collaborator
There was a problem hiding this comment.
Let's add a comment here that describes what the DQMode is i.e
uploaded - run the DQ checks on the uploaded file only, without master. Do not push for approval to master
master - run the DQ checks on the uploaded file, include checks on the current master, and push for approval to master
| }: { | ||
| params: UploadParams; | ||
| dq_mode: "uploaded" | "master"; | ||
| }) => api.uploads.review(params, dq_mode), |
Collaborator
There was a problem hiding this comment.
I think this flow may upload the same file twice. assessment creates a file upload through api.uploads.review(...), but the returned id is only stored locally as reviewUploadId. When the user continues to success, pendingSchoolDataPayload is still present and uploadId is still empty, so success calls api.uploads.upload(...) again. Should we persist the reviewed upload id and run the master check against that same upload instead?
| const isSubmitMode = hasPendingPayload; | ||
|
|
||
| const uploadFile = useMutation({ | ||
| mutationFn: api.uploads.upload, |
Collaborator
There was a problem hiding this comment.
Here is the other upload, where I think the uploadId is generated instead of using the same one from assesment.
| const isFileChecked = dqStatus === DQStatus.FILE_CHECKED; | ||
| const isCompleted = dqStatus === DQStatus.COMPLETED; | ||
|
|
||
| const checkTypeLabel = isCompleted |
Collaborator
There was a problem hiding this comment.
Since this string is going to be shown, I'd suggest to be some more humanized and inline with the rest of the text from the page, currently it would show:
Check type: UPLOADED_FILE_CHECK
Use instead:
- Master data check
- Uploaded file check
Resolved conflicts: - approval_requests.py: kept main's approval_status write + unconditional commit - schemas/upload.py: merged dq_mode (feature/dq) with mode/approval_status/data_owner/rows fields (main) - uploadSlice.ts: merged "Mixed" mode variant (main) with pendingSchoolDataPayload (feature/dq) - types/upload.ts: merged dq_mode (feature/dq) with new UploadResponse fields (main) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
brianmusisi
approved these changes
Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
feat: Commits that add a new featureSummary
What does this PR do
this PR creates a new Step for Data quality checks either with master or normally.