feat: Deletion and renaming of columns with upgrade of delta sharing server

[bidhan-nagarro]

What type of PR is this?

• feat: Commits that add a new feature

Summary

What does this PR do

Upgrades Delta sharing server to support deletion and renaming of columns

how to test

Phase 1: Trigger Schema Migration

1. Sensor: In the Dagster UI, ensure migrations__schema_sensor is ON.
2. Watch Logs: Wait for the migrate__schema job to finish.
3. Verification:
   • Check the metaschema.school_geolocation table in Trino.
   • Shows the output of DESCRIBE delta_lake.school_geolocation.<country_code>; showing the new column name.

---

Phase 2: Ingest Data

1. Upload: Use the Ingestion Portal to upload
2. Process:
   • Trigger the school_master_geolocation__raw_file_uploads_sensor.
   • Observe the logs. You should see: Detected renames: {'school_name': 'official_school_name'}.
   • Shows the Dagster logs where it executes ALTER TABLE ... RENAME COLUMN.

---

Phase 3: Data Validation

1. Query: Run the following in Trino:

   SELECT official_school_name FROM delta_lake.school_geolocation.<country_code> LIMIT 5;

2. Shows the successful query results with actual data under the new column name.

---

Phase 4: Data Sharing

1. Start Sharing: Stop Dagster and start the giga-data-sharing server.
2. Client Test: Use a Delta Sharing client (e.g., Python delta-sharing library) to fetch the table.
3. **This confirms the version upgrade (1.3.10) fixed the sharing issue.

---