feat: TestCases with 70% coverage

[bidhan-nagarro]

What type of PR is this?

• test: Commits that add missing tests or correcting existing tests

Summary

What does this PR do

This PR adds unit test cases and tries 70% code coverage

how to test

1. Trigger the Test Suite locally (or on CI/CD):

   • Open your terminal and ensure you are in the dagster directory (or the root if using task).
   • Run the complete test suite. If you are using the Taskfile, you can run:

     task test

   • Alternatively, you can run the tests directly via poetry:

     cd dagster
     poetry run pytest --cov=src --cov-report=term-missing

2. Verify All Tests Pass:

   • Wait for pytest to finish executing the unit test cases we wrote (including Spark Session tests, Data Quality tests, and Asset tests).
   • Verification Checks:
     • The output should indicate that 100% of the newly added/fixed unit tests passed successfully.
   • Screenshot 5: Take a screenshot of the terminal output showing 100% passes and zero failures.

3. Verify Code Coverage:

   • Review the final table printed by pytest-cov at the end of the test execution.
   • Verification Checks:
     • The TOTAL coverage percentage line at the very bottom should be over 70%, verifying that the new coverage requirements are met.

[brianmusisi]

@bidhan-nagarro, most of these tests are not actually testing any actual functionality. Everything is being mocked, and so we are not testing any actual functionality.

What we need to do test that when we perform transformations, the results we get are what we expect. For example, if we have duplicates, those should be picked up.

If we receive invlaid values, they are being handled. That's what these should do. I don't see this being done anywhere for any of the assets

[brianmusisi]

Fix merge conflicts and then should be fine. Do not merge, I will merge when I'm back