Skip to content

AGENTS.md

Latest commit

 

History

History
401 lines (288 loc) · 21.8 KB

AGENTS.md

File metadata and controls

401 lines (288 loc) · 21.8 KB
Raw

Instructions for Coding Agents

This file contains instructions for developers working on the Azure Search and OpenAI demo application. It covers the overall code layout, how to add new data, how to add new azd environment variables, how to add new developer settings, and how to add tests for new features.

Always keep this file up to date with any changes to the codebase or development process. If necessary, edit this file to ensure it accurately reflects the current state of the project.

Overall code layout

  • app: Contains the main application code, including frontend and backend.
    • app/backend: Contains the Python backend code, written with Quart framework.
      • app/backend/approaches: Contains the different approaches
        • app/backend/approaches/approach.py: Base class for all approaches
        • app/backend/approaches/chatreadretrieveread.py: Chat approach, includes query rewriting step first
        • app/backend/approaches/promptmanager.py: Manages loading and rendering of Jinja2 prompt templates
        • app/backend/approaches/prompts/query_rewrite.system.jinja2: Jinja2 template used to rewrite the query based off search history into a better search query
        • app/backend/approaches/prompts/chat_query_rewrite_tools.json: Tools used by the query rewriting prompt
        • app/backend/approaches/prompts/chat_answer.system.jinja2: Jinja2 template for the system message used by the Chat approach to answer questions
        • app/backend/approaches/prompts/chat_answer.user.jinja2: Jinja2 template for the user message used by the Chat approach, including sources
      • app/backend/prepdocslib: Contains the document ingestion library used by both local and cloud ingestion
        • app/backend/prepdocslib/blobmanager.py: Manages uploads to Azure Blob Storage
        • app/backend/prepdocslib/cloudingestionstrategy.py: Builds the Azure AI Search indexer and skillset for the cloud ingestion pipeline
        • app/backend/prepdocslib/csvparser.py: Parses CSV files
        • app/backend/prepdocslib/embeddings.py: Generates embeddings for text and images using Azure OpenAI
        • app/backend/prepdocslib/figureprocessor.py: Generates figure descriptions for both local ingestion and the cloud figure-processor skill
        • app/backend/prepdocslib/fileprocessor.py: Orchestrates parsing and chunking of individual files
        • app/backend/prepdocslib/filestrategy.py: Strategy for uploading and indexing files (local ingestion)
        • app/backend/prepdocslib/htmlparser.py: Parses HTML files
        • app/backend/prepdocslib/integratedvectorizerstrategy.py: Strategy using Azure AI Search integrated vectorization
        • app/backend/prepdocslib/jsonparser.py: Parses JSON files
        • app/backend/prepdocslib/listfilestrategy.py: Lists files from local filesystem or Azure Data Lake
        • app/backend/prepdocslib/mediadescriber.py: Interfaces for describing images (Azure OpenAI GPT-4o, Content Understanding)
        • app/backend/prepdocslib/page.py: Data classes for pages, images, and chunks
        • app/backend/prepdocslib/parser.py: Base parser interface
        • app/backend/prepdocslib/pdfparser.py: Parses PDFs using Azure Document Intelligence or local parser
        • app/backend/prepdocslib/searchmanager.py: Manages Azure AI Search index creation and updates
        • app/backend/prepdocslib/servicesetup.py: Shared service setup helpers for OpenAI, embeddings, blob storage, etc.
        • app/backend/prepdocslib/strategy.py: Base strategy interface for document ingestion
        • app/backend/prepdocslib/textparser.py: Parses plain text and markdown files
        • app/backend/prepdocslib/textprocessor.py: Processes text chunks for cloud ingestion (merges figures, generates embeddings)
        • app/backend/prepdocslib/textsplitter.py: Splits text into chunks using different strategies
      • app/backend/app.py: The main entry point for the backend application.
    • app/functions: Azure Functions used for cloud ingestion custom skills (document extraction, figure processing, text processing). Each function bundles a synchronized copy of prepdocslib; run python scripts/copy_prepdocslib.py to refresh the local copies if you modify the library.
    • app/frontend: Contains the React frontend code, built with TypeScript, built with vite.
      • app/frontend/src/api: Contains the API client code for communicating with the backend.
      • app/frontend/src/components: Contains the React components for the frontend.
      • app/frontend/src/locales: Contains the translation files for internationalization.
        • app/frontend/src/locales/da/translation.json: Danish translations
        • app/frontend/src/locales/en/translation.json: English translations
        • app/frontend/src/locales/es/translation.json: Spanish translations
        • app/frontend/src/locales/fr/translation.json: French translations
        • app/frontend/src/locales/it/translation.json: Italian translations
        • app/frontend/src/locales/ja/translation.json: Japanese translations
        • app/frontend/src/locales/nl/translation.json: Dutch translations
        • app/frontend/src/locales/ptBR/translation.json: Portuguese translations
        • app/frontend/src/locales/tr/translation.json: Turkish translations
      • app/frontend/src/pages: Contains the main pages of the application
  • infra: Contains the Bicep templates for provisioning Azure resources.
  • evals: Contains evaluation configs, datasets, and results.
    • evals/results: Contains raw per-run eval output folders. Use descriptive setup-based names for repeated runs, such as gpt54-low-top5-run1.
    • evals/results_summaries: Contains derived grouped summaries such as baseline.json and baseline.md.
    • evals/results_comparisons: Reserved for derived candidate-vs-baseline comparison artifacts.
    • evals/eval_compare.py: Compares eval result folders and reports averages, confidence intervals, and paired significance tests.
  • tests: Contains the test code, including e2e tests, app integration tests, and unit tests.

Adding new data

New files should be added to the data folder, and then either run scripts/prepdocs.sh or scripts/prepdocs.ps1 to ingest the data.

Adding a new azd environment variable

An azd environment variable is stored by the azd CLI for each environment. It is passed to the "azd up" command and can configure both provisioning options and application settings. When adding new azd environment variables, update:

  1. infra/main.parameters.json : Add the new parameter with a Bicep-friendly variable name and map to the new environment variable
  2. infra/main.bicep: Add the new Bicep parameter at the top, and add it to the appEnvVariables object
  3. .azdo/pipelines/azure-dev.yml: Add the new environment variable under env section
  4. .github/workflows/azure-dev.yml: Add the new environment variable under env section

You may also need to update:

  1. app/backend/prepdocs.py: If the variable is used in the ingestion script, retrieve it from environment variables here. Not always needed.
  2. app/backend/app.py: If the variable is used in the backend application, retrieve it from environment variables in setup_clients() function. Not always needed.

Adding a new setting to "Developer Settings" in RAG app

When adding a new developer setting, update:

  • frontend:

    • app/frontend/src/api/models.ts : Add to ChatAppRequestOverrides
    • app/frontend/src/components/Settings.tsx : Add a UI element for the setting
    • app/frontend/src/locales/*/translations.json: Add a translation for the setting label/tooltip for all languages
    • app/frontend/src/pages/chat/Chat.tsx: Add the setting to the component, pass it to Settings

  • backend:

    • app/backend/approaches/chatreadretrieveread.py : Retrieve from overrides parameter
    • app/backend/app.py: Some settings may need to be sent down in the /config route.

When adding tests for a new feature

All tests are in the tests folder and use the pytest framework. There are three styles of tests:

  • e2e tests: These use playwright to run the app in a browser and test the UI end-to-end. They are in e2e.py and they mock the backend using the snapshots from the app tests. (Before running e2e tests, make sure to run npm run build in app/frontend first to build the frontend code.)
  • app integration tests: Mostly in test_app.py, these test the app's API endpoints and use mocks for services like Azure OpenAI and Azure Search.
  • unit tests: The rest of the tests are unit tests that test individual functions and methods. They are in test_*.py files.

When adding a new feature, add tests for it in the appropriate file. If the feature is a UI element, add an e2e test for it. If it is an API endpoint, add an app integration test for it. If it is a function or method, add a unit test for it. Use mocks from tests/conftest.py to mock external services. Prefer mocking at the HTTP/requests level when possible.

When you're running tests, make sure you activate the .venv virtual environment first:

source .venv/bin/activate

To check for coverage, run the following command:

pytest --cov --cov-report=annotate:cov_annotate

Open the cov_annotate directory to view the annotated source code. There will be one file per source file. If a file has 100% source coverage, it means all lines are covered by tests, so you do not need to open the file.

For each file that has less than 100% test coverage, find the matching file in cov_annotate and review the file.

If a line starts with a ! (exclamation mark), it means that the line is not covered by tests. Add tests to cover the missing lines.

Sending pull requests

When sending pull requests, make sure to follow the PULL_REQUEST_TEMPLATE.md format.

Upgrading dependencies

Python backend dependencies

To upgrade a particular package in the backend, use the following command, replacing <package-name> with the name of the package you want to upgrade:

cd app/backend && uv pip compile requirements.in -o requirements.txt --python-version 3.10 --upgrade-package <package-name>

After upgrading, run tests to verify compatibility:

source .venv/bin/activate
pytest tests/

npm frontend dependencies

To upgrade a particular package in the frontend:

  1. Navigate to the frontend directory:

    cd app/frontend

  2. Upgrade the package (replace <package-name> with the package you want to upgrade):

    npm install <package-name>@latest

  3. Build the frontend to verify the upgrade works:

    npm run build

  4. Run all tests to ensure nothing broke:

    # Run e2e tests from the root directory
cd ../..
source .venv/bin/activate
pytest tests/e2e.py

  5. Commit changes if the upgrade is successful:

    git add package.json package-lock.json
git commit -m "chore: upgrade <package-name> to <version>"

Important notes for frontend upgrades:

  • When upgrading React or related core packages, you may need to upgrade multiple packages together (e.g., react, react-dom, @types/react, @types/react-dom)
  • Some upgrades may require code changes for API compatibility - check the package's changelog
  • For major version upgrades of UI libraries like Fluent UI or MSAL, review breaking changes carefully. Manual tests are required for any MSAL changes since the E2E tests do not cover authentication flows.
  • If npm reports peer dependency conflicts, the .npmrc file has legacy-peer-deps=true which allows the install to proceed. This is currently needed because react-helmet-async declares peer dependencies on React 17/18, but works fine with React 19.

Manual test plan for authentication changes (msal, msal-browser, authentication.py)

The unit tests mock msal at the client level, so any change to msal, @azure/msal-browser, cryptography, or app/backend/core/authentication.py needs a live deploy against Entra to confirm the on-behalf-of (OBO) flow, token cache, and Container Apps Easy Auth integration still work.

Use a dedicated azd env with login + access control enabled so the OBO code path is actually exercised.

  1. Create the env and enable auth: (any env name works — the steps below use <AUTH_ENV_NAME> as a placeholder)

    azd env new <AUTH_ENV_NAME> --subscription <SUB_ID> --location <REGION>
azd env set AZURE_USE_AUTHENTICATION true
azd env set AZURE_ENFORCE_ACCESS_CONTROL true
azd env set AZURE_ENABLE_UNAUTHENTICATED_ACCESS false
azd env set AZURE_AUTH_TENANT_ID <YOUR_TENANT_ID>
azd env set AZURE_TENANT_ID <YOUR_TENANT_ID>
# If the chosen region is capacity-constrained for Azure AI Search,
# override just the Search location:
azd env set AZURE_SEARCH_SERVICE_LOCATION <OTHER_REGION>

  2. Provision + deploy:

    ./scripts/auth_init.sh   # creates the client + server Entra app registrations
azd up -e <AUTH_ENV_NAME>

    azd up runs auth_update.sh as a postprovision hook to update the client app's redirect URIs.

  3. Ingest data for this env. The .md5 marker files under data/ are shared across envs and cause prepdocs to skip everything on a fresh env, leaving the search index empty. Remove them first, then re-ingest:

    rm data/*.md5
./scripts/prepdocs.sh

  4. Apply an ACL to at least one document for your user oid (find your oid via az ad signed-in-user show --query id -o tsv):

    python scripts/manageacl.py -v --acl-action add --acl-type oids \
  --acl <YOUR_OID> \
  --url "https://<storage-account>.blob.core.windows.net/content/Northwind_Standard_Benefits_Details.pdf"

  5. Verify auth enforcement with curl (both should return HTTP 401 because Container Apps Easy Auth blocks unauthenticated traffic before the app sees the request):

    BASE=https://<your-backend-fqdn>
curl -sS -o /dev/null -w "%{http_code}\n" "$BASE/auth_setup"
curl -sS -o /dev/null -w "%{http_code}\n" -X POST "$BASE/chat" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hi"}]}'

  6. Verify the OBO flow in the browser:

    • Load the app URL — you should be redirected to Entra sign-in.
    • Sign in with the tenant user whose oid you ACL'd.
    • Ask a question that only the ACL'd document can answer (e.g. "What is included in the Northwind Standard plan?"). You should get an answer with citations only from that document.
    • Ask a question about a document that is NOT ACL'd to you (e.g. "What is included in the Northwind Health Plus plan?"). You should get "I don't know" / no citations. This confirms the OBO token was issued by msal.ConfidentialClientApplication.acquire_token_on_behalf_of and correctly passed to Azure AI Search as the access-control filter.

  7. Verify the popup login flow works in local dev. The deployed Container Apps site uses Easy Auth redirect for login; the local dev server uses the MSAL popup flow, which is a completely different code path. Do NOT skip this — the msal-browser 5.x popup requires /redirect to serve the dedicated app/frontend/redirect.html bridge page (in dev the vite server owns it via a /redirect → /redirect.html middleware rewrite; in prod the Quart /redirect route serves the built redirect.html from static/), and previous upgrades have silently broken this.

    # Terminal 1: backend, pointed at the deployed login env's config
azd env select <AUTH_ENV_NAME>
PORT=50505 ./app/start.sh

# Terminal 2: frontend dev server
cd app/frontend && BACKEND_PORT=50505 npm run dev

    Open http://localhost:5173/, click Login, complete the Entra popup, and confirm:

    • The popup closes automatically after auth (does NOT leave the popup stuck on http://localhost:5173/redirect#code=...).
    • The Login button in the top bar switches to your username.
    • Ask a question and confirm ACL-filtered citations come back the same as in step 6.
    • Click Logout, then Login again. The popup should complete cleanly a second time.

    If the popup gets stuck on the redirect URL, check that /redirect serves the dedicated app/frontend/redirect.html page (built from app/frontend/src/redirect.ts) which runs broadcastResponseToMainFrame() from @azure/msal-browser/redirect-bridge. Per MSAL best practice this must be a minimal HTML page with ONLY the bridge script — no routing, no other application code. msal-browser 5.x uses a BroadcastChannel handshake and a truly blank redirect page no longer works.

  8. Optional: log out and reload on the deployed site. Confirms Easy Auth logout works and re-auth kicks in cleanly.

If any step fails, check container logs — AuthError from app/backend/core/authentication.py usually indicates the OBO token exchange or token validation broke.

Manual test plan for msgraph-sdk / microsoft-kiota-* changes

msgraph-sdk (and its transitive microsoft-kiota-* deps) is only imported by scripts/auth_init.py and scripts/auth_update.py, which register Entra client + server apps for the login-enabled deploy. The unit tests in tests/test_auth_init.py mock GraphServiceClient end-to-end, so a bump can pass CI while breaking a real Graph call (usually due to renamed request-body classes or new required fields).

Validate against a live tenant. If you don't already have a login-enabled deploy, follow steps 1–2 of the msal test plan above to create + provision an auth-enabled azd env (any env name works — the steps below use $AZURE_ENV_NAME from your current azd env).

  1. First run — exercises PATCH + query paths only. If the client and server app registrations from a prior run already exist, auth_init.sh short-circuits the applications.post() create path:

    ./scripts/auth_init.sh

    Look for Application already exists, not creating new one — that means the POST paths were skipped. The PATCH paths (applications.by_application_id().patch()) for permissions + known-client-apps did run, plus oauth2_permission_grants queries.

  2. Force the create path to exercise applications.post(), service_principals.post(), and applications.by_application_id().add_password.post() (where major SDK bumps most often break):

    # Grab the existing app IDs from the azd env
CLIENT_APP=$(azd env get-value AZURE_CLIENT_APP_ID)
SERVER_APP=$(azd env get-value AZURE_SERVER_APP_ID)

# Delete both Entra app registrations (safe on a test env)
az ad app delete --id $CLIENT_APP
az ad app delete --id $SERVER_APP

# Clear the cached IDs and secrets so auth_init recreates from scratch
azd env set AZURE_CLIENT_APP_ID ""
azd env set AZURE_CLIENT_APP_SECRET ""
azd env set AZURE_SERVER_APP_ID ""
azd env set AZURE_SERVER_APP_SECRET ""

./scripts/auth_init.sh

    You should see Creating application registration (twice — once for server, once for client), followed by Granted admin consent for ... messages. Every msgraph SDK code path in auth_init.py is now exercised.

  3. Test the update path by running the postprovision hook that updates client-app redirect URIs:

    ./scripts/auth_update.sh

    Look for Application update for client app id ... complete.

  4. Re-run the msal test plan end-to-end (browser sign-in, ACL'd doc question, non-ACL'd doc question) since the fresh apps will have new client IDs. azd deploy backend after auth_init.sh picks up the new IDs.

If POST fails with a serialization / model-class error (e.g. AttributeError on a model instance, or a 400 from Graph complaining about a missing property), the bumped SDK likely renamed or relocated a request-body class. Check the msgraph-sdk release notes and update the corresponding import in scripts/auth_init.py.

Checking Python type hints

To check Python type hints, use the following command:

ty check

Note that we do not currently enforce type hints in the tests folder, as it would require adding a lot of # type: ignore comments to the existing tests. We only enforce type hints in the main application code and scripts.

Python code style

Do not use single underscores in front of "private" methods or variables in Python code. We do not follow that convention in this codebase, since this is an application and not a library.

Starting the app locally

The simplest way to start the app is with ./app/start.sh (or ./app/start.ps1 on Windows). This builds the frontend and starts the backend — no separate frontend process needed unless you want hot reloading.

To avoid port conflicts (e.g. if another instance is already running), pick a random port:

PORT=50506 ./app/start.sh

On Windows (PowerShell):

$env:PORT = 50506
./app/start.ps1

If you also need the frontend dev server with hot reloading (for UI changes), run it separately with the matching BACKEND_PORT:

# Terminal 1: backend
PORT=50506 ./app/start.sh

# Terminal 2: frontend with HMR
cd app/frontend && BACKEND_PORT=50506 npm run dev

Tips for coding agents: Always specify your own random port via PORT to avoid colliding with a developer's running instance or other parallel agents. The start scripts will detect if the port is already in use and tell you to pick a different one.

Deploying the application

To deploy the application, use the azd CLI tool. Make sure you have the latest version of the azd CLI installed. Then, run the following command from the root of the repository:

azd up

That command will BOTH provision the Azure resources AND deploy the application code.

If you only changed the Bicep templates and want to re-provision the Azure resources, run:

azd provision

If you only changed the application code and want to re-deploy the code, run:

azd deploy

If you are using cloud ingestion and only want to deploy individual functions, run the necessary deploy commands, for example:

azd deploy document-extractor
azd deploy figure-processor
azd deploy text-processor