Skip to content

feat: Add configurable page size for project_items stream - #572

Merged
edgarrmondragon merged 2 commits into
MeltanoLabs:mainfrom
KennedyMurphy:km/configurable-project-items-page-size
Jun 8, 2026
Merged

feat: Add configurable page size for project_items stream#572
edgarrmondragon merged 2 commits into
MeltanoLabs:mainfrom
KennedyMurphy:km/configurable-project-items-page-size

Conversation

@KennedyMurphy

Copy link
Copy Markdown
Contributor

Summary

Add a stream_options.project_items.page_size config option for the project_items stream.

The default remains 100, matching the current hardcoded GitHub GraphQL page size and preserving existing behavior. Users can lower the value, for example to 50, to reduce per-request GraphQL query complexity for large Projects-v2 boards with many custom fields.

Example:

{
  "stream_options": {
    "project_items": {
      "page_size": 50
    }
  }
}

Motivation

ProjectItemsStream dynamically includes all known project fields in each item query. For large GitHub Projects-v2 boards with many custom fields, items(first: 100, ...) can produce expensive GraphQL requests that are more likely to hit GitHub timeout/query-complexity behavior.

Lowering the page size keeps cursor pagination semantics unchanged, so the result set is preserved, while reducing the amount of data and field-value expansion requested per GraphQL call. The tradeoff is more requests.

Validation

  • uv run pytest tests/test_tap.py -q -k project_items
  • uvx ruff check tap_github tests/test_tap.py
  • uv run mypy tap_github

Note: uv run ty check tap_github was attempted locally but the process was killed with exit code 137.


Generated with Pi.

Co-authored-by: AI (Pi/GPT-5.5) <noreply@pi.dev>
@KennedyMurphy
KennedyMurphy requested a review from a team as a code owner June 3, 2026 19:33
@KennedyMurphy KennedyMurphy changed the title Add configurable page size for project_items stream feat: Add configurable page size for project_items stream Jun 4, 2026
@edgarrmondragon

Copy link
Copy Markdown
Member

@KennedyMurphy thanks for the PR!

Can you fix the merge conflict?

Co-authored-by: AI (Pi/GPT-5.5) <noreply@pi.dev>
@KennedyMurphy

KennedyMurphy commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

@edgarrmondragon Fixed — merged latest upstream/main into the branch and resolved the tests/test_tap.py import conflict. The PR is now mergeable; pre-commit.ci is passing.

Local checks run:

  • uv run pytest tests/test_tap.py::test_project_items_query_uses_default_page_size tests/test_tap.py::test_project_items_query_uses_configured_page_size tests/test_tap.py::test_pull_requests_stream_skips_repos_with_pull_requests_disabled tests/test_tap.py::test_pull_requests_stream_delegates_when_pull_request_capability_is_not_false tests/test_tap.py::test_issues_stream_delegates_when_has_issues_is_false tests/test_tap.py::test_pull_requests_stream_keeps_generic_404_retriable -q
  • /tmp/tap-github-precommit/bin/pre-commit run --files README.md tap_github/organization_streams.py tap_github/tap.py tests/test_tap.py

Note: full uv run pytest tests/test_tap.py -q hit unauthenticated GitHub API rate limiting locally, so I verified the conflict-adjacent tests directly.


Generated with Pi.

@edgarrmondragon edgarrmondragon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @KennedyMurphy!

@edgarrmondragon edgarrmondragon changed the title feat: Add configurable page size for project_items stream feat: Add configurable page size for project_items stream Jun 8, 2026
@edgarrmondragon
edgarrmondragon added this pull request to the merge queue Jun 8, 2026
Merged via the queue into MeltanoLabs:main with commit c196849 Jun 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants