GitHub Data Connector Deployment Guide

Production operating guide for the GitHub data connector covering authentication, GitHub API rate limits, and operational tuning.

Authentication & Secrets

The GitHub connector uses the GitHub REST and GraphQL APIs with a personal access token (PAT) or GitHub App installation token.

Parameter	Description
github_token	PAT or installation token. Use ${secrets:...} to resolve from a secret store.

Tokens must be sourced from a secret store in production. Scope the PAT to the minimum required permissions:

• Public repo data only: no token required, but see the rate-limit note below.
• Private repos: repo scope.
• Issues/PRs: repo (private) or public_repo (public).
• Org-level data: read:org.

For long-running deployments, prefer GitHub App tokens (installation tokens) over user PATs — they have higher rate limits (15,000/hr vs 5,000/hr per authenticated user) and are not tied to a specific user account.

Resilience Controls

Rate Limiting

GitHub's REST API rate limits:

Auth mode	Limit
Unauthenticated	60 requests/hr per IP
Authenticated (PAT)	5,000 requests/hr
GitHub App installation	15,000 requests/hr
Enterprise Server (typical)	Configurable

The connector respects GitHub's Retry-After and X-RateLimit-Reset headers and backs off accordingly. When the remaining budget falls below a small threshold, requests pause until the next reset window.

Pagination

Page width is chosen per table, not fixed at GitHub's 100-item maximum. GitHub enforces a per-request compute budget (GraphQL API resource limits announcement) that is separate from — and reached long before — the 500,000-node ceiling, and a page wide enough to exceed it is rejected outright with Resource limits for this query exceeded, every node in the page returned as null. Tables whose rows expand into many nested connections therefore request narrower pages: pulls is requested 25 at a time in both comment modes, while issues and milestones still use 100.

Datasets backed by high-volume endpoints (e.g., repos.commits on a monorepo) may require many hours to initially hydrate. Use incremental acceleration with a since filter where possible.

pulls scan ceilings differ by query mode

In github_query_mode: auto, the connector fails with Maximum pagination iterations (1000) exceeded after 1,000 pagination iterations (1,001 total page fetches counting the initial page), rather than silently truncating. At 25 rows per page, that bounds a pulls dataset at 25 x 1001 = 25,025 pull requests per refresh. In github_query_mode: search, GitHub Search has its own 1,000-result limit, which is the effective cap for a single query.

The page was wider in earlier releases, putting the arithmetic ceiling at 100,100 — but on a repository large enough for that to bind, the wider page was rejected by the compute budget and returned no rows at all. The number of pull requests actually reachable went up, not down.

Retry Behavior

Transient 5xx responses are retried with exponential backoff up to a bounded retry count. Permanent errors (401 Unauthorized, 404 Not Found, 422 Validation Failed) surface immediately.

Capacity & Sizing

• Throughput: Bounded by the rate limit, not network or CPU. Plan dataset refresh intervals to stay within the hourly budget.
• Latency: Expect ~100-500ms per paginated request against github.com; lower for GitHub Enterprise Server on the same network.
• Initial bootstrap: For high-volume datasets (e.g., all commits in a busy monorepo), the first materialization may exhaust the hourly budget across several runs. Plan staged ingestion if needed.

Metrics

The GitHub connector does not register connector-specific dataset-level instruments in the current release. Monitor via:

• Spice query execution metrics (query_duration_ms, query_returned_rows, query_failures) from runtime.metrics.
• GitHub's own rate-limit UI at /settings/tokens for token-level quota tracking.

See Component Metrics for general configuration.

Task History

GitHub API calls participate in task history through the HTTP client's span. Each page fetch is a child of the enclosing sql_query or accelerated_table_refresh task.

Known Limitations

• Read-only: The connector is read-only; writes (issue creation, PR comments) are not supported.
• GraphQL-only endpoints: Some GitHub data (e.g., discussions, project v2) requires GraphQL; check the connector's documented supported endpoints.
• GitHub Enterprise Cloud with IP allowlisting: The Spice runtime's outbound IP must be allow-listed.
• Secondary rate limits: GitHub enforces abuse-detection "secondary" rate limits on concentrated bursts, independent of the hourly primary limit. If hit, the connector backs off.

Troubleshooting

Symptom	Likely cause	Resolution
401 Bad credentials	PAT expired / revoked / wrong value.	Rotate the PAT; update the secret store.
403 rate limit exceeded	Primary hourly rate limit hit.	Increase refresh interval; switch to GitHub App auth for higher quota; use incremental refresh with since.
403 Secondary rate limit	Burst of concurrent requests tripped abuse detection.	Reduce concurrent refresh; connector will back off automatically.
404 Not Found on a private repo	Token lacks repo scope.	Regenerate PAT with repo scope.
Very slow initial hydration	Large dataset + strict rate limit.	Run first refresh off-peak; use since/updated_since for incremental refreshes.