Skip to main content
Connectors

CircleCI connector

Set up the CircleCI connector in Kaivo: authentication, configuration, the 12 BigQuery tables it syncs, and answers to common questions.

Written By Lauri Raivio

Last updated 16 days ago

Kaivo is a fully managed data platform that syncs your CircleCI data into a Google BigQuery warehouse and keeps it up to date automatically. There is no pipeline to build and no infrastructure to run, so you can spend your time analysing your data from CircleCI instead of moving it.

What is the CircleCI connector

Sync your CircleCI data into BigQuery with Kaivo to analyse build and pipeline performance across your projects.

CategoryTech
AuthenticationAPI key
SetupSelf-service

Getting started with the CircleCI connector

  1. Sign up for Kaivo and create a workspace.
  2. Connect your CircleCI account.
  3. Choose which tables to sync.
  4. Wait for the initial sync to finish.
  5. Query your data in BigQuery or your favourite AI or BI tool.

Authenticating CircleCI

Authenticate with your API Key.

FieldDescription
API Key

Configuring the CircleCI connector

When you set up the connector, you provide:

FieldDescription
Organization ID

Your organization ID. Find it at https://app.circleci.com/settings/organization/circleci/<org-id>/overview.

Start Date

Any data before this date will not be fetched.

Project ID

Your project ID. Find it in the project settings at https://app.circleci.com/settings/project/circleci/<org-slug>/<project-id>.

Workflow ID

Workflow IDs to sync. Each workflow ID can be found in the pipeline build URL, e.g. https://app.circleci.com/pipelines/circleci/.../workflows/<workflow-id>.

Tables and columns synced from CircleCI

Kaivo syncs 12 tables from CircleCI into a dedicated dataset in your BigQuery warehouse. Click any table to see its columns and types.

ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
created_atSTRING
idSTRING
nameSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
org_idSTRING
project_idSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: insights_branches__branches

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
org_idSTRING
project_data__metrics__success_rateFLOAT64
project_data__metrics__throughputFLOAT64
project_data__metrics__total_credits_usedFLOAT64
project_data__metrics__total_duration_secsFLOAT64
project_data__metrics__total_runsFLOAT64
project_data__trends__success_rateFLOAT64
project_data__trends__throughputFLOAT64
project_data__trends__total_credits_usedFLOAT64
project_data__trends__total_duration_secsFLOAT64
project_data__trends__total_runsFLOAT64
project_idSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: insights_metrics__all_branches

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: insights_metrics__all_workflows

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: insights_metrics__project_workflow_branch_data

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
branchSTRING
metrics__p95_duration_secsFLOAT64
metrics__success_rateFLOAT64
metrics__total_credits_usedFLOAT64
metrics__total_runsFLOAT64
trends__p95_duration_secsFLOAT64
trends__success_rateFLOAT64
trends__total_credits_usedFLOAT64
trends__total_runsFLOAT64
workflow_nameSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: insights_metrics__project_workflow_data

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
metrics__p95_duration_secsFLOAT64
metrics__success_rateFLOAT64
metrics__total_credits_usedFLOAT64
metrics__total_runsFLOAT64
trends__p95_duration_secsFLOAT64
trends__success_rateFLOAT64
trends__total_credits_usedFLOAT64
trends__total_runsFLOAT64
workflow_nameSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
created_atSTRING
durationFLOAT64
executor__typeSTRING
executor__resource_classSTRING
latest_workflow__idSTRING
latest_workflow__nameSTRING
nameSTRING
numberFLOAT64
organization__nameSTRING
parallelismFLOAT64
pipeline__idSTRING
project__external_urlSTRING
project__idSTRING
project__nameSTRING
project__slugSTRING
queued_atSTRING
started_atSTRING
statusSTRING
stopped_atSTRING
web_urlSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: jobs__contexts

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: jobs__messages

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: jobs__parallel_runs

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
indexFLOAT64
statusSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
adminBOOL
analytics_idSTRING
basic_email_prefsSTRING
bitbucket_authorizedBOOL
containersFLOAT64
created_atSTRING
days_left_in_trialFLOAT64
dev_adminBOOL
email_authentication__email_verifiedBOOL
email_authentication__loginSTRING
in_beta_programBOOL
loginSTRING
nameSTRING
num_projects_followedFLOAT64
parallelismFLOAT64
privacy_optoutBOOL
pusher_idSTRING
selected_emailSTRING
sign_in_countFLOAT64
studentBOOL
trial_endSTRING
web_ui_pipelines_first_opt_inBOOL
web_ui_pipelines_optoutSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: me__all_emails

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: me__enrolled_betas

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: me__github_oauth_scopes

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
created_atSTRING
idSTRING
numberFLOAT64
project_slugSTRING
stateSTRING
trigger__typeSTRING
trigger__actor__loginSTRING
trigger__received_atSTRING
trigger_parameters__circleci__actor_idSTRING
trigger_parameters__circleci__event_timeSTRING
trigger_parameters__circleci__event_typeSTRING
trigger_parameters__circleci__provider_actor_idSTRING
trigger_parameters__circleci__provider_loginSTRING
trigger_parameters__circleci__provider_nameSTRING
trigger_parameters__circleci__trigger_typeSTRING
trigger_parameters__event__descriptionSTRING
trigger_parameters__event__titleSTRING
trigger_parameters__git__branchSTRING
trigger_parameters__git__checkout_shaSTRING
trigger_parameters__git__checkout_urlSTRING
trigger_parameters__git__refSTRING
trigger_parameters__git__repo_urlSTRING
trigger_parameters__github_app__branchSTRING
trigger_parameters__github_app__checkout_shaSTRING
trigger_parameters__github_app__commit_author_emailSTRING
trigger_parameters__github_app__commit_author_nameSTRING
trigger_parameters__github_app__commit_messageSTRING
trigger_parameters__github_app__commit_shaSTRING
trigger_parameters__github_app__commit_timestampSTRING
trigger_parameters__github_app__commit_titleSTRING
trigger_parameters__github_app__content_refSTRING
trigger_parameters__github_app__default_branchSTRING
trigger_parameters__github_app__forcedSTRING
trigger_parameters__github_app__ownerSTRING
trigger_parameters__github_app__refSTRING
trigger_parameters__github_app__repo_full_nameSTRING
trigger_parameters__github_app__repo_idSTRING
trigger_parameters__github_app__repo_nameSTRING
trigger_parameters__github_app__repo_urlSTRING
trigger_parameters__github_app__total_commits_countSTRING
trigger_parameters__github_app__user_avatarSTRING
trigger_parameters__github_app__user_idSTRING
trigger_parameters__github_app__user_nameSTRING
trigger_parameters__github_app__user_usernameSTRING
trigger_parameters__github_app__web_urlSTRING
trigger_parameters__webhook__bodySTRING
updated_atSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: pipelines__errors

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
branches__circleci_project_setup__is_using_workflowsBOOL
branches__circleci_project_setup__last_success__added_atSTRING
branches__circleci_project_setup__last_success__build_numFLOAT64
branches__circleci_project_setup__last_success__is_2_0_jobBOOL
branches__circleci_project_setup__last_success__is_workflow_jobBOOL
branches__circleci_project_setup__last_success__outcomeSTRING
branches__circleci_project_setup__last_success__pushed_atSTRING
branches__circleci_project_setup__last_success__statusSTRING
branches__circleci_project_setup__last_success__vcs_revisionSTRING
branches__circleci_project_setup__latest_completed_workflows__build__created_atSTRING
branches__circleci_project_setup__latest_completed_workflows__build__idSTRING
branches__circleci_project_setup__latest_completed_workflows__build__statusSTRING
branches__circleci_project_setup__latest_workflows__build__created_atSTRING
branches__circleci_project_setup__latest_workflows__build__idSTRING
branches__circleci_project_setup__latest_workflows__build__statusSTRING
branches__main__is_using_workflowsBOOL
branches__main__last_success__added_atSTRING
branches__main__last_success__build_numFLOAT64
branches__main__last_success__is_2_0_jobBOOL
branches__main__last_success__is_workflow_jobBOOL
branches__main__last_success__outcomeSTRING
branches__main__last_success__pushed_atSTRING
branches__main__last_success__statusSTRING
branches__main__last_success__vcs_revisionSTRING
branches__main__latest_completed_workflows__generate_config__created_atSTRING
branches__main__latest_completed_workflows__generate_config__idSTRING
branches__main__latest_completed_workflows__generate_config__statusSTRING
branches__main__latest_workflows__generate_config__created_atSTRING
branches__main__latest_workflows__generate_config__idSTRING
branches__main__latest_workflows__generate_config__statusSTRING
default_branchSTRING
followingBOOL
has_usable_keyBOOL
ossBOOL
reponameSTRING
usernameSTRING
vcs_typeSTRING
vcs_urlSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__circleci_project_setup__pusher_logins

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__circleci_project_setup__recent_builds

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
added_atSTRING
build_numFLOAT64
is_2_0_jobBOOL
is_workflow_jobBOOL
outcomeSTRING
pushed_atSTRING
statusSTRING
vcs_revisionSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__circleci_project_setup__running_builds

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__main__pusher_logins

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__main__recent_builds

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
added_atSTRING
build_numFLOAT64
is_2_0_jobBOOL
is_workflow_jobBOOL
outcomeSTRING
pushed_atSTRING
statusSTRING
vcs_revisionSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: projects__branches__main__running_builds

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
avatar_urlSTRING
idSTRING
nameSTRING
slugSTRING
vcs_typeSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
idSTRING
loginSTRING
nameSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
idSTRING
nameSTRING
organization_idSTRING
organization_nameSTRING
organization_slugSTRING
slugSTRING
vcs_info__default_branchSTRING
vcs_info__providerSTRING
vcs_info__vcs_urlSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
created_atSTRING
idSTRING
nameSTRING
pipeline_idSTRING
pipeline_numberFLOAT64
project_slugSTRING
started_bySTRING
statusSTRING
stopped_atSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.
ColumnTypeDescription
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
typeSTRING
idSTRING
job_numberFLOAT64
nameSTRING
project_slugSTRING
started_atSTRING
statusSTRING
stopped_atSTRING
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: workflow_jobs__dependencies

ColumnTypeDescription
_kaivo_parent_idSTRINGForeign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_idSTRINGPrimary key that uniquely identifies the row. Auto-generated by Kaivo.
valueJSON
_kaivo_extracted_atTIMESTAMPTimestamp that shows when the row was extracted. Auto-generated by Kaivo.

How the CircleCI sync works

After the first load, Kaivo keeps your BigQuery warehouse up to date for you. Where CircleCI supports it, each sync pulls only new and changed records so it stays fast; otherwise it refreshes the whole table. Every record keeps its original ID, so you won't get duplicate rows.

Frequently asked questions

How long does the initial sync take for CircleCI?

It depends on how much history is in your CircleCI account. Most initial syncs finish within minutes, while large accounts can take a few hours. After that, syncs only fetch new and changed records, so they're much faster.

Can I sync only some tables or columns?

Yes. You pick which tables to sync when you set up the connection and can change the selection later. Tables you don't select are never copied to your warehouse.

What happens when CircleCI's schema changes?

New fields are never added automatically. You choose which fields to sync, so data you haven't selected (sensitive personal data, for example) never lands in your warehouse. When a new field appears, it becomes available for you to add. What happens to removed or renamed fields depends on a table's sync mode: full-refresh tables always match what's currently in CircleCI, so dropped fields disappear, while incremental tables keep their existing columns and history, so an old field stays and newly added fields fill in over time.

How do I handle GDPR or data deletion requests?

Your data lives in your own Kaivo-managed BigQuery warehouse, so the most direct option is to delete or anonymise specific records right in BigQuery. If you delete data in CircleCI instead, full-refresh tables drop it on the next sync, while incremental tables keep it, so you would remove the row in BigQuery or ask us to run a full refresh. To remove everything, delete the CircleCI connector in Kaivo and all of its synced data is deleted with it.

Common use cases for CircleCI data

Pipeline performance

Use pipelines and workflows to track run frequency, duration, and failures over time.

Reliability

Use insights_metrics to find the slowest and most failure-prone workflows.

Project view

Join projects with jobs to compare CI health across repositories.

Use CircleCI data in your AI and BI tools

Once CircleCI data lands in your Kaivo-managed BigQuery warehouse, you can explore it with AI tools or any BI tool that connects to BigQuery. Here's how the most common destinations work with CircleCI data.

Claude

Use Kaivo's MCP server to give Claude secure, workspace-scoped access to your data. Setup guide →

Power BI

Microsoft's BI tool with a native BigQuery connector. Supports direct query and scheduled refresh. Setup guide →

Data Studio

Free Google BI tool with native BigQuery support. One-click connection to your Kaivo warehouse; great for SMB teams on Google Workspace. Setup guide →

Tableau

The premium analytics standard, with native BigQuery integration. Setup guide →

Google Sheets

Use Connected Sheets to query BigQuery directly from a spreadsheet, with no SQL. Setup guide →

Excel

Connect via Power Query's BigQuery connector. Setup guide →

Metabase

Open-source BI tool with strong BigQuery support. Setup guide →

See our pricing page for CircleCI connector pricing and plan details.

  • Adform: Sync Adform to BigQuery.
  • Amplitude: Sync Amplitude to BigQuery.
  • Auth0: Sync Auth0 to BigQuery.
  • Convex: Sync Convex to BigQuery.
  • GitHub: Sync GitHub to BigQuery.
  • GitLab: Sync GitLab to BigQuery.

Was this helpful?

Still need help? Share an idea