Skip to content
Book a demoStart free
Connectors

PyPI connector

Set up the PyPI connector in Kaivo: authentication, configuration, the 3 BigQuery tables it syncs, and answers to common questions.

PyPI connector pricing, tier and credits on kaivo.io

Kaivo is a fully managed data platform that syncs your PyPI data into a Google BigQuery warehouse and keeps it up to date automatically. There is no pipeline to build and no infrastructure to run, so you can spend your time analysing your data from PyPI instead of moving it.

Sync your PyPI package data into BigQuery with Kaivo to track releases, download stats, and adoption trends for your packages.

CategoryTech
AuthenticationOther
SetupSelf-service
  1. Sign up for Kaivo and create a workspace.
  2. Connect your PyPI account.
  3. Choose which tables to sync.
  4. Wait for the initial sync to finish.
  5. Query your data in BigQuery or your favourite AI or BI tool.

Follow the steps below to connect your PyPI account.

When you set up the connector, you provide:

Field Description
PyPI Package
Name of the project/package. Can only be in lowercase with hyphen. This is the name used using pip command for installing the package.

| | Package Version |

Version of the project/package. Use it to find a particular release instead of all releases.

|

Kaivo syncs 3 tables from PyPI into a dedicated dataset in your BigQuery warehouse. Click any table to see its columns and types.

project (26 columns)
Column Type Description
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
info__version STRING A string containing the distributions version number in the format specified in PEP 440. Corresponds to https://packaging.python.org/specifications/core-metadata/#version
info__author STRING The name of the company or individual who created the project
info__author_email STRING The author’s email address
info__bugtrack_url STRING URL to find issues and bugs for the project
info__description STRING Corresponds to https://packaging.python.org/specifications/core-metadata/#description
info__description_content_type STRING Corresponds to https://packaging.python.org/specifications/core-metadata/#description-content-type
info__docs_url STRING URL to the project’s documentation
info__download_url STRING [DEPRECATED]
info__home_page STRING URL to project home page
info__keywords STRING Keywords to use for project searching
info__license STRING Project’s open source license
info__maintainer STRING Project maintainer name
info__maintainer_email STRING Project maintainer email address
info__name STRING Project’s raw (non-normailzed name)
info__package_url STRING URL to the project page
info__platform STRING [DEPRECATED]
info__project_url STRING URL to the project page
info__project_urls JSON Additional URLs that are relevant to your project. Corresponds to https://packaging.python.org/specifications/core-metadata/#project-url-multiple-use
info__release_url STRING URL of the release page of the version of the project
info__requires_python STRING Python runtime version required for project. Corresponds to https://packaging.python.org/specifications/core-metadata/#requires-python
info__summary STRING A one-line summary of what the distribution does.. Corresponds to https://packaging.python.org/specifications/core-metadata/#summary
info__yanked BOOL If the version has been yanked. As defined in PEP 592
info__yanked_reason STRING Reason for applying PEP 592 version yank
last_serial INT64 Monotonically increasing integer sequence that changes every time the project is updated
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value JSON
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
details STRING A description of the vulnerability
id STRING
link STRING
source STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: project__vulnerabilities__aliases

Section titled “Subtable: project__vulnerabilities__aliases”
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: project__vulnerabilities__fixed_in

Section titled “Subtable: project__vulnerabilities__fixed_in”
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
release (26 columns)
Column Type Description
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
info__version STRING A string containing the distributions version number in the format specified in PEP 440. Corresponds to https://packaging.python.org/specifications/core-metadata/#version
info__author STRING The name of the company or individual who created the project
info__author_email STRING The author’s email address
info__bugtrack_url STRING URL to find issues and bugs for the project
info__description STRING Corresponds to https://packaging.python.org/specifications/core-metadata/#description
info__description_content_type STRING Corresponds to https://packaging.python.org/specifications/core-metadata/#description-content-type
info__docs_url STRING URL to the project’s documentation
info__download_url STRING [DEPRECATED]
info__home_page STRING URL to project home page
info__keywords STRING Keywords to use for project searching
info__license STRING Project’s open source license
info__maintainer STRING Project maintainer name
info__maintainer_email STRING Project maintainer email address
info__name STRING Project’s raw (non-normailzed name)
info__package_url STRING URL to the project page
info__platform STRING [DEPRECATED]
info__project_url STRING URL to the project page
info__project_urls JSON Additional URLs that are relevant to your project. Corresponds to https://packaging.python.org/specifications/core-metadata/#project-url-multiple-use
info__release_url STRING URL of the release page of the version of the project
info__requires_python STRING Python runtime version required for project. Corresponds to https://packaging.python.org/specifications/core-metadata/#requires-python
info__summary STRING A one-line summary of what the distribution does.. Corresponds to https://packaging.python.org/specifications/core-metadata/#summary
info__yanked BOOL If the version has been yanked. As defined in PEP 592
info__yanked_reason STRING Reason for applying PEP 592 version yank
last_serial INT64 Monotonically increasing integer sequence that changes every time the project is updated
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value JSON
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
details STRING A description of the vulnerability
id STRING
link STRING
source STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: release__vulnerabilities__aliases

Section titled “Subtable: release__vulnerabilities__aliases”
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.

Subtable: release__vulnerabilities__fixed_in

Section titled “Subtable: release__vulnerabilities__fixed_in”
Column Type Description
_kaivo_parent_id STRING Foreign key referencing _kaivo_id in the parent table. Auto-generated by Kaivo.
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
value STRING
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.
stats (3 columns)
Column Type Description
_kaivo_id STRING Primary key that uniquely identifies the row. Auto-generated by Kaivo.
total_packages_size INT64 Total size of all packages combined in bytes
_kaivo_extracted_at TIMESTAMP Timestamp that shows when the row was extracted. Auto-generated by Kaivo.

After the first load, Kaivo keeps your BigQuery warehouse up to date for you. Where PyPI supports it, each sync pulls only new and changed records so it stays fast; otherwise it refreshes the whole table. Every record keeps its original ID, so you won’t get duplicate rows.

How long does the initial sync take for PyPI?

Section titled “How long does the initial sync take for PyPI?”

It depends on how much history is in your PyPI account. Most initial syncs finish within minutes, while large accounts can take a few hours. After that, syncs only fetch new and changed records, so they’re much faster.

Yes. You pick which tables to sync when you set up the connection and can change the selection later. Tables you don’t select are never copied to your warehouse.

What happens when PyPI’s schema changes?

Section titled “What happens when PyPI’s schema changes?”

New fields are never added automatically. You choose which fields to sync, so data you haven’t selected (sensitive personal data, for example) never lands in your warehouse. When a new field appears, it becomes available for you to add. What happens to removed or renamed fields depends on a table’s sync mode: full-refresh tables always match what’s currently in PyPI, so dropped fields disappear, while incremental tables keep their existing columns and history, so an old field stays and newly added fields fill in over time.

How do I handle GDPR or data deletion requests?

Section titled “How do I handle GDPR or data deletion requests?”

Your data lives in your own Kaivo-managed BigQuery warehouse, so the most direct option is to delete or anonymise specific records right in BigQuery. If you delete data in PyPI instead, full-refresh tables drop it on the next sync, while incremental tables keep it, so you would remove the row in BigQuery or ask us to run a full refresh. To remove everything, delete the PyPI connector in Kaivo and all of its synced data is deleted with it.

Use stats to track package downloads over time.

Join project with release to document your release history.

Bring PyPI data into BigQuery to study how adoption of your packages grows.

Once PyPI data lands in your Kaivo-managed BigQuery warehouse, you can explore it with AI tools or any BI tool that connects to BigQuery. Here’s how the most common destinations work with PyPI data.

Use Kaivo’s MCP server to give Claude secure, workspace-scoped access to your data. Setup guide →

Microsoft’s BI tool with a native BigQuery connector. Supports direct query and scheduled refresh. Setup guide →

Free Google BI tool with native BigQuery support. One-click connection to your Kaivo warehouse; great for SMB teams on Google Workspace. Setup guide →

The premium analytics standard, with native BigQuery integration. Setup guide →

Use Connected Sheets to query BigQuery directly from a spreadsheet, with no SQL. Setup guide →

Connect via Power Query’s BigQuery connector. Setup guide →

Open-source BI tool with strong BigQuery support. Setup guide →

See our pricing page for PyPI connector pricing and plan details.

  • Adform: Sync Adform to BigQuery.
  • Amplitude: Sync Amplitude to BigQuery.
  • Auth0: Sync Auth0 to BigQuery.
  • Convex: Sync Convex to BigQuery.
  • GitHub: Sync GitHub to BigQuery.
  • GitLab: Sync GitLab to BigQuery.