Amazon S3 connector
Set up the Amazon S3 connector in Kaivo: authentication, configuration, the BigQuery tables it syncs, and answers to common questions.
Amazon S3 connector pricing, tier and credits on kaivo.io
Kaivo is a fully managed data platform that syncs your Amazon S3 data into a Google BigQuery warehouse and keeps it up to date automatically. There is no pipeline to build and no infrastructure to run, so you can spend your time analysing your data from Amazon S3 instead of moving it.
What is the Amazon S3 connector
Section titled “What is the Amazon S3 connector”Sync CSV, JSON, and Excel files from your Amazon S3 buckets into BigQuery with Kaivo, with no manual uploads or custom scripts to maintain.
| Category | Files & Databases, Tech |
|---|---|
| Authentication | API key |
| Setup | Self-service |
Getting started with the Amazon S3 connector
Section titled “Getting started with the Amazon S3 connector”- Sign up for Kaivo and create a workspace.
- Connect your Amazon S3 account.
- Choose which tables to sync.
- Wait for the initial sync to finish.
- Query your data in BigQuery or your favourite AI or BI tool.
Authenticating Amazon S3
Section titled “Authenticating Amazon S3”Authenticate with your Amazon S3 credentials. You provide:
| Field | Description |
|---|---|
| Bucket | |
| The name of the bucket to read from. |
| | Region |
The region the bucket is in, such as eu-north-1.
| | Access Key ID |
The access key of an IAM user that is allowed to s3:ListBucket on the bucket and s3:GetObject on its contents.
| | Secret Access Key |
The secret access key belonging to the same IAM user.
|
Configuring the Amazon S3 connector
Section titled “Configuring the Amazon S3 connector”When you set up the connector, you provide:
| Field | Description |
|---|---|
| Name | |
| A name to identify the tables parsed from the file(s). |
| | Paths |
One or more paths inside the bucket, such as exports/*.csv. Use ** to match across directories. Multiple files are combined together and should follow the same schema.
| | Delimiter |
A one-character string used to separate fields. If no value is given, auto-detection is used instead. An explicit value is recommended for reliability.
| | Quote Character |
A one-character string used to quote fields containing special characters.
| | Escape Character |
A one-character string used to remove any special meaning from the following character.
| | Text Encoding |
File character encoding, such as utf-8 or latin-1. Defaults to utf-8-sig, which allows an optional BOM (byte order marker).
| | JSON Path Prefix |
A path inside the JSON structure, in which the data items can be found. Path components must be separated by periods. Use “item” to indicate an array item.
| | Multiple JSON Values |
Allow multiple top-level JSON values. Required for NDJSON (Newline-Delimited JSON) or JSONL (JSON Lines) files.
| | Comments |
Allow C-style // ... and /* ... */ comments. Required for JSONC (JSON with Comments) files.
|
Tables and columns synced from Amazon S3
Section titled “Tables and columns synced from Amazon S3”The available streams and columns mirror the tables and fields in your own database, so they are determined when the connection runs rather than listed here.
How the Amazon S3 sync works
Section titled “How the Amazon S3 sync works”After the first load, Kaivo keeps your BigQuery warehouse up to date for you. Where Amazon S3 supports it, each sync pulls only new and changed records so it stays fast; otherwise it refreshes the whole table. Every record keeps its original ID, so you won’t get duplicate rows.
Frequently asked questions
Section titled “Frequently asked questions”How long does the initial sync take for Amazon S3?
Section titled “How long does the initial sync take for Amazon S3?”It depends on how much history is in your Amazon S3 account. Most initial syncs finish within minutes, while large accounts can take a few hours. After that, syncs only fetch new and changed records, so they’re much faster.
Can I sync only some tables or columns?
Section titled “Can I sync only some tables or columns?”Yes. You pick which tables to sync when you set up the connection and can change the selection later. Tables you don’t select are never copied to your warehouse.
What happens when Amazon S3’s schema changes?
Section titled “What happens when Amazon S3’s schema changes?”New fields are never added automatically. You choose which fields to sync, so data you haven’t selected (sensitive personal data, for example) never lands in your warehouse. When a new field appears, it becomes available for you to add. What happens to removed or renamed fields depends on a table’s sync mode: full-refresh tables always match what’s currently in Amazon S3, so dropped fields disappear, while incremental tables keep their existing columns and history, so an old field stays and newly added fields fill in over time.
How do I handle GDPR or data deletion requests?
Section titled “How do I handle GDPR or data deletion requests?”Your data lives in your own Kaivo-managed BigQuery warehouse, so the most direct option is to delete or anonymise specific records right in BigQuery. If you delete data in Amazon S3 instead, full-refresh tables drop it on the next sync, while incremental tables keep it, so you would remove the row in BigQuery or ask us to run a full refresh. To remove everything, delete the Amazon S3 connector in Kaivo and all of its synced data is deleted with it.
Common use cases for Amazon S3 data
Section titled “Common use cases for Amazon S3 data”Load flat files
Section titled “Load flat files”Turn files a team already drops in a bucket into queryable BigQuery tables on a schedule.
Recurring exports
Section titled “Recurring exports”Pick up scheduled exports from systems that have no API but can write to S3.
Combine with app data
Section titled “Combine with app data”Join file data with your CRM or finance tables in BigQuery instead of in a spreadsheet.
Use Amazon S3 data in your AI and BI tools
Section titled “Use Amazon S3 data in your AI and BI tools”Once Amazon S3 data lands in your Kaivo-managed BigQuery warehouse, you can explore it with AI tools or any BI tool that connects to BigQuery. Here’s how the most common destinations work with Amazon S3 data.
Claude
Section titled “Claude”Use Kaivo’s MCP server to give Claude secure, workspace-scoped access to your data. Setup guide →
Power BI
Section titled “Power BI”Microsoft’s BI tool with a native BigQuery connector. Supports direct query and scheduled refresh. Setup guide →
Data Studio
Section titled “Data Studio”Free Google BI tool with native BigQuery support. One-click connection to your Kaivo warehouse; great for SMB teams on Google Workspace. Setup guide →
Tableau
Section titled “Tableau”The premium analytics standard, with native BigQuery integration. Setup guide →
Google Sheets
Section titled “Google Sheets”Use Connected Sheets to query BigQuery directly from a spreadsheet, with no SQL. Setup guide →
Connect via Power Query’s BigQuery connector. Setup guide →
Metabase
Section titled “Metabase”Open-source BI tool with strong BigQuery support. Setup guide →
See our pricing page for Amazon S3 connector pricing and plan details.