Connect your data

A source is somewhere your workspace reads rows from. It can be a Google Sheet, a table in Databricks or Snowflake, a Postgres database, or a dataset saved in this workspace. Once attached, all of them work the same way — enrichment jobs, backtests and evaluation fixtures all read through a source.

What you can connect

Source	Read from it	Write results back	Use in a backtest
Google Sheet	yes	yes	after capturing it
Postgres table	yes	yes	after capturing it
Databricks table	yes	no	after capturing it
Snowflake table	yes	no	after capturing it
Saved dataset	yes	no	yes

Warehouse tables are read-only: jobs can read from them, but results are written somewhere you choose — usually a sheet. If your organization does not offer a source type, it will not appear.

Connect your credentials

Workspace settings → Integrations → Data connections.

Your connection is yours. Each member connects their own, and a teammate’s will not work for you — if someone else has already connected the warehouse you will see that it is connected, but you still need your own to attach a table.

Fill in the card and press Connect. We test the credentials against the real service before saving anything, so “Connected” means it actually worked. If it did not, you will be told whether the credentials were rejected, the host could not be reached, or it timed out.

Credentials are encrypted and never shown again — not even to you. To change one, connect again.

Test connection re-checks a saved credential. Use it if jobs have started failing; a credential that has expired or been rotated will show as needing to be reconnected.

Disconnect erases the stored credential and pauses every job that was reading through it. Nothing is deleted — your sources keep their setup and their history, and reconnecting starts them up again. The card tells you how many sources are affected before you do it.

If the person, or the dataset, behind a source goes away

Disconnecting is reversible. Two things are not:

• A teammate’s user account is deleted. Their credentials go with it, so the sources they attached can no longer be read by anyone.
• A dataset in the library is deleted. Any source that wrapped it has nothing left to read.

In both cases the affected sources are detached and their jobs pause, with the reason shown on the job — “connection deleted” or “dataset deleted”. Your setup and your enrichment history are kept, but reconnecting will not help this time: there is nothing left to reconnect to. Attach the source again — through a live connection, or to another dataset. A job that wrote to the old source can be pointed at the new one and switched back on. A job that read from it cannot be switched back on: create a new job over the new source. Its history stays with the old one.

A workspace where this happened is safe to leave alone: paused jobs cost nothing and keep retrying nothing.

Attach a source

Jobs → Sources → Attach, or from a batch frame in the canvas.

Choose the kind of source, then fill in the one step it needs:

• Google Sheet — paste the sheet’s link, choose a tab, and map the columns.
• Saved dataset — choose one from this workspace’s library. Nothing else to fill in.
• Databricks or Snowflake — give the catalog (Databricks) or database (Snowflake), the schema and the table, then press Preview table to check you have the right one before mapping columns.
• Postgres table — give the table (and the schema, if it is not public), press Preview table, then map columns. Postgres is the one SQL source that can also be written to: tick Write back on any column enrichment should fill in.

A source type you have not connected yet appears greyed out, with a note telling you to connect it first.

Choosing a key column

Every source needs one column that identifies a row — this is how the product knows which row it has already handled. Choose something that does not change: an ID rather than a name someone might edit.

Changed-at column (warehouse tables)

If your table has a “last updated” timestamp, name it. Without it, every row is treated as needing work on every run. On a large table, that matters.

Saved datasets

A dataset is a fixed set of rows kept in this workspace. Datasets are what backtests and evaluations run against, because they do not change underneath a run.

Find them under Jobs → Datasets. There are three ways to make one.

Paste rows

Straight into the form. Good for a short list.

Import a CSV or JSON file

Jobs → Datasets → Import file.

• Tell us which column is the key if it is not called key.
• If your file records when each row became true — an announcement date, a closing price’s date — name that column. That turns the file into real history you can backtest over.
• If you leave it out, every row is marked as known from the moment you uploaded it.

A file can hold up to 50,000 rows and 4 MB. Over either limit, the import is refused and tells you — nothing is quietly cut off.

Capture from a source

Jobs → Datasets → Capture from a source.

This reads a source you have already attached and saves the result to the library. It is how a live table becomes something you can backtest against.

One thing to understand about capture:

A capture is a snapshot from now on — not a reconstruction of the past.

Every row is recorded as known at the moment you captured it. So a single capture used in a backtest over a past date range will find nothing at all — correctly, because none of that data was knowable back then.

To build something you can backtest over, choose Add to an existing dataset instead of creating a new one. Each capture adds only the rows whose value actually changed, stamped with the time you captured them. Capture the same source again next week and next month, and the dataset becomes a real history: each date in a backtest sees the value that was current then.

To keep doing this without remembering to, tick Repeat on a schedule when you capture, and choose how often: every hour, every 6 hours, every day or every week. Times are in UTC. Captures can run at most once an hour, because each one reads the whole source. The schedule starts once the first capture succeeds.

If your data already records when each row became true (a CSV imported with a known-at column, for example), capture keeps those dates rather than replacing them, and it is backtestable straight away.

The read happens in the background, so the dataset appears in the list a moment later rather than immediately. The same size limits apply, and a source too large to store is refused rather than partly saved.

Capturing on a schedule

A dataset that captures on a schedule shows it under its name in Jobs → Datasets, for example “Captures every day at 06:00 UTC · next 30 Oct, 06:00”. From there you can Pause, Resume or Stop repeating. Stopping keeps the dataset and everything already captured. A dataset can have one schedule. You can still capture into it by hand at any time.

A scheduled capture skips a turn, and tries again at the next one, when:

• a backtest, a study or an evaluation that uses this dataset is still running. Adding rows while one runs would make it stop, because it checks that its data has not changed underneath it.
• your workspace’s subscription has lapsed, or it has reached its usage limit. Captures pick up again by themselves once that is resolved.

A schedule pauses itself, and says why, when carrying on would only fail again:

• Paused — dataset is full. The dataset reached the 50,000-row or 4 MB limit. New captures are refused rather than deleting older history, because that would change what a backtest over an earlier period sees. Remove records, or capture into a new dataset, then resume.
• Paused — source detached, source access revoked, connection deleted or source cannot be read. Reconnect or re-attach the source, then resume. Resuming is refused while the source is still unavailable.

Where a dataset came from

Every dataset remembers how it was made. Under its name in Jobs → Datasets you will see one of:

• Captured from Name — it was captured from that source. Adding more captures to it keeps the same label.
• Imported from a file
• Pasted

A dataset made before this label existed shows nothing there — we do not guess. Detaching or deleting a source never deletes the datasets captured from it; they keep their rows and their label.

The other way round, each source in Jobs → Sources says how many datasets have been captured from it.

If something is not working

“Connect this in Integrations first” — you do not have your own credential for that source type yet. A teammate’s does not count.

“This credential stopped working” — the saved credential no longer works. It may have been rotated or expired. Connect again.

Your jobs are paused — the credential they read through was disconnected, or the source was detached. Reconnect or re-attach, then switch the job back on. Nothing was lost. Scheduled captures that read from the same source pause the same way. Resume them from Jobs → Datasets.

A job says “connection deleted” or “dataset deleted” — what it read through was deleted, not just disconnected, so there is nothing to reconnect. Attach the source again — to a live connection or another dataset. If the job wrote to the deleted source, point it at the new one and switch it on. If it read from it, create a new job over the new source; switching the old one on is refused, on purpose, because it would only pause again on its next run.

“Invalid identifier” — a catalog, schema, table or column name contains characters we will not put into a query. Use plain names, without quotes, semicolons or dots inside a single field.

“Already attached” — this workspace already has an active source pointing at exactly this sheet, table or dataset.

A job failed saying it could not write — its target is a read-only source, such as a warehouse table or a dataset. Point the job at a sheet instead.

What this costs

Reading your own database or warehouse uses your compute, billed by that provider — we do not add a charge for it. Enrichment that calls a model is billed as usual; see Usage and billing.