Flowsight configuration for Convex

The Convex connector is developed by DataMetis — a set of custom NiFi processors, packaged as a NAR, that runs on your Openflow runtime. Flowsight drives it for you: you set it up through the four-step wizard and watch it from the monitoring dashboard, without ever touching the NiFi canvas.

Flowsight replicates documents from the tables of a Convex deployment into Snowflake. It performs an initial bulk snapshot load of each table's current contents, then keeps the destination current through continuous incremental sync — change data capture of inserts, updates, and deletes.

What you can do with it

• Full replication — mirror an entire deployment's tables into a Snowflake schema and keep them continuously up to date.
• Selective replication — replicate a chosen subset of tables rather than the whole deployment.
• Analytics over operational data — land Convex application data in Snowflake to query, join, and model alongside other warehouse data, without affecting the source deployment.
• Type-faithful landing — preserve Convex's value-type distinctions (int64 vs. float64, bytes vs. string) rather than flattening them to generic JSON.

How a table gets replicated

Each selected table moves through three phases automatically:

1. Schema introspection — Flowsight samples the source table, infers a column schema from the observed documents, and creates the destination schema and table in Snowflake with idempotent CREATE … IF NOT EXISTS DDL.
2. Snapshot load — the current contents are paged through Convex's streaming export endpoint and applied to the target table.
3. Incremental sync — once the snapshot completes, row-level changes are polled continuously and merged in. The cursor handed off from the snapshot guarantees no gap between the two phases.

Set it up

The wizard walks you through four steps — pick Set up on the Convex card to begin:

1. Prerequisites — confirm your runtime and a Convex deploy key.
2. Select runtime — bind the Openflow runtime that will run the connector.
3. Connect endpoints — enter your Convex deployment URL and deploy key.
4. Select tables — choose which tables to replicate.

When you finish, the sync starts and the Monitoring tab lights up.

Before you start: you need an Openflow runtime (Small or larger), a Convex deploy key with permission to read its tables, and a Snowflake database, schema, and warehouse for the load.

How your data lands

Replicated tables land in <database>.<schema> and carry Convex's system columns plus two columns Flowsight manages.

Data type mapping

System columns

Flowsight-managed columns

Query the replicated data

Deletes are soft deletes — the row stays and _SNOWFLAKE_DELETED is set to TRUE. Filter them out to get the live set:

SELECT *
FROM <database>.<schema>.<table>
WHERE _SNOWFLAKE_DELETED = FALSE;

Because rows merge on _id with last-write-wins on _ts, each _id appears once and reflects the most recent change observed.

Schema changes are absorbed

While replication runs, Flowsight handles source schema changes within these rules:

• New fields appear as new nullable columns on the next load that observes them.
• Widening types are applied; a column that must hold mixed types is rebuilt as VARIANT.
• Narrowing type changes fail rather than silently losing fidelity. The affected table pauses; reconcile the source type or the destination column, then it resumes.

Adding and removing tables

To add a table, create it in Convex (or include it in your selection): the next discovery cycle registers it and runs its snapshot. To stop replicating a table, remove it from the selection. Removing a table stops further updates to its destination table — it does not drop the table or its data from Snowflake.

Controlling sync frequency

Incremental sync cycles run sequentially and cannot overlap. If a cycle takes longer than the configured interval, the in-progress cycle finishes before the next one starts, rather than running concurrent cycles.

Under the hood

Flowsight installs the Convex connector as a NiFi Archive (NAR) and configures it for you — you never wire the canvas. For reference, this is how it moves data:

• Journal → stream → merge. Change records land in an insert-only journal table (<table>_JOURNAL) through Snowpipe Streaming — row-level ingestion, with no internal stage, PUT, or COPY INTO. A scheduled MERGE then reads an append-only Snowflake Stream (<table>_STREAM) and reconciles records into the structured target table, keyed on _id with last-write-wins on _ts. Deletes arrive as tombstone rows with _SNOWFLAKE_DELETED = TRUE.
• Durable resume. The resume cursor is persisted to a _CDC_SYNC_STATE control table, so a runtime that loses its NiFi state recovers without re-snapshotting every table.
• Cadence and cleanup. The MERGE runs on a timer only while the runtime is up — a paused connector incurs no Snowflake compute — and a retention sweep reclaims journal rows older than 48 hours.
• Authentication. Inside a Snowflake-hosted runtime the connector uses the runtime's own session-token identity; the Convex deploy key is sent as an Authorization: Convex <key> header.

Limitations

• Root-component tables only — Flowsight replicates tables in the deployment's root component.
• Schema is inferred by sampling — fields that never appear in the sampled documents aren't represented until they're observed.
• Sequential incremental cycles — cycles run one at a time; a long cycle finishes before the next starts rather than overlapping.
• Single global delta cursor — incremental changes for all tables are consumed through one ordered stream.
• No view replication — Convex has no views; only base tables are replicated.

Troubleshooting