Commit Graph

43 Commits

Author SHA1 Message Date
Riccardo Busetti df7a0ca3f7 feat(replication): Evaluate new product name (#47066) 2026-06-24 16:36:23 +02:00
Coenen Benjamin 2ce01473bb feat(replication): add better form to create ducklake replication (#47069)
## What kind of change does this PR introduce?

Add support to use supabase projects as a pg catalog and storage when
adding a ducklake replication.

## What is the current behavior?

Only simple form with raw input text for custom parameters is available.

## What is the new behavior?

Being able to select supabase project to directly use projects in
supabase for the ducklake.
I also fixed a warning we had in the console for this form (cf
screenshot)

## Additional context

[API Changes ](https://github.com/supabase/platform/pull/34282)


https://github.com/user-attachments/assets/4ff9ee65-6ba4-4f17-9ea1-9aebad34171c

<img width="862" height="228" alt="Capture d’écran 2026-06-18 à 09 58
50"
src="https://github.com/user-attachments/assets/1592c3be-807e-426f-9a5a-84979e05d93c"
/>

### Test scenario
Follow the screencast, go to your supabase project (better if it's in
ap-southeast-1)
Create a test table with 1 row for example
-> Database -> Replication -> New destination -> Select ducklake and use
supabase option
-> Keep the same current supabase project selected for both catalog and
storage
-> Create destination -> You'll get a warning about the storage and
credentials
-> Confirm creation
-> Wait until it's in status Running, if it's runing then it works


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added DuckLake replication destination with **Use Supabase** and
**Custom parameters** modes.
* Added DuckLake bucket selection with a **“New bucket”** creation
dialog.
* Added/expanded BigQuery, Analytics Bucket, and Snowflake destination
configuration.

* **Improvements**
* Updated DuckLake create vs edit behavior: mode selection is hidden in
edit mode and configuration is mapped correctly for the selected
variant.
* Enhanced field-level validation (including whitespace-only handling)
and added clearer validation issue messages.
* Added a cross-region warning for DuckLake when catalog and storage
regions differ.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-06-19 16:07:20 +02:00
Joshen Lim ea909e998b Local only: skip loading destinations if ETL API is not set up (#47031)
## Context

> [!IMPORTANT]  
> Changes in this PR only apply to the local environment - there should
not be any changes to staging (nor production)

Given that read replicas currently sit under database replication, the
UI currently waits for replication destinations to load before rendering
the page. However for local development, setting up of the ETL API isn't
necessary nor applicable for everyone so this indirectly adds friction
if we just want to work with read replicas.

## Changes involved
- Opting to skip retrying fetching ETL related requests if the error
returned is "replication API URL is not configured"
  - This is indicative that the local platform isn't set up for ETL yet
- ^ Database replication page will hence not wait for ETL requests to
succeed before finally rendering the UI
  - Node diagram will also then render properly (just read replicas)
- Add a small admonition to visualize this
<img width="1079" height="301" alt="image"
src="https://github.com/user-attachments/assets/32bd5d2f-a76e-417e-bedf-9a04de3bb305"
/>

## To test
- Will only be able to test locally - basically just head over to the
database replication page (unless you somehow already have ETL API set
up locally)
- But can also verify that there's no changes on staging preview


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved error messaging when ETL is not configured in local
development environments
* Enhanced error handling for replication API failures with better
non-retryable error detection

* **Improvements**
* Refined replication diagram rendering based on destination setup state
  * Updated dropdown menu interactions for read replica management

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-19 17:27:44 +08:00
Riccardo Busetti cb6c361f25 feat(replication): Improve replication validation and fix scroll (#46943) 2026-06-16 14:38:25 +00:00
Joshen Lim 1baaded0bb Consolidate execute-sql-query into execute-sql-mutation (#46944)
## Context

Just some clean up as I was going through stuff
- `useExecuteSqlQuery` is deprecated and not used at all
- As such `execute-sql-query` is technically irrelevant, the more
relevant file is `execute-sql-mutation`
- Hence opting to consolidate `execute-sql-query` into
`execute-sql-mutation`
- Also removing `ExecuteSqlError` since its just re-exporting the
`ResponseError` type

There's a lot of file changes but its essentially just updating the
importing statements across the files
2026-06-16 00:07:16 +08:00
Victor Farazdagi 61754a0eec feat(studio): add Snowflake replication destination (#46767)
## Details of change

Adds Snowflake to the Studio replication destination flow:
- destination selection and display
- create/edit form fields
- validate/create/update payload serialization
- generated Platform API types

Snowflake remains gated behind `etlEnableSnowflakePrivateAlpha`.

**Note:** I have configured `etlEnableSnowflakePrivateAlpha` in
ConfigCat ("all" in staging and tied to my own org id in prod).

## Details of Verification Process

- Studio focused Vitest coverage for form serialization and diagram
mapping
- Studio typecheck
- ESLint on changed Studio replication files
- Local `mise fullstack:dev` smoke test to confirm the Snowflake form
renders ok.

<img width="937" height="569" alt="image"
src="https://github.com/user-attachments/assets/8d6b3a87-1f9d-4a59-91da-be719714ea49"
/>


Full create/validate E2E depends on the Platform PR and ETL runtime
rollout.


## Review Requests

Please check the Snowflake wire payload matches the Platform/ETL
contract and that gating/edit/display behavior follows the existing ETL
destination patterns.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Snowflake added as a supported replication destination (private-alpha
gated), including UI for selecting and configuring connection and auth
(account, user, database, schema, role, private key, optional
passphrase).
* **Validation**
* Form validation and submission now handle Snowflake-specific
required/optional fields.
* **Tests**
* Unit tests added for Snowflake form behavior and replication-type
detection.
* **API**
* Destination create/update/validate flows extended to accept Snowflake
payloads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-11 10:25:57 +03:00
Coenen Benjamin 71775c368e refactor(replication): remove useless expire snapshot configuration (#46341)
Remove useless expire snapshot configuration

> This is an alpha feature not already deployed to any of our customers
so breaking changes are not an issue.


Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
2026-05-25 17:20:56 +02:00
Coenen Benjamin 151a1792d9 feat(studio): add support for new ducklake destination in replication UI (#45370) 2026-04-29 21:42:05 +02:00
Han Qiao 67deabf67e fix: create etl publication as postgres (#45043)
## What kind of change does this PR introduce?

Bug fix

## What is the current behavior?

Creating a schema only branch fails because ETL publication is owned by
`supabase_etl_admin` which users have no access.

## What is the new behavior?

Since ETL supports user managed publications, create them through pgmeta
so it's owned by `postgres` role instead.

## Additional context

mirrors [upstream
etl](https://github.com/supabase/etl/blob/main/etl-api/src/db/publications.rs#L22-L51)
implementation


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Added guards to prevent creating publications when project or
connection info is missing, with clearer error logging.
* Ensure the project connection string is explicitly passed so
publications target the correct database.

* **Refactor**
* Publication creation now executes generated SQL directly against the
database, with correct handling of empty or selected table lists and
proper identifier quoting for reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
2026-04-27 19:45:26 +08:00
Joshen Lim 7f5865872a Enforce noUnusedLocals and noUnusedParameters in tsconfig.json + fix all related issues (#45264)
## Context

Enforce `noUnusedLocals` and `noUnusedParameters` in tsconfig.json + fix
all related issues
2026-04-27 17:42:34 +08:00
Riccardo Busetti 8347877957 feat(etl): Add UI elements to disable external replication (#45035) 2026-04-24 08:17:58 +02:00
Riccardo Busetti a23e673993 ref(etl): Improve UX around table resets (#44843) 2026-04-14 16:21:36 +02:00
Charis 3b7052b5a9 cleanup: fix import order and prefixes for studio/data (#44501) 2026-04-03 09:15:57 +02:00
Riccardo Busetti d41ab5b4ff ref(etl): Update configuration and add new parameters (#43126) 2026-03-02 16:01:44 +01:00
Riccardo Busetti 67ed8f18ee feat(etl): Add new setting for controlling table copy (#42853) 2026-02-24 08:14:23 +01:00
Joshen Lim 6ffc3fc94c chore(studio): Replication UI updates for new replica CTA + URL query state handling (#42378)
## Context

Related to unifying read replicas into database replication page which
is currently in internal testing

## Changes involved

- Updates the "New replica" CTA in `DatabaseSelector` (e.g within the
SQL Editor) to link to the replication page instead of settings
infrastructure
- Refactor replication UI to use URL query states for new destination +
edit destination

## To test

- [ ] Verify that the "New replica" CTA links correctly to replication
page if flag is on
- [ ] Verify that new + edit destination UI works as expected
- Note that there's some server side validation for ETL replication now
so might be tricky to test creating an actual ETL replication
- Minimally can test creating a read replica and ensure that the UI
behaviour is all as expected

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added unified replication experience with URL-based state management
for destination creation and editing

* **Refactor**
* Simplified replication panel component interfaces and consolidated
destination data fetching logic
* Enhanced edit flows to leverage URL parameters for seamless navigation

* **Chores**
  * Marked legacy read replica creation panel as deprecated
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-03 15:54:21 +08:00
Joshen Lim 1536ee1064 Joshen/fe 2287 summary nodegraph in replication page (#42042)
* init

* Init 2

* Finalize

* bit more details in the summary graph

* Moreeee details

* Clean

* 🧹

* FIIIXX

* Fix rabbit

* Simplify long polling logic to only depend on database for destinations UI

* ADdress rabbit

* Address feedback
2026-01-23 17:59:52 +08:00
Riccardo Busetti 1238a19fe4 feat(etl): Expose more configuration parameters (#41758)
* feat(etl): Expose more configuration parameters

* Nit nudges

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-01-07 15:36:32 +08:00
Riccardo Busetti d795240928 feat(etl): Add integration with new validation endpoint (#41731)
* feat(etl): Add integration with new validation endpoint

* Fix

* Fix

* Fix

* Fix

* UI nudges + clean up + refactors

* Fix TS

* Sort issues

* Small improvement

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-01-07 00:32:08 +08:00
Riccardo Busetti c62cd07b86 feat(etl): Implement batch rollback (#41552) 2026-01-06 04:13:25 +01:00
Joshen Lim 31d2e8ac5b Shift ETL destination components into a separate folder (#41457)
* Shift ETL destination components into a separate folder

* Small fix
2025-12-18 18:55:07 +08:00
Ivan Vasilov 0d5be306ef chore: Bump React Query to v5 (#40174)
* Bump the deps, refactor deprecated code.

* Migrate keepPreviousData usage.

* Migrate all uses of InfiniteQuery.

* Fix refetchInterval in queries.

* Migrate all use of isLoading to isPending in mutations.

* Fix accessing location in claim-project.

* Fix a bug in duplicate query keys.

* Migrate all queries to use isPending.

* Revert "Fix accessing location in claim-project."

This reverts commit 2a07df64b5.

* Revert the rss.xml file to master.
2025-12-10 10:10:29 +01:00
Riccardo Busetti 1a3ba9f15a ref(etl): Rename ETL Replication to Replication and update docs (#40769)
* ref(etl): Rename ETL Replication to Replication and update docs

* Fix

* Fix

* Fix

* Fix

* Fix
2025-11-25 20:45:50 +08:00
Riccardo Busetti 91464b8fe8 ref(etl): Rename Replication to ETL Replication (#40202)
* ref(etl): Rename Replication to ETL Replication

* Fix

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-11-07 12:02:40 +08:00
Ivan Vasilov 4da5d019ae chore: Migrate some stray queryKeys to the RQ 5 syntax (#40086)
Fix some stray queryKeys.
2025-11-04 09:33:49 +01:00
Ivan Vasilov 8b657165b5 chore: Migrate to use custom type for ReactQuery queries and mutations (#40073)
* Add custom types for queries, mutations and infinite queries.

* Migrate all queries to use the new type.

* Migrate all infinite queries to useCustomInfiniteQueryOptions.

* Migrate all mutations to use useCustomMutationOptions.

* Add type to all imports in `types` folder.
2025-11-03 13:18:13 +01:00
Joshen Lim 64e3e047eb Final final cleaning up barrel files (#40018)
* Final final cleaning up barrel files

* Fix merge conflict
2025-10-31 14:02:59 +08:00
Joshen Lim 0910908561 Swap analytics buckets endpoint (#39610)
* Swap analytics buckets endpoint

* Fix type issues

* fix

* Update API types

* Fix tests

* nit

* Final fixes

* Fix

* chore(studio): analytics buckets contents (#39701)

* first commit

* improvements

* misc

* cleanup

* remove tappable whole

* design updates

* analytics improvements

* polish

* add clashed case

* fix pairing status

* connect tables basics

* add multiselect

* restore prod version for non-feature preview users

* fix padding inconsistency

* terminology fix

* better empty state

* fix terminology

* block table connection dialog button

* Nit refactors

* minor nits

* Add comment

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>

* Default circumstance to fresh in ImportForeignSchemaDialog

* Hook up connecting tables for analytics buckets (#39906)

* Hook up connecting tables for analytics buckets

* Address comments

* Clean up iceberg wrapper when deleting analytics bucket (#39902)

* Clean up iceberg wrapper when deleting analytics bucket

* Clean up s3 access key when deleting analytics bucket

* Make connect table disabled for now

* Most clean up and refactoring

* Refactor RQ

* Nit

* Refactor

* nit

---------

Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
2025-10-29 19:10:10 +08:00
Ivan Vasilov da4a40e308 chore: Migrate RQ functions to use object syntax style (#39895)
* Migrate all uses of invalidateQueries to use object syntax.

* Migrate the remainder of useInfiniteQuery.

* Migrate all setQueriesData.

* Migrate all fetchQuery uses.

* Migrate some leftover functions from RQ.

* Fix issues found by Charis.
2025-10-28 10:43:14 +01:00
Alaister Young 8855d05803 chore(studio): swap react-query to object syntax (#39842)
* chore(studio): swap react-query to object syntax

* Fix small issues found

* Fix realtime settings

* Nit

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-10-27 09:38:27 +01:00
Raminder Singh a21fa43d8f feat: add iceberg (analytics bucket) destination in UI (#39324) 2025-10-16 06:56:11 +05:30
Riccardo Busetti f07c9cad3a ref(replication): Add alpha banner and remove batch size (#38588) 2025-09-11 15:25:39 +08:00
Riccardo Busetti 48d76a1c6b feat(replication): Add UI for updating pipeline versions (#38473) 2025-09-10 11:26:37 +02:00
Riccardo Busetti 5e4cc212aa feat(replication): Add ability to copy pipeline id and filter logs by pipeline id (#38407) 2025-09-04 15:47:01 +02:00
Riccardo Busetti 77356bf946 feat(replication): Significantly improve the replication UI behavior (#38237) 2025-08-29 12:20:51 +02:00
Riccardo Busetti e9424b98de feat(replication): Delete destination and pipeline atomically (#37790) 2025-08-11 08:23:15 +02:00
Riccardo Busetti 1ea8c656db feat(replication): Add new UI for new table state handling (#37662)
* feat(replication): Add new UI for new table state handling

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Improve

* Fix

* Fix

* Fix TS

* Some UI tweaks

* Tweaaakk

* Fix semantics

* Small optimization

* Some adjustments

* Bit more fixes

* Small UI tweak to simplify UI

* Fixy fix

* gahhh tiny tiny tiny

* Last bit of cleanup

* I swear one final tweak

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-08-07 12:16:03 +02:00
Joshen Lim 5da5ad06c0 Shift pipeline status page to its own route (#37431)
* Shift pipeline status page to its own route

* Fix back link
2025-07-29 16:33:14 +08:00
Riccardo Busetti 0ecddf1901 feat(replication): Implement UI to render pipeline replication status (#37376) 2025-07-24 13:04:07 +02:00
Riccardo Busetti a77d40e43b feat(replication): Update the API calls for replication endpoints (#36866) 2025-07-08 16:34:03 +02:00
Raminder Singh b411ed3421 chore: rename sink to destination for the etl api (#35931)
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
2025-05-26 21:30:10 +05:30
Raminder Singh 520428680d feat: call the new api endpoints to handle partial failures in pg_replicate (#35418)
* feat: use the new tenant-source api to atomically create the tenant and source in pg_replicate

* feat: use the new sink-pipeline api to atomically create the sink and pipeline in pg_replicate

* fix: remove some unused imports

* feat: use the new sink-pipeline api to atomically update the sink and pipeline in pg_replicate

* feat: deleting the sink cascades to delete the pipeline

* chore: update api types

* fix: revert accidental update to types

* remove unused code

* add a comment explaining why deleting only sink is enough
2025-05-10 12:04:37 +05:30
Raminder Singh 23ceb9072d Add replication UI in Studio (#30090)
* add dummy sinks and pipelines pages

* update api types

* show empty sources state

* show empty replication state at /replication

* create source when enable replication button is clicked

* improve replication page when replication is enabled

* replace sources page with publications page

* publications table

* show publications in table

* create publication wip

* show toast error instead of throwing an exception

* user can now delete a publication

* show empty sinks page

* create and list sinks

* add ui to delete a sink

* show pipelines on the pipelines page

* add ui to create and delete pipelines

* get pipeline status wip

* show pipeline status wip

* show correct label on action buttons

* start and stop pipelines

* remove a couple of console.logs

* fix error when deleting a pipeline

* only consider replication enabled when a source with name = ref is present

* add source and sink names

* correct colspan for 'no pipelines' row

* hide 'supabase_realtime' publication on ui

* move filtering to fetch query

* show sink name in ui

* show source/sink names on pipelines page

* fix start/stop status shown on ui

* fix prettier formatting

* update api types

* extract pipeline action button as a separate component

* fix a crashing page

* fixed publications page crash

* update to match with changes in api

* add new replication page under database

* hide replication page behind feature flag

* update types

* update api types

* show destinations empty state

* add destinations table

* factor out components from Destinations table

* show status dot

* move pipeline fetch query to parent component

* add ability to enable/disable a pipeline

* show loader when starting or stopping a pipeline

* fix a bug in which loading & empty states were shown together

* fix a bug in which error & empty states were shown together

* wrap in default layout

* add new destination panel

* add type field

* fix a forwardRef error

* fix layout

* create destination

* delete destination

* add ability to create or delete destinations with pipelines

* create source if missing

* show only a single error

* add an enable switch

* new layout

* add subsections

* comment out unused code

* show enabled switch only in the header

* close panel when destination is created

* disable buttons when api requests in flight

* reduce panel size

* remove commented out code

* treat max size and max fill secs as numbers

* use drop down to show publications

* simpler vertical layout

* add separators

* add form validation

* remove publications drop down padding

* add new publication button

* hide advanced settings behind an accordion

* add some margin between icon and text

* show publications panel on clicking new publication button

* add header to new publication panel

* fix validation not running for publication drop down

* create publication in the new publication panel

* add table selector in new publication panel

* update api types

* remove old code

* update platform.d.ts

* update navigation bar utils

* remove a redirect from replication page to publications page

* ask user for confirmation before deleting destination

* edit destination panel

* edit destination panel values fixed

* bug fixes

* fix prettier formatting

* enable/disable pipeline after editing

* rename snake_case params to camelCase

* loading button when editing

* remove merge markers

* update api types

* add max_staleness parameter in sinks for bigquery

* add read replicas flow diagram to replication page

* remove an unused import

* Revert "add read replicas flow diagram to replication page"

This reverts commit 8852d7847b457885603dba786141a8aaf8e99350.

* add panel to warn users about additional cost before creating a destination

* hide replication page contents behind a feature flag

* fix merge conflicts

* styling changes

* revert static flag

* styling updates

* fixes

* fix switch

* copy

* fix layout

---------

Co-authored-by: Saxon Fletcher <saxonafletcher@gmail.com>
2025-04-23 10:52:46 +05:30