Commit Graph

3186 Commits

Author SHA1 Message Date
Ingvar Stepanyan 1b9e35129c Move C# bindings into the tree (#324)
This allows to make and test ABI-breaking changes much more easily than before.

For now moved them under crates/bindings-csharp - on one hand, it's not a crate, but on another it's the most logical place within our current project structure if we think of `crates` folder as general `projects` folder.
2023-09-26 15:33:30 +00:00
Mario Montoya 9aecc256de Adding multi-column indexes to the DB (#243)
* Initial draft for multi-column indexes

* Fix index seek for multi-columns

* Restore use of ColId

* Fix compilation

* Merge

* Addressing some PR comments

* Clippy

* Clarify usage of AlgebraicValue for indexes

* .
2023-09-25 16:32:29 +00:00
joshua-spacetime f2d038d8b2 perf(308): Push index scans below join (#313)
* perf(308): Push index scans below join

Fixes #308.

* changes from review
2023-09-22 10:32:48 -05:00
joshua-spacetime cf2584fe80 feat(297): generate index scans for range predicates (#306)
Fixes #297.
2023-09-21 11:49:20 -05:00
dbrinkmanncw 9a4e8f1d73 TypescriptSDK quickstart example (#15)
* Initial commiut

* Update quickstart example

* Cleanup

* Update quickstart to use identity changes

* Remove misplaced files

* Switch generated folder name to match quickstart guide

* Switch generated folder name to match quickstart guide

---------

Co-authored-by: NateTheDev1 <nthnlrichards@gmail.com>
Co-authored-by: Derek Brinkmann <dbrinkmann@citadelstudios.net>
2023-09-21 11:55:51 -04:00
Phoebe Goldman fa689ba399 [SDK] Separate client cache from connection to avoid a deadlock in cloud-next (#248)
Prior to this commit, accesses to the global most-recent client cache state in the Rust SDK
acquired the global mutex on CONNECTION.
This lead to a deadlock during bootstrapping in cloud-next.

With this commit, it is now possible to access the global most-recent client cache state
without acquiring the CONNECTION mutex.
2023-09-21 08:58:01 -04:00
joshua-spacetime bf204654ff feat(311): Make right hand side of join generic (#312)
Fixes #311.
2023-09-21 07:33:16 -05:00
Phoebe Goldman fca7c27946 Clippy: forbid println and friends (#305)
We've had recurring issues with `println` calls sneaking in
where `log` crate macros would be more appropriate.
This commit adds a Clippy warning for uses of the global I/O macros,
i.e. `print`, `println`, `eprint`, `eprintln` and `dbg`.

The lint is disabled by a more-specific `clippy.toml` in the `cli` and `sqltest` crates,
as well as using `allow` attributes in `standalone`'s `subscommands` module.

Additionally, this commit converts a handful of prints in `core/utils` to `log::info`.
2023-09-20 13:24:26 -04:00
joshua-spacetime f9c5de8132 feat(296): index range iterator (#303)
Fixes #296.

The previous btree index iterator returned entries for a single key.
This change set adds an iterator for returning a range of values.
2023-09-20 09:30:04 -07:00
Kim Altintop ea2bae4583 Cloud next (#173)
* Client API changes to accomodate new cloud architecture

The exact trait interfaces for `client-api` is TBD

* Ensure we're not blocking when accessing the filesystem

* Derive Clone for SendGridController

* Add YOLO error variant to InsertDomainResult

* Rebase

* fixup! Rebase

* Fix SpacetimeType for Address

* Temporarily disable message / frame size limits for SDK WS

* Remove get_database_instance_state from API trait

It's an internal (worker db) thing, which does not need to be satisfied
by impls.

* Update indexes when updating a database

It turns out that the order of the index definitions in the proposed
schema may differ from those returned from the catalog, causing valid
(i.e. no-op) updates to be rejected. While at it, allow updating table
indexes so as long as the (column) schema remains unchanged.

* Update indexes when updating a database

It turns out that the order of the index definitions in the proposed
schema may differ from those returned from the catalog, causing valid
(i.e. no-op) updates to be rejected. While at it, allow updating table
indexes so as long as the (column) schema remains unchanged.

* Fix -S instead of -s for update-module smoke test

-s now means "server", -S "skip clippy", changed in:

a1e9984 (Multiple server configurations for CLI (#214), 2023-09-01)

* Invalidate schema cache when committing a tx

* Use long options in update-module.sh, fix unused warning

* Add test asserting schema_for_table reflects index updates
2023-09-20 07:42:37 +02:00
Nathaniel Richards a592f0e260 Added support for previous and next key 2023-09-19 13:57:07 -04:00
Piotr Sarnacki 3d4e3849fc Allow disabling logging to disk (#304) 2023-09-19 10:27:35 +02:00
Nathaniel Richards 32b6e94d05 Added docs about images 2023-09-18 13:20:17 -04:00
Nathaniel Richards 68c9eea342 Added tag documentation 2023-09-18 13:13:57 -04:00
Nathaniel Richards b6a5510e80 Docs config generating well 2023-09-18 13:12:07 -04:00
Nathaniel Richards 6731f29499 First commit 2023-09-18 12:25:05 -04:00
joshua-spacetime 4490bea49b feat(260): Add index scans for equality predicates to query engine (#295)
Fixes #260.
2023-09-16 21:06:15 +00:00
james gilles 1bfe4c54c4 One-off query support in SpacetimeDB core. (#198) 2023-09-15 13:35:59 -04:00
Ingvar Stepanyan c47e571d2d Avoid relying on global CLI in tests (#294)
This switches tests from invoking a spacetimedb CLI found on PATH to always using up-to-date version by depending on the CLI crate as a library.

---------

Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
2023-09-14 09:15:32 -04:00
Ingvar Stepanyan 676dfbe10c Refactor Option<Vec<_>> in Config (#292) 2023-09-13 19:10:43 +00:00
Ingvar Stepanyan f98694f707 Add C# integration test (#291) 2023-09-13 15:53:07 +01:00
Ingvar Stepanyan b1c69834da Bump C# for ABI 4.0 (#282) 2023-09-13 14:30:51 +02:00
Mazdak Farrokhzad 12a7e7f572 use derive_more::From (#281) 2023-09-13 11:40:23 +00:00
joshua-spacetime 39e7298f80 perf: remove table headers from rows in query engine (#271)
Fixes #270
2023-09-13 12:54:54 +02:00
Kim Altintop 55a6386dc2 Update indexes when updating a database (#257)
* Update indexes when updating a database

It turns out that the order of the index definitions in the proposed
schema may differ from those returned from the catalog, causing valid
(i.e. no-op) updates to be rejected. While at it, allow updating table
indexes so as long as the (column) schema remains unchanged.

* Invalidate schema cache when committing a tx

* Use long options in update-module.sh, fix unused warning

* Add test asserting schema_for_table reflects index updates
2023-09-13 07:28:39 +02:00
joshua-spacetime cc36a2e87c perf: read DataKey instead of recomputing it in subscriptions (#276)
Fixes #259 

(1) Updates MemTable to use RelValue instead of ProductValue
(2) Adds a DataKey member to DataRef and RelValue
(3) Subscriptions compute DataKey only when not present on row
2023-09-12 15:16:09 -07:00
Kim Altintop 8f23b6a932 Make database init / update transactional (#279)
* Make database init / update transactional

That is, run both schema modifications as well as init/update reducers
in the same transaction.

* Add commentary
2023-09-12 19:02:50 +00:00
Phoebe Goldman 32ac808804 SDK test suites (#258)
* Simple SDK test harness; SDK test module

Yet to come: actual SDK tests.

* Quiet clippy lint about "too many arguments"

* Lints are named with underscores, not dashes...

* Will this make clippy shut up?

* Go nuclear on disabling `too_many_attributes` lint. Sigh.

* WIP SDK test client, and fix bugs in Rust codegen

Compiling the module_bindings for the sdk-test module revealed two bugs:

- Enums holding structs generated incorrectly,
  unpacking the struct into the enum's payload.

- Recursive types would cause the codegen to attempt to recursively import
  the current module into itself.

* One (1) actual runnable test in the sdk crate

* Exclude test-client from CI

The sdk tests already build this crate (though they don't clippy or fmt it).

Attempting to build, test, fmt or clippy it as-is will fail
because the module_bindings are not committed.
This is intentional, as the SDK test suite wants to generate the module_bindings
during its run.

* Rustfmt ignore generated module_bindings

It turns out `cargo fmt` doesn't actually support the `--exclude` option
the way `cargo clippy` does.

Instead, just `#[rustfmt::skip]` the `mod module_bindings;` decl.

* Actually commit test file...

God, I'm so bad at remembering to commit new files.

Anyway, add a test for deleting rows with primitive unique fields.

* Make CLI tool available in tests workflow

The SDK tests need to run `spacetime start`, `spacetime generate` and `spacetime publish`.

* Test update events with primitive pk types; split test-client into files

* Tests with `Identity` fields in tables

* Tests for reducer callbacks, both successful and failing

* Tests with vecs of stuff, with structs, with enums

* Test that should fail, test that uses a large table with many columns

* Test for resubscribe functionality

* Test of reauth; fix major bug in `TestCounter`

I misread `Condvar::wait_timeout_while` as `Condvar::wait_timeout_until`,
and flipped my predicate.
This led to false negatives (i.e. tests that passed that shouldn't have).

* A fistful of doc comments

* Avoid race condition running multiple tests with same client project

This commit fixes a race condition which sometimes caused the SDK tests to fail
because multiple `spacetime generate` processes running concurrently
would clobber each others' output,
potentially deleting it while a `cargo build` or `cargo run` process was running.

Now, the test harness will only run `spacetime generate` at most once
for any given directory.

* Add env_logger to SDK test client

* RUST_LOG=trace when running test clients

* quieter logs in test client: only warn-level and higher
2023-09-12 12:50:47 -05:00
Mario Montoya c539e0ab2a Provisional 'subscribe to all' functionality (#132)
* Provisional 'subscribe to all' functionality

* Update test for subscribe to thread the tx

* Apply suggestions from code review

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>

* Fix lints

---------

Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2023-09-12 16:23:18 +02:00
Noa 50e8e28463 Module/instance thread rework (#222)
* Module/instance thread rework

* Update docker action versions
2023-09-11 19:43:38 +00:00
joshua-spacetime 987b0091f8 Revert "perf: subscriptions compute DataKey only when not present with row (#268)" (#274)
This reverts commit a328108a5e.
2023-09-11 19:08:27 +00:00
SteveBoytsun 9dc90bb04c Fixed json serializer getting progressively slower (#226)
Co-authored-by: Steve <steve@codefics.com>
2023-09-11 12:35:01 -05:00
Noa ddc3d95190 Switch back to cargo-binstall in Dockerfile (#251) 2023-09-11 17:17:29 +00:00
Alessandro Asoni 6fdcd8888a Add debug logs with load progress (#143) 2023-09-09 14:58:00 +02:00
joshua-spacetime a328108a5e perf: subscriptions compute DataKey only when not present with row (#268)
Fixes #259 

(1) Updates MemTable to use RelValue instead of ProductValue
(2) Adds a DataKey member to DataRef and RelValue
(3) Subscriptions compute DataKey only when not present on row
2023-09-08 15:39:24 -07:00
Kim Altintop c010bfadca test: Only update active toolchain (#223)
That is, avoid doing any more work than necessary.
2023-09-07 19:40:19 +00:00
Mario Montoya 31dbb8171f Fix recovery of sequences after restart (#158)
* Remove unused datastore traits

We previously anticipated factoring the database in such a way that
the typed vs untyped distinctions, and the transactional vs not,
semantics would be more visible. We have not used these distinctions
and no longer need to expose these traits.

* Fix recovery of sequences after restart

---------

Co-authored-by: George Kulakowski <george@clockworklabs.io>
2023-09-06 07:47:24 +02:00
Kamil Jakubus a56ea25efd Remove protobufjs from devDependencies 2023-09-02 15:44:07 +02:00
Phoebe Goldman a1e9984840 Multiple server configurations for CLI (#214)
Alters the CLI's configuration format to support storing multiple server configurations,
and having each server configuration store an (optional) default identity.
Many CLI commands take an additional -s argument, the server on which to operate,
which defaults to the configured default_server.
Using -s consistently requires renaming the publish skip-clippy short flag to -S.

Per discussion with @jdetter , this PR does not directly associate identities with servers.
Instead, each server configuration stores the server's "fingerprint" (i.e. PEM-formatted ECDSA public key),
and the CLI uses that public key to decode tokens to determine if they apply to a given server.
This architecture allows the CLI to behave reasonably
when multiple server configurations use the same set of tokens,
e.g. if multiple distinct URLs resolve to the same SpacetimeDB instance.
For example, one could imagine a configuration with server configurations for both http://127.0.0.1:3000
and http://localhost:3000, which should use the same set of identities.
2023-09-01 17:29:20 +00:00
Phoebe Goldman 5dcf616b69 Add /identity/public-key route to get ECDSA public key (#165)
This commit adds a new endpoint to the `/identity` block,
which returns the public key STDB uses to decode JWTs.
This will allow clients, incl. the CLI and SDKs,
to verify multiple identity/token pairs without spamming STDB with requests.

The public key can also be used as a "fingerprint", to identify the server.
This will allow clients to determine:
- When multiple URIs or addresses refer to the same STDB instance.
- When a STDB instance has rotated its keys, so past identities become invalid.
2023-09-01 10:40:03 -04:00
Mario Montoya 5389b8d1e6 Create st_constraints system table (#225)
* Create st_constraints

* Add constranints to bootstraping

* Clippy
2023-08-31 18:40:28 +00:00
Mario Montoya bb322b545e Fix reopening of db in test, keep stable the binary data on files (#190)
* Fix reopening of db in test, keep stable the binary data on files

* Fix conflict
2023-08-31 11:30:40 -05:00
joshua-spacetime 33e7d07947 chore: add more tracing to vm (#246) 2023-08-30 07:01:19 -07:00
George Kulakowski be289daf6f Update wasmer (#220)
* we need wasmer 4.1.1 or newer

* Add a newly required dependency on wasmer-types

* from_trap -> from::<Trap>

* Finish upgrading to wasmer 4

* Bump to Rust 1.72.0

---------

Co-authored-by: Noa <coolreader18@gmail.com>
2023-08-29 18:07:35 +00:00
Ingvar Stepanyan cc632ef9f6 Make ColumnIndexAttribute bitflags in the ABI (#212)
As we're adding more variants, it's getting harder to keep track of the "A implies B" relationships between enum variants and to correctly do checks like "is this column autoinc" in different places.

Using bitflags makes expressing those relationships more natural and checking simpler - e.g. you can just do `contains(AUTO_INC)` as a single bit operation instead of having to remember and list all possible variants that imply autoinc, especially if we're going to add more flags in the future.

Initially I was going to make this change just in C# module generator and convert between ABI enum and bitflags over there, but it seems better to do this at ABI level so it simplifies Rust handling as well.
2023-08-29 11:35:57 -05:00
Piotr Sarnacki 653ed4d103 Add labels to docker containers (#252) 2023-08-29 16:26:26 +02:00
Noa d52d0dc7e1 Fix logspam (#229) 2023-08-28 12:21:53 -05:00
Phoebe Goldman 74df2e3285 [SDK] on-disconnect callbacks (#235)
This commit adds the usual 3 callback-ey functions for on-disconnect events,
which fire when a connection closes.
These fire either when the connection is closed by the client,
or by the server.

- `on_disconnect` to register.
- `once_on_disconnect` to register one-shots.
- `remove_on_disconnect` to un-register.
2023-08-28 11:51:01 -04:00
Phoebe Goldman 9b084950cf [CLI] Report text of error response in logs (#241)
Prior to this commit, the `logs` subcommand would print an unhelpfully generic HTTP error
on failure. Notably, this error did not include the actual response body.

With this commit, the error includes the HTTP response body.
For example, trying to print logs for a server you don't own shows:

Error: Identity does not own database, expected: 69c07c49846bcf11d7b7cb411018ded585a5a6d9045f8a1a39fa638e9db48436 got: 915cff645f94f52467c43734d54a26d565abb69c7506e30127dd616ca4790414
2023-08-26 08:58:42 -04:00
John Detter 9ff8273f2d --locked is required now, otherwise our deps can break our build (#242)
Co-authored-by: Boppy <no-reply@boppygames.gg>
2023-08-25 18:28:47 +00:00