`spacetime delete` may print the database tree and ask for confirmation.
We did not flush stdout, causing the tree to be interleaved with the
yes/no prompt.
# Expected complexity level and risk
1
# Description of Changes
Add back the instructions for regenerating CLI docs, which were removed
in https://github.com/clockworklabs/SpacetimeDB/pull/3343. I also made a
script for it.
This also fixes the CI checking this file, which was silently broken in
the same PR.
I have **not** verified that this works in Git Bash in Windows.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] CI fails if I change the CLI reference
- [x] CLI reference looks visually reasonable on a local `pnpm dev`
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Uses the `sourcemap` crate to map text locations in the bundle to text
locations in the original source code.
# Expected complexity level and risk
1 - essentially only related to diagnostics
# Testing
- [x] Manually tested
- [ ] Add an automated test for backtrace output
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3729
I genuinely don't know what came over me.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5 very straightforward but not strictly trivial
# Testing
Adds automated integration tests (written in Rust and run with `cargo
test`, although note this comment from @matklad about integration tests
for the future
https://internals.rust-lang.org/t/running-test-crates-in-parallel/15639/2):
- [x] Can publish an updated module if no migration is required
- [x] Can publish an updated module if auto-migration is required (with
the yes-break flag true/false)
- [x] Cannot publish if a manual migration is required
- [x] Can publish if a manual migration is required but the user
specified `--delete-data`
- [x] Can publish if a manual migration is required by the user
specified `--delete-data=on-conflict`
- [x] No data deletion occurs if no migration is required and
`--delete-data=on-conflict` is specified
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Bumping versions to 1.11.0 in preparation for an upcoming release.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Existing CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Fixes:
- [This
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441556875647647816)
by exporting the `SubscriptionHandle` type with the `REMOTE_MODULE` type
applied.
- [This
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441559246213746749)
by converting to `camelCase` for column names in code generation.
- Fixes an issue where `onMyReducer` callbacks were passing arguments as
variadic params, while the types indicated they would be passed as an
object. `onMyReducer((ctx, argA, argB, argC) => {})` vs
onMyReducer((ctx, { argA, argB, argC}) => {})`
- [Fixes an
issue](https://github.com/clockworklabs/SpacetimeDB/issues/3503#issuecomment-3566715928)
where the table type name was used instead of the table name in code
generation for constructing tables.
- [Fixes
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441886030436499466)
with `ScheduleAt` being used in non-table types.
- [Fixes
issue](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2168)
where template projects do not use the correct lifecycle reducer setup
# API and ABI breaking changes
Adds a new export, and fixes casing in code genreation.
# Expected complexity level and risk
2
# Testing
- I have tested that the `SubscriptionHandle` is correctly exported
- I have tested that the constraint name is now output in `camelCase`
- I have tested that `onMyReducer` callbacks now return a single
argument
- I have tested that the table name (and view name) is now used instead
of the type name for code generation
- I have tested that the new lifecycle reducers correctly compile
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This upgrades the SpacetimeDB version to 1.10.0.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
This is just a version bump - not tested.
# Description of Changes
This reverts commit 53f692dec6 (#3568)
That PR broke the build if any of the directories used as templates are
not clean. It spammed lots of errors like:
```
error: `C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/client.pdb` wasn't a utf-8 file
--> C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/client.pdb:0:49
|
error: `C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/ref/client.dll` wasn't a utf-8 file
--> C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/ref/client.dll:0:3
|
error: `C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/refint/client.dll` wasn't a utf-8 file
--> C:\Users\boppy\clockwork\SpacetimeDB\crates\cli/.templates/parent_parent_sdks_csharp_examples~_quickstart-chat_client/obj~/Debug/net8.0/refint/client.dll:0:3
```
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] `cargo build` passes on my repo now, which it wasn't before this
commit
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We technically don't need to `git ls-files` when getting the CLI
templates because when we compile the release builds we work with a
clean git clone, thus it's unlikely any untracked files will be created
in the templates directory.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] I've manually checked that the init command still generates the
templates properly
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Upgrade to version 1.9.0.
# API and ABI breaking changes
None - just a version upgrade.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I verified that the license has been updated
- [x] The version number looks correct (1.9.0)
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
The `spacetime dev` and `spacetime generate` commands always overwrite
the `.env.local` and `module_bindings` files, even if the contents are
identical. This can be a nuisance if your client is watching for changes
and reloading, as `spacetime dev` will modify both of those at different
times in the process.
This change will first read the contents of the files and see if a write
is necessary.
For writing `.env.local` as part of `spacetime dev`, the file contents
were already read so no real impact should exist.
For `module_bindings`, I've done some initial profiling with 181
generated typescript files and separately 161 rust files and noticed no
performance impact.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
I've ran this code on a project that generates 181 source files for
typescript and 161 for rust. It correctly writes files when changes have
been made and leaves the files unmodified if no changes are made.
For review testing, I would recommend running both with and without my
change on a much more complex project if it's available. I would think
not writing the contents of each file would mitigate the cost to read
most of the time, but perhaps in certain project makeup scenarios and
storage performance characteristics, this could possible be marginally
slower sometimes, though I have not observed that to be the case.
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
This PR modifies the `--delete-data` flag on `spacetime publish` and
adds the `--delete-data` flag on `spacetime dev`.
In particular instead of `--delete-data` being a boolean, it is now a an
enum:
- `always` -> corresponds to the old value of `true`
- `never` -> corresponds to the old value of `false`
- `on-conflict` -> clears the database, but only if publishing would
have required a manual migration
This flag does NOT change any behavior about prompting users to confirm
if they want to delete the data. Users will still be prompted to confirm
UNLESS they pass the separate `--yes` flag.
`spacetime dev` gets the same `--delete-data` flag. The default value of
`never` is equivalent to the existing behavior. `spacetime dev`
continues to publish with `--yes` just as before. This behavior is
unchanged.
# API and ABI breaking changes
Adds the flags specified above. This is NOT a breaking change to the
CLI. Passing `--delete-data` is the equivalent of
`--delete-data=always`.
This IS technically a breaking change to the `pre_publish` route. As far
as I'm aware this is *only* used by our CLI however.
> IMPORTANT SIDE NOTE: I would argue that `--break-clients` should
really be renamed to `--yes-break-clients` because it actually behaves
like the `--yes` force flag, but only for a subset of the user prompts.
I have not made this change because it would be a breaking change, but
if the reviewers agree, I will make this change.
# Expected complexity level and risk
2, Very small change, but if we get it wrong users could accidentally
lose data. I would ask reviewers to think about ways that users might
accidentally pass `--delete-data --yes`.
# Testing
- [ ] I have not yet tested manually.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Apparently, when `spacetime generate` passes files to a linter, the
files are passed in as relative paths, which did not play properly with
`dotnet format` (so we effectively borked it).
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] Unity testsuite no longer fails on expected diffs
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Fixes a subscription plan caching bug related to client-specific views.
Before this fix, you could define a client-specific view:
```rust
fn my_view(ctx: &ViewContext) -> Option<Player> {
ctx.db.player().identity().find(ctx.sender)
}
```
And subscribe to it as follows:
```sql
SELECT * FROM my_view
```
Note this view is implicitly parameterized by `:sender`, however when
generating a query hash for this subscription, this fact would not be
taken into account which would result in this query being cached and
reused for all callers.
After this fix, a query hash is generated for this subscription as
though it were given as:
```sql
SELECT * FROM my_view WHERE identity = :sender
```
# API and ABI breaking changes
None
**Note for CLI code owners:**
I had to touch the `subscribe` cli command file. No updates to the api.
It just needed to be updated to look for views in the module def.
# Expected complexity level and risk
1
# Testing
- [x] Added a regression smoketest
# Description of Changes
`spacetime generate` calls `dotnet format`.. but it did so with
`--folder`, without going to any particular directory. The result would
be that `dotnet format` tries to format _whatever directory it was run
from.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] `python3 -m smoketests namespaces` no longer hangs on my machine
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
The `config` added `maincloud` in the constructors so the test need
update.
# Expected complexity level and risk
1
# Testing
- [x] Re-run the test until pass
# Description of Changes
This template was never properly tested, and it contains wrong API usage
that can't even compile.
The only problem is that in order to make it work the person table has
to be public, and I'm not sure if we want to make that change.
Alternatively we could make basic-rust code not subscribe to anything
and just connect to the database.
# Expected complexity level and risk
2
# Testing
- [x] I compiled the template and checked that it works against
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- This upgrades the versions of all SDKs, the CLI, etc. to 1.8.0
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I verified that all versions seem to be updated including the BSL
license update <!-- maybe a test you want to do -->
We have 1 `1.7.0` that didn't get upgraded automatically because it is
part of the module bindings for a template:
```
crates/cli/.templates/parent_parent_crates_bindings-typescript_examples_quickstart-chat/src/module_bindings/index.ts: cliVersion: '1.7.0',
```
A case could possibly be made for bumping the template but it shouldn't
cause any issues as the module bindings directory should just get
regenerated by the user. @cloutiertyler should we be bumping module
bindings for templates when we upgrade versions?
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
This adds authorization to the relevant API endpoints, updates the CLI
commands and adds smoketests for the teams feature.
**Note**: Authorizing SQL (incl. subscriptions) is a bit more involved,
and submitted as a separate PR in the series.
Depends-on: https://github.com/clockworklabs/SpacetimeDB/pull/3519
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
# Description of Changes
This reverts commit #3496.
# API and ABI breaking changes
Technically maybe yes? But definitely nothing is using the new code yet.
# Expected complexity level and risk
1
# Testing
CI only
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
# Description of Changes
Bump versions to 1.7.0 in preparation for the release.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
# Testing
- [x] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
When building under Nix, Git metadata is not available within the
sandbox, as we use `lib.cleanSource` on our source directory. This is
important because it avoids spurious rebuilds and/or determinism
hazards.
The build was broken due to our new `spacetime init` template system
accessing Git metadata in the CLI's build.rs
to filter out non-git-tracked files from the templates. The Flake
sandbox does this automatically (even without `lib.cleanSource`!), so
when building under Nix it's unnecessary to do twice. (I remain
unconvinced that it's necessary to do in non-Nix builds either, as CI
builds should have a clean checkout and local dev builds don't need
clean templates, but the behavior was already in master and I didn't
feel comfortable removing it.)
As an enhancement, I've also found a Nix-ey way to embed our Git commit
hash in builds. Previously, builds under Nix had the empty string
instead of a commit hash, because we included the `git` CLI tool but
scrubbed the necessary metadata. Now, we inject an environment variable
from the Nix flake, and don't make the `git` CLI tool available at all.
This has the convenient upside of allowing Nix builds to reference
`dirtyRev` in builds with a dirty worktree, which should reduce
confusion.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3? I didn't have a strong understanding of what the CLI build script was
doing, and to what extent it was doing things intentionally versus for
convenience. As such, it's possible that I've inadvertently damaged
something load-bearing.
# Testing
- [x] Built with `nix build`, ran `spacetime init`, chose the
`basic-rust` template, and got a reasonable-looking template
instantiation.
- [ ] Hopefully we have automated tests for this?
# Description of Changes
This PR implements a new command for the CLI: `spacetime dev`. If run
from outside of a project directory, it will call `init` in interactive
mode to allow user to create a project. If run from a project directory,
it will ask for a module to conect to and it will:
* subscribe to logs
* observe files in the `spacetimedb` directory and automatically publish
to SpacetimeDB
One caveat is that instead of reusing the `logs` code I did some
repetition cause currently the logs code acquires a lock that lives
through await points. It is fixable, but due to limited time I decided
to go with a bit of code duplication. It shouldn't be very hard to fix
later, though.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
This is a draft of the new functionality for `spacetime init`. In order
to run it with built-in templates you have to set the path to the config
file:
```
export SPACETIMEDB_CLI_TEMPLATES_FILE=crates/cli/.init-templates.json
```
In the future it will fetch the list from GH.
A few notes:
* the previous functionality of `spacetime init` does not work at the
moment
* the code needs a bit more cleanup and tests before merging
* there is a bit of a mix in how we generate empty server and client
projects. For Rust we use the existing way of generating. For TypeScript
we clone an empty project from the repo. I wanted to play with both ways
of doing things, and I'm still not sure which is better. Generation in
Rust means that the generated code will match the CLI version and not
necessarily whatever is in Git. On the other hand, for the builtin
templates we will be fetching the newest version from GH, which I guess
might also not what we want, ie. we probably want only stable templates.
More discussion is needed here
* we use `spacetimedb` directory for the server files
* I don't particularly like the inability to disable interactive mode
easily. We discussed disabling it by default if all of the required
arguments are passed, but I don't think it's feature proof. For example,
if someone relies on a non-interactive mode, and we add a new required
argument, instead of printing a message `missing --foo`, we will
automatically launch interactive mode, which is harder to debug. That's
why I think I'd prefer to implement `--non-interactive` argument
* it's kind of hard to keep the legacy behaviour. If you don't pass any
arguments, we go into interactive mode. In the legacy version, we would
print required arguments. If someone passes `--lang` or `--project-path`
explicitly, I guess we could run the legacy workflow, but not sure if
it's worth it, as the command was marked as unstable anyway
* the project path defaults to the project name, but I think we should
probably replace change whitespaces to dashes, or at least ask for the
project path with the project name being the default (or both)
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This exposes client credentials in reducer calls for rust.
# API and ABI breaking changes
API Changes:
The main API change is the addition of `AuthCtx` and the `sender_auth`
in `ReducerContext`. This also adds JwtClaims, which has some helpers
for getting commonly used claims.
ABI Changes:
This adds one new functions `get_jwt`. This uses
`st_connection_credentials` to look up the credentials associated with a
connection id.
This adds ABI version 10.2.
# Expected complexity level and risk
2. This adds new ABI functions
# Testing
I've done some manual testing with modified versions of the quickstart.
We should add some examples that use the new API.
# Description of Changes
We didn't fix this after merging the 1.6.0 version bump into the
TypeScript PR.
I had originally fixed this in the release branch, but that patch got
lost once I recreated the branch, so we should release an updated CLI
(probably just update the existing GH release).
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Existing CI only
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Notify user in CLI that js/ts is beta quality.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Currently based on #3361
Implements most of the TS module API (not yet a function for type
aliases).
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Extremely basic module stuff works
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Noa <coolreader18@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Bumping versions to 1.6.0 in preparation for upcoming release.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Existing CI only
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Host-side changes extracted from #3327
I added AUTO_INC_OVERFLOW even though we don't currently ever return it,
in order to future-proof so it's already there when we start emitting
it.
Prepublish was failing because it was expecting a wasm module
unconditionally, so now it takes ?host_type.
I tweaked JS deser to accept null/undefined when the unit type or an
option type is expected.
I switched to bsatn, because the native sats->js translator wasn't
matching what js was expecting.
I renamed the sys module: my thinking is that `spacetime:` as a scheme
will help disambiguate it, and maybe it could also be used for IMC in
the future or something? And I believe we had discussed wanting this to
be versioned, similar to wasm imports.
Trying to get a borrowed str from deserialize_js doesn't work, because
v8 strings don't store utf8.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] All this was done in the course of getting an actual typescript
module to successfully publish.
# Description of Changes
It turns out that cargo automatically uses the latest semver-compatible
versions of dependencies, which is not what we expected. tl;dr
specifying `1.5.0` actually means `>=1.5.0 <2.0.0`, but we actually
intend `1.5.*`.
This PR updates our `upgrade-version` tool, and re-runs it to fix the
dep versions.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] I ran `cargo bump-versions 1.5.0 --rust-and-cli` to regenerate the
other committed files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Update:
This PR did all of the below but was split. Now it just does:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js`
This PR:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js` needs to be used.
2. Bumps V8 to 140.2.
3. Shares more logic with WASM and makes some minor refactorings to
energy/budget logic.
4. Moves logic from `WasmInstanceEnv` to `InstanceEnv` and friends.
5. Makes JS modules actually work in terms of `create_instance`,
`make_actor`,
6. Fleshes out `call_reducer` with timeouts and long-running logs added
as well.
7. Adds all the syscalls with associated documentation as well.
# API and ABI breaking changes
None
# Expected complexity level and risk
2? It's only available on unstable and mostly touches V8 stuff.
# Testing
Follow up PRs will add unit tests for parts.
We'll also need to add integration tests for whole modules.
# Description of Changes
PR contains:
* CLI changes for the `pre_publish` endpoint when publishing a module.
* The regular `--yes` flag will not bypasses the *break clients* warning
prompt — an extra confirmation is now required. For CI, a hidden flag
`--break-clients` is added.
* Added smoketest.
* Some trivial naming changes in `client-api-*` crates for consistency
reasons.
* `pre_publish` route to accept similar Body size limit as `publish`
route.
# API and ABI breaking changes
an additive API change, does not break anything.
# Expected complexity level and risk
2
# Testing
- Existing smoketests passing for backward compatibility.
- New smoketest for add columns
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Move the global.json file to the repo root, which should make `dotnet`
resolve the correct SDK regardless of the directory it is run in.
Where a copy of the global.json file is needed, symlink it to the one at
the root.
Should fix "C#/Unity - Test Suite".
# Description of Changes
Extends log records with a new field, `function`, which stores the name
of the reducer being executed when the log was produced. I have chosen
to name this field `function` rather than `reducer` because we will soon
be adding procedures, which are not reducers but will also be valid
values for this field.
While making this change, I noticed inconsistent values for injected
logs. Previously, we injected logs in three places, with different
values for the record fields:
1. `SystemLogger` (used when publishing and updating) set `filename:
Some("spacetimedb")` and `target: None`.
2. `log_reducer_error` (used for reducer error returns) set `filename:
None` and `target: Some(reducer)`, with `reducer` being the name of the
reducer.
3. `ModuleHost::inject_logs` (used for calls to nonexistent reducers and
calls with ill-typed arguments) set `filename: Some("external")` and
`target: None`.
With this change, I have decided that injected logs universally have
`filename: Some("__spacetimedb__")` and `target:
Some("__spacetimedb__")`. I have chosen the double-underscore convention
for reserved names to avoid confusion should a user name a source file
or reducer `spacetimedb`. I am not terribly attached to using
`spacetimedb` here, and would be happy to change to some other string,
like `internal` or `system`.
I have further decided that `log_reducer_error` and `inject_logs` both
have access to a sensible non-`None` value for `function` and so should
set it to the name of the reducer. The `target` field is not used for
this. Note that in cases where a client attempts to call a non-existent
reducer, the `function` field of the log record will be set to that
client-supplied non-existent reducer name.
# API and ABI breaking changes
Changes our log record format. This is at least backwards-compatible (as
in, newer consumers can read older logs) because the new field is
optional. I am unsure if it is forwards-compatible (as in, older
consumers reading newer logs) because I don't know if `serde_json` will
ignore unknown fields or will error.
# Expected complexity level and risk
1
# Testing
- [x] Manually ran `quickstart-chat` (modified to log messages and name
changes) and saw sensible output.
# Description of Changes
Bumping remaining files so that everything is at 1.4.0.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Closes#3219
This adds the Unreal SDK, the new Unreal test cases, updates the test
runner to handle Unreal, codegen updates for Unreal, and a QuickStart
Chat.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
2 - This impacts the subcommand generate.rs to include unrealcpp and
crates/testing to expand for Unreal
# Testing
- [x] Run the new Unreal tests
- [x] Run any previous automation testing - with all the changes to
generate/testing I'm uncertain if there is an impact
- [x] Review the new CLI generate documentation changes
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Closes
[#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686).
Add support for listening using the [PG wire
protocol](https://www.postgresql.org/docs/current/protocol.html) so `pg`
clients could be used against the database.
# API and ABI breaking changes
The output of `duration` is changed to `rfc3339`, instead of the way is
made with `sats` because is what is done in `pg`, see note below.
# Expected complexity level and risk
2
~~There is open questions that are in the [ticket
#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686). Also
the crate used here require `RustTls`, so it could be good idea to
decide if~~:
* ~~Rewrite a big chunk of code to use `OpenSSL`~~
* ~~Move to `RustTls`
https://github.com/clockworklabs/SpacetimeDB/pull/1700~~
* ~~Pay for the extra compilation cost~~.
I open another port(`5433`) to listen for `pg` connections using `ssl`.
Need to be decided if this is the way or instead try to multi-plex the
current port for both protocols.
# Testing
Only manual testing so far. Solving the above questions allow me to
implement some unit tests. Also, not yet integrated into cloud for the
same reasons.
- [x] Adding some test for the binary encoding of special and primitive
types
- [x] Smoke test using `psql` that connect to the db instance and run
some queries
- [x] Manually inspect using a UI database explorer how infer the types,
some of this tools generate special widgets when displaying `json,
duration, etc`
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Implements [subscribing to durable
commits](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1594).
The setting works on a per-connection level, and essentially just delays
sending transaction updates until the transaction is reported as durable
by the database.
For connectionless SQL operations, the setting works per-request. No SQL
syntax is provided by this patch to toggle the configuration.
After some deliberation, I opted to obtain the offset when a transaction
commits (as opposed to when it starts). This creates some mild
inconvenience, because we prevent the transaction from committing until
the corresponding subscription updates are enqueued.
The strategy is, however, more correct should we ever support weaker
isolation levels, and it is easier to document.
Follow-ups include:
- Provide SQL syntax (`SET synchronous_commit = ON` or something)
- C# and TypeScript SDKs
- Reference docs?
# API and ABI breaking changes
Not breaking, but adds a parameter to the subscribe and sql endpoints.
# Expected complexity level and risk
4
To the author's understanding, ordering of outbound messages is not
changed by this patch, even if there are messages that don't have a
transaction offset (such as error messages). I.e. while waiting for the
transaction offset of a message to become durable, no message enqueued
after that message will be delivered. This may not be desirable in some
cases.
The patch may contain concurrency bugs, e.g. awaiting futures that may
never resolve.
# Testing
- [x] Implemented a new test in the `module_subscription_actor` module
- [x] Added unit tests for the core logic in `ClientConnectionReceiver`
It would be desirable to also have integration-level tests, but I'm
currently unsure how to write those without being able to control if and
when the database reports an offset as durable.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Please note, much of the code changed in this PR is generated code.
This change updates the TypeScript SDK to use a new `spacetimedb`
TypeScript library which lives under the `/crates/bindings-typescript`
folder alongside `/crates/bindings-csharp`. Just like with the C#
bindings library, the types for `AlgebraicType` and `RawModuleDef` are
now code generated with a script in `/crates/codegen/examples`.
Pulling this out into a library allows us to use the same types and
serialization code on both the server and the client for TypeScript
modules.
In the process of making this change I also found and fixed several
issues with the TypeScript code generation. Namely an issue with
recursive types and an issue with the `never` type. I also removed any
use of `namespace`s since those are a TypeScript only feature, and we
want to have JavaScript + types so that we can use the generated code
with ESBuild without TSC.
I have also improved the npm/pnpm scripts to be able to generate
TypeScript code for us automatically by running `pnpm generate` for any
place that we have to generate typescript code. Namely:
- Quickstart module bindings
- AlgebraicType/ModuleDef for TypeScript module library
- Client API messages for the TypeScript API
- TestApp module bindings
# API and ABI breaking changes
IMPORTANT! This is an API breaking change for clients, as such it should
be a major version change. However, I am going to see if I can shim in
the old API as best as possible to make it compatible.
Notably, we were previously exporting APIs that end users do not need,
and I don't think it would ever occur to them to use, namely the whole
`AlgebraicValue` API and also the `AlgebraicType` API. In principle, no
one should have a need for these, although it was technically possible
for them to use it.
Indeed, we could potentially even just remove AlgebraicType completely
from the API by directly code generating the serialization code.
Listed below are all of the **BREAKING** changes to the API and their
effect:
- `AlgebraicType` is now a structural union literal type instead of a
class (nominal type), this is a consequence of generating the type with
our code gen. Users did not have a reason to use `AlgebraicType`
directly.
- The `AlgebraicValue` type has been removed entirely. This was
previously a class that was exported from the SDK, but very unlikely to
be used by users.
- The `ProductValue` type has been removed.
- The `ReducerArgsAdapter` and `ValueAdapter` types, which were used
with AlgebraicValues have been removed.
- Generated code has changed incompatibly so users will have to
regenerate code when upgrading to this version. Technically a breaking
change, but pretty easy to fix.
Listed below are the non-breaking API changes:
- I am now exporting generated product types as the default export in
addition to how I was exporting them before
- For generated sum type `T`, I am now exporting a `TVariants` namespace
which has the types of all the variants for `T`. This was previously
exposed on the namespace `T`, but was inaccessible in modules other than
the one it was defined in because I also export a type `T` in addition
to namespace `T` and in the type position `T` was being interpreted as a
type rather than a namespace. It's unclear why TypeScript resolved it as
the `T` namespace within the module in which is was defined previously.
Anyway, since the old types were apparently unobservable to users, I've
replaced them with the types in `TVariants`. (Open to other names for
this namespace).
- I fixed a bug where never types (sum types with no variants) were not
correctly generated.
# Expected complexity level and risk
3. The most complex thing about the PR are the potential impacts to the
API. I am reasonably certain, but not 100% certain that I have accounted
for everything above.
# Testing
- [x] I ran `pnpm test` which runs all the tests in the repo including
an integration test which tests the connection to SpacetimeDB. I also
fixed broken tests.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Simple patch version bump. Because we forgot to bump the version
number for 1.3.1 this is bumping from 1.3.0 => 1.3.2
# API and ABI breaking changes
0
# Testing
- Verified that the version number in the license has been updated
- Verified the output of `spacetime --version`:
```
$ spacetime --version
spacetime Path: C:\Users\boppy\AppData\Local\SpacetimeDB\bin\current\spacetimedb-cli.exe
Commit: 0027d094e1
spacetimedb tool version 1.3.2; spacetimedb-lib version 1.3.2;
```
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Bumped all versions to 1.3.0 in preparation for an upcoming minor
release.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
The `subscribe` command would drop the connection without sending a
close frame. Doing so creates a warning on the server, which can be
distracting when debugging other connections issues.
# Expected complexity level and risk
1
# Testing
Use the `subscribe` command with a local database and compare server
logs before
and after.
Mario Alejandro Montoya Cortes