CI - fix LLM benchmark workflows (#5181)

# Description of Changes

Updates `llm-benchmark-periodic.yml` and
`llm-benchmark-validate-goldens.yml` to new CI infrastructure.

- Switch to the current runner (`spacetimedb-new-runner-2`)
- Drop the dead `localhost:5000` container + `--privileged`
- Build the local SpacetimeDB server the benchmark harness needs, and
use that
  same local CLI for publishing

# API and ABI breaking changes

None. CI-only.

# Expected complexity level and risk

1 — workflow-only, no production code.

# Testing

- [ ] Run both workflows via `workflow_dispatch` and confirm they pass
This commit is contained in:
bradleyshep
2026-06-05 09:01:21 -04:00
committed by GitHub
parent 31699135d0
commit 7a2a697ee4
2 changed files with 16 additions and 20 deletions
+8 -10
View File
@@ -29,19 +29,10 @@ concurrency:
jobs:
run-benchmarks:
runs-on: spacetimedb-new-runner
container:
image: localhost:5000/spacetimedb-ci:latest
options: >-
--privileged
runs-on: spacetimedb-new-runner-2
timeout-minutes: 180
steps:
- name: Install spacetime CLI
run: |
curl -sSf https://install.spacetimedb.com | sh -s -- -y
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Checkout master
uses: actions/checkout@v4
with:
@@ -75,6 +66,13 @@ jobs:
- name: Build llm-benchmark tool
run: cargo install --path tools/xtask-llm-benchmark --locked
- name: Build SpacetimeDB server for benchmark harness
run: |
cargo ci smoketests prepare
mkdir -p "$HOME/.local/bin"
ln -sf "$GITHUB_WORKSPACE/target/release/spacetimedb-cli" "$HOME/.local/bin/spacetime"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Run benchmarks
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
@@ -15,11 +15,7 @@ concurrency:
jobs:
validate-goldens:
runs-on: spacetimedb-new-runner
container:
image: localhost:5000/spacetimedb-ci:latest
options: >-
--privileged
runs-on: spacetimedb-new-runner-2
timeout-minutes: 60
strategy:
@@ -28,11 +24,6 @@ jobs:
lang: [rust, csharp, typescript]
steps:
- name: Install spacetime CLI
run: |
curl -sSf https://install.spacetimedb.com | sh -s -- -y
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Checkout master
uses: actions/checkout@v4
with:
@@ -70,6 +61,13 @@ jobs:
- name: Build llm-benchmark tool
run: cargo install --path tools/xtask-llm-benchmark --locked
- name: Build SpacetimeDB server for benchmark harness
run: |
cargo ci smoketests prepare
mkdir -p "$HOME/.local/bin"
ln -sf "$GITHUB_WORKSPACE/target/release/spacetimedb-cli" "$HOME/.local/bin/spacetime"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Validate golden answers (${{ matrix.lang }})
env:
MSBUILDDISABLENODEREUSE: "1"