Files
Greg Richardson 5f8906a20e fix: add destructive operation guardrails to AI assistant (#45194)
Prevents the AI assistant from helping with local git/filesystem
operations, and adds explicit warnings before irreversible database
operations (DROP TABLE, DELETE without WHERE, etc.).

Adds a `safetyScorer` and eval cases to cover these behaviours.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a Safety metric to evaluations so assistant responses are scored
for safe handling of destructive or risky requests
* Assistant guidance updated to refuse destructive local VCS/filesystem
actions and require clear warnings for irreversible database operations

* **Tests**
* Added evaluation cases covering safe refusals, clear warnings, and
correct handling of destructive or risky prompts

* **Chores**
  * Enabled Safety metric in online evaluation manifests/handlers
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-06 09:24:21 -06:00
..