- Allow classifier flags to be specified at runtime, overriding configured flags
- Add a generic `apis_enabled` flag to disable all external API calls in the classifier
The motivation for this is to allow torrents to be much more quickly re-indexed by disabling API calls only for individual classifier runs.
For querying search result items, we try 2 strategies:
- the default strategy is always tried, and is usually the most performant
- for certain searches where items are filtered to a small number of results, and ordered with a limit, the default strategy can be very slow, so we try a CTE strategy, with order and limit on a materialized view of the complete results, and we put it in a race with the default strategy
Various optimisations to torrent processing:
- Only persist new content
- Republish failed hashes if any succeeded, instead of failing entire job
- Only delete torrent_contents records where necessary
- Increase default batch sizes
A rework of the torrent creation workflow: previously a `Torrent` record was always created with a corresponding `TorrentContent` record, which would usually be empty; following this a `classify_torrent` queue job would attempt classification then update the `TorrentContent` record.
Following this update, `Torrent` records will always be created in isolation, and a `process_torrent` job will then run in the queue. This will not only classify the torrent, but also perform any other tasks like search reindexing. For torrents that have already been matched to a piece of content, rematching will not occur (unless specified in the CLI command, see below), which saves a significant amount of work.
A new entity type, `TorrentHint` has been created for providing hints to the classifier (currently used only by the import tool). Previously any hints for the classifier were added directly to the `TorrentContent` record, which was a conflation of 2 different things (the classification result, and hints for the classifier), which is problematic when it comes to reclassification.
Additionally, a new CLI command, `reprocess` has been added, which will reprocess all torrents, classify them, and update the search index. For already matched torrents, rematching will only occur when passing the `--rematch` flag.
A few reasons for this change:
- It will prevent unclassified (but classifiable) torrents showing at the top of the list in the WebUI, which is confusing
- The new search index will require reindexing of all torrents, and the CLI command provides a simple way to do this
- In future, we'll want to hang further steps off the `process_torrent` job, such as rules-based deletion, and this provides the groundwork for that
Mike Gibson