av1conv.sh

Streamlined AV1 Conversion Script with Comprehensive Batch Analytics

Description

This is a highly optimised Bash script for Linux that automatically scans and converts video files to AV1 format using SVT-AV1 and ffmpeg. The script has been completely rewritten for performance and reliability, featuring intelligent source analysis, Dolby Vision-aware processing (HDR10 conversion for profiles 7/8 with smart skipping elsewhere), comprehensive audio/subtitle handling, smart film grain synthesis, and detailed batch processing statistics.

If you find it useful, consider buying me a coffee: https://ko-fi.com/geekphreek

Why This script? Well, I love automation. I love Bash and I love FFMPEG. I also have a NAS that pulls a lot of legally obtained video files and they vary in size, a lot. I don't have that much space to store them so I have this script to run every day and try to shrink them down. Every time I come across something I wish the script could do, I simply add it in. A few people asked me how I handle it and I shared this with the community. Enjoy.

NOTE: Latest update v2.8.2

This release improves encoding quality defaults and adds adaptive HDR/framerate-aware encoding:

Key Improvements in v2.8.2:

• CHANGED: Default svt_tune from 1 (PSNR) to 0 (VQ/subjective quality) — optimises for human perception rather than mathematical accuracy
• CHANGED: Default svt_fast_decode from 1 to 0 — removes decode-optimisation constraints that sacrifice encoding efficiency; modern AV1 playback devices handle standard decode fine
• IMPROVED: Surround audio bitrate defaults increased — 5.1 content: 384k → 512k, 7.1 content: 384k → 640k for better surround separation
• ADDED: HDR-aware CRF adjustment — automatically reduces CRF by 3 for HDR content to prevent banding in dark scenes and gradients
• ADDED: Framerate-aware GOP sizing — targets ~4 seconds based on actual framerate instead of fixed 120 frames (24fps→96, 30fps→120, 60fps→240)
• FIXED: Film grain + denoise interaction for Film content — grain synthesis mode now correctly denoises original grain before applying synthetic grain, preventing double-graining
• FIXED: Lock file race condition — atomic mkdir replaces file-based locking to prevent parallel instance conflicts
• FIXED: Multiple robustness improvements from code review:
  • mkvmerge return code checked in audio track assignment (prevents silent metadata failures)
  • mktemp failures handled gracefully in metadata tag creation
  • Empty file hashes no longer propagated to skip list
  • Skip list cache safe during iteration (no more mid-loop array modification)
  • Parallel mode file counter uses flock to prevent race conditions
  • ensure_temp_filename() capped at 100 iterations to prevent hangs
  • XML special characters escaped in MKV metadata tags
  • Invalid audio bitrate values (e.g. 1.5k) rejected with warning
  • Webhook JSON fallback properly escapes quotes in messages
  • Skip list temp file uses mktemp instead of predictable filename

Previous Update (v2.8.1):

• FIXED: Added functional verification of ffprobe during dependency checking

Previous Update (v2.8.0):

• ADDED: Atmos/DTS:X preservation - automatically detects and copies object-based audio (TrueHD Atmos, E-AC3 Atmos, DTS:X) alongside Opus transcode for maximum compatibility
• ADDED: Plex Integration with --test-plex - triggers library scan after each encode (configure plex_url and plex_token)
• ADDED: VMAF Preview with --vmaf-preview - side-by-side visual comparison testing 5 CRF values with VMAF scores before encoding
• ADDED: Email notifications - sends batch summary at start and completion (configure SMTP settings, auto-downloads sendEmail if needed)
• ADDED: Custom output directory with --output-dir - preserves relative directory structure from source
• ADDED: Encoding time estimation - displays estimated time before each encode based on historical data
• UPDATED: Bundled FFmpeg to latest snapshot (N-122073-g7b773aba82, Dec 9 2025) with SVT-AV1 3.x improvements

Film Grain Intelligence

I have been struggling with using FFMpeg methods to detect grain. I've simplified how I do this. If you want to use the script and not detect grain, it simply uses your default settings. If you opt to have --detect-grain enabled, it'll use an ML model I have trained on just over 700 movies and TV shows, to try and work out the best settings to use.

From the detection it will do one of the following:

• FILM - Sets your CRF lower, turns off denoising and tries to preserve the original image as best as it can
• ANIMATION - Allows denoising, ignores grain, runs pretty much as you usually set
• TV - Allows denoising, but adds a light synthetic grain

To back that up, the base CRFs have been nudged down a notch: the global default now sits at 28, animation gets 29, TV hovers around 26, and film enjoys a very kind 22. The idea is simple—keep the shadows smooth without letting the files balloon.

There is an included executable for Linux, called media_classifier. This is a python3 built executable and essentially just tries to identify what type of media it is working on when you use --detect-grain. You can use the --detect-grain-test option of the script to test what it believes your media should be. You will find that some of the newer animation shows from the big companies are identified as FILM. This is because of how complex the animation is with shaders, and the rest.

Key Features

• Single-pass video analysis for maximum performance
• Automatic Dolby Vision detection with HDR10 conversion for profiles 7/8
• Intelligent film grain synthesis with preset-aware scaling
• Comprehensive HDR preservation for non-Dolby Vision content
• Quality-first SVT-AV1 encoding path with no experimental hardware code
• Parallel file processing for efficient batch operations
• Complete audio transcoding to Opus (with AAC fallback) for optimal quality and compression
• Smart subtitle handling (text and PGS formats)
• Detailed encoding statistics and metadata preservation
• Comprehensive batch summary with time metrics and compression analysis
• Lifetime statistics tracking across all runs with cumulative totals
• Intelligent space tracking (handles both compression gains and size increases)
• Real-time progress monitoring with running totals
• Shellcheck compliant code for reliability
• Expanded input format support covering MXF/WMV/FLV/3GP/ASF/VOB/RMVB/F4V/DivX/XviD

Installation

1. Place av1conv.sh in your PATH (e.g., /usr/local/bin/ or ~/bin/)
2. Make it executable: chmod +x av1conv.sh
3. Ensure all dependencies are installed (see Dependencies section)
4. Run the script with your preferred options

Verifying VMAF Support

To verify the bundled FFmpeg has full VMAF support:

chmod +x test_vmaf_bundled.sh
./test_vmaf_bundled.sh

The test will:

• Verify FFmpeg snapshot installation (N-122073-g7b773aba82)
• Confirm SVT-AV1 3.x encoder availability
• Confirm VMAF filter support
• Check for VMAF quality models
• Perform a test VMAF calculation

Usage

av1conv.sh [options]

General Options

• -h, --help - Display help message and exit
• -v, --verbose - Enable verbose output with detailed logging

Core Options

• -d, --dir DIR - Specify the directory to scan (default: /mnt/movies/)
• -1, --av1 - Force AV1 encoding for all files (including HEVC)
• -f, --force - Process files without prompting
• -F, --force-reencode - Force re-encoding even if files are already in AV1 format
• -c, --crf VALUE - Set CRF value (default: 28, range: 0-63, lower values for dark content)
• -p, --preset VALUE - Set preset value (default: 5, range: -1 to 13, optimised for quality/speed balance)
• -J, --parallel VALUE - Number of concurrent encodes (default: 1, max: 4; requires tmux for live monitoring)
• -g, --gop VALUE - Set GOP size (default: 120, optimised for streaming server seeking performance)
• -i, --ignore - Disable automatic filtering of CAM/WORKPRINT/TELESYNC files (filtering enabled by default)
• -l, --lazy - Run ffmpeg with nice for lower CPU priority
• -s, --size VALUE - Minimum file size to process (default: 1G, e.g., 500M, 2G)
• --since-date YYMMDD - Only process files modified after this date (e.g., 251014 for 14 Oct 2025)
• -r, --remove - Remove the input file after successful encoding
• -R, --resize-1080p - Resize videos to 1080p if source resolution is higher
• --resize-720p - Resize videos to 720p if source resolution is higher
• --allow-larger - Keep AV1 files even when larger than original source (default: revert to original)

Webhook Notification Options

• --webhook-url URL - Set webhook endpoint for real-time notifications
  • Telegram: https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>
  • Generic: Any endpoint accepting JSON POST with {"text": "message"} format
• --webhook-level LEVEL - Control notification verbosity (default: none)
  • none - No notifications
  • minimal - Batch start/end + errors only
  • summary - File-level summaries + minimal notifications
  • all - Detailed progress including per-file encoding steps
• --webhook-retries N - Number of retry attempts for failed deliveries (default: 3, range: 1-10)
• --test-webhook - Validate webhook configuration by sending test message

Notification Behaviour:

• Webhooks are disabled during startup to prevent spam (enabled after dependency checks)
• Failed deliveries are logged to /tmp/encode.log after all retries exhausted
• Supports Telegram, Discord, Slack, ntfy.sh, Microsoft Teams, and custom endpoints
• Safe to leave webhook_url="" empty - script continues normally without errors

Output Options

• --output-dir PATH - Write encoded files to a different directory instead of in-place

Output Directory Behaviour: By default, encoded files are written to the same directory as the source. Use --output-dir to write to a different location while preserving the relative directory structure:

# Source: /mnt/movies/h264/Action/Movie.mkv
# Output: /mnt/movies/av1/Action/Movie.mkv
./av1conv.sh -d /mnt/movies/h264 --output-dir /mnt/movies/av1

Note: If using -r (remove source) with --output-dir, the script will warn you that source files will be deleted while output goes to a different location. A 5-second pause allows you to abort if this wasn't intended.

Audio Options

• --stereo – Force downmix of surround audio tracks to stereo (useful for reducing audio complexity).
• -ab, --audiobitrate VALUE – Override default audio bitrate. Must end with 'k' (e.g., 128k). Defaults:
  • Mono: 64k
  • Stereo: 128k
  • 5.1 Surround: 512k
  • 7.1 Surround: 640k

Audio Codec Priority: The script uses Opus by default for superior quality and compression, with automatic fallback to AAC if Opus is unavailable.

Language & Subtitle Options

Note: These options are configured by editing variables at the top of the script.

• preferred_audio_language - Set preferred audio language (3-letter ISO 639-2 code, default: eng)
• preferred_subtitle_language - Set preferred subtitle language (3-letter ISO 639-2 code, default: eng)
• forced_subtitles_only - Only include forced/SDH subtitles by default (default: true)

Language Examples:

• English: eng
• French: fre
• German: ger
• Spanish: spa
• Italian: ita
• Japanese: jpn

Subtitle Behaviour:

• When forced_subtitles_only=true: Only includes forced subtitles (foreign dialogue, signs, etc.)
• When forced_subtitles_only=false: Includes all subtitle tracks in preferred language
• Script automatically detects forced subtitles from disposition flags and titles (Forced, SDH, CC)

SVT-AV1 Specific Options

• --svt-tune VALUE - Set tune value (default: 0 - VQ) [0=VQ (recommended), 1=PSNR, 2=SSIM (experimental)]
• --svt-overlays VALUE - Enable/disable overlays (default: 0, may harm quality/seeking)
• --svt-fast-decode VALUE - Set fast decode value (default: 0, off gives better compression)
• --svt-lookahead VALUE - Set lookahead frames (default: 120, range: 0-120)
• --svt-enable-qm VALUE - Enable quantization matrices (default: 1, range: 0-1)
• --svt-qm-min VALUE - Set min quantization matrix value (default: 0, range: 0-15)
• --svt-qm-max VALUE - Set max quantization matrix value (default: 15, range: 0-15)
• --svt-tile-columns VALUE - Set number of tile columns (default: 2, better multithreaded decoding)

Film Grain Options

• --svt-film-grain VALUE - Set film grain level (default: 0, range: 0-50)
  • 0: Disabled
  • 1-5: Light grain (good for digital content)
  • 6-10: Moderate grain (good for most film content)
  • 11-20: Strong grain (for heavy film grain content)
  • 21-50: Very strong grain (rarely needed)
• --detect-grain - Enable automatic film grain detection and intelligent scaling
• --no-detect-grain - Disable automatic film grain detection (even if enabled in config)
• --detect-grain-test FILE - Run the detector against a single file and exit with the verdict

Film Grain Intelligence

The script automatically scales film grain levels based on your chosen preset to balance quality and encoding speed:

• Preset 7: Reduces detected grain by 1/3
• Preset 8+: Halves detected grain level
• This prevents encoding performance issues while preserving visual quality

VMAF Quality Testing

What is VMAF?

VMAF (Video Multi-Method Assessment Fusion) is Netflix's perceptual video quality metric that predicts how humans perceive video quality. Scores range from 0-100, where higher is better:

• 95-100: Transparent quality (visually lossless)
• 90-94: Excellent quality (minor compression artefacts)
• 85-89: Very good quality (noticeable but acceptable)
• 80-84: Good quality (visible compression)
• <80: Lower quality (significant compression artefacts)

Important: VMAF measures quality loss from encoding, not absolute quality. A low-quality source (e.g., CAM rip, heavily compressed web-DL, or low-bitrate stream) will produce low VMAF scores even with minimal encoding loss. VMAF scores are relative to your source material—if you start with poor quality input, you cannot achieve high VMAF scores regardless of encoding settings. The script compares your encode against the original source, not against a theoretical perfect reference.

Adaptive CRF Search

When --test-vmaf is enabled, the script performs intelligent quality testing:

1. Sample Extraction: Extracts representative clips from your video (default: 5 samples × 30 seconds)
2. Test Encoding: Encodes samples at different CRF values
3. Quality Measurement: Calculates VMAF scores comparing original vs encoded
4. Adaptive Search: Automatically adjusts CRF to hit your target VMAF score
5. Full Encode: Uses the optimal CRF found for the complete file

This ensures consistent quality across your entire library, automatically compensating for:

• Complex scenes (action, grain, detail) → Lower CRF
• Simple scenes (animation, talking heads) → Higher CRF
• Different source quality levels

VMAF Testing Options

--test-vmaf              # Enable VMAF quality testing
--target-vmaf 95         # Target quality score (80-100, default: 95)
--vmaf-samples 5         # Number of test samples (1-10, default: 5)

Resolution-Aware Models

The script automatically uses the appropriate VMAF model:

• 4K content (≥2160p): Uses 4K VMAF model (vmaf_4k_v0.6.1.json)
• HD/FHD content: Uses standard VMAF model (vmaf_v0.6.1.json)

VMAF Testing Examples

# Transparent quality for archival (VMAF ≥95)
av1conv.sh -d /mnt/archive --test-vmaf --target-vmaf 95 -f

# Balanced quality for streaming (VMAF ≥90)
av1conv.sh -d /mnt/tv --test-vmaf --target-vmaf 90 -f

# Quick test with fewer samples (faster, slightly less accurate)
av1conv.sh -d /mnt/movies --test-vmaf --target-vmaf 93 --vmaf-samples 3 -f

# Maximum quality preservation with slow preset
av1conv.sh -d /mnt/uhd --test-vmaf --target-vmaf 98 -p 4 -f

Sample Output

║              VMAF QUALITY TESTING - ADAPTIVE CRF SEARCH            ║
Target VMAF: 95 | Initial CRF: 28

Iteration 1/5: Testing CRF 28
  Sample 1/5: 00:05:23-00:05:53 (30.0s)
    VMAF Score: 92.47
  Sample 2/5: 00:15:12-00:15:42 (30.0s)
    VMAF Score: 94.23
  ...
  Average VMAF for CRF 28: 93.15

Iteration 2/5: Testing CRF 26
  Average VMAF for CRF 26: 95.42

║                    VMAF TESTING COMPLETE                           ║
Optimal CRF: 26
Predicted VMAF: 95.42
Target VMAF: 95

Quality Assessment: Transparent (visually lossless)

Performance Considerations

• Testing Time: ~2-5 minutes per file (depends on sample count and preset)
• Accuracy: More samples = more accurate but slower (5 samples recommended)
• Best For: Archival content, mixed-quality sources, quality-critical workflows
• Skip For: Batch processing where speed is critical, uniform quality sources

VMAF Limitations and Expectations

Source Quality Matters: VMAF cannot improve poor source material. Common scenarios:

• High-quality Blu-ray rip: Can easily achieve VMAF 95+ with appropriate CRF
• Web-DL (good bitrate): Can typically achieve VMAF 90-95
• Compressed streaming source: May struggle to reach VMAF 85+, even at low CRF
• CAM/HDTS/Low-bitrate sources: Often cannot exceed VMAF 75-80 regardless of settings

If you're consistently unable to reach your target VMAF score even at very low CRF values (e.g., CRF 18-20), your source quality is likely the limiting factor, not your encoding settings. In these cases, the script will use the lowest tested CRF to preserve as much quality as possible from the available source material.

The CRF/VMAF Trade-off: Lower CRF values always produce higher VMAF scores, but with exponentially increasing file sizes and encode times. This creates a "chicken and egg" problem:

• CRF 18: Might achieve VMAF 98, but 3-4x larger files and 2-3x longer encode times
• CRF 24: Might achieve VMAF 93, with balanced file size and reasonable encode times
• CRF 28: Might achieve VMAF 88, with excellent compression and fast encoding

Use VMAF Preview Wisely: The --vmaf-preview feature is designed to help you make informed decisions by showing side-by-side comparisons. Don't blindly chase the highest VMAF score - use your eyes and judge what looks acceptable to you. A VMAF score of 90 that you can't visually distinguish from the source is better than spending 3x the encode time to achieve VMAF 95 with imperceptible improvements. VMAF is a guide, not a gospel! Trust your own visual assessment and find the sweet spot between quality, file size, and encoding time that works for your needs.

Batch Processing Analytics

Real-Time Progress Monitoring

Progress: [3/5] | Skipped: 1 | Space Saved: 2.34G

• Live file counter with running totals
• Skipped file tracking for Dolby Vision and corrupted files
• Real-time space savings (or size increases) as encoding progresses

Comprehensive Batch Summary

At the end of each batch, the script displays a detailed summary table:

+--------------------------------------------------------------------------+
| BATCH PROCESSING COMPLETE - FINAL SUMMARY                                |
+--------------------------------------------------------------------------+
| Metric                 | Count/Size             | Details                |
+--------------------------------------------------------------------------+
| Total Files Found      | 12                     | All eligible files     |
| Files Processed        | 10                     | Successfully encoded   |
| Files Skipped          | 2                      | Dolby Vision/errors    |
+--------------------------------------------------------------------------+
| Size & Compression Analysis                                              |
+--------------------------------------------------------------------------+
| Original Total Size    | 47.23G                 | Before encoding        |
| Final Total Size       | 28.91G                 | After encoding         |
| Total Space Saved      | 18.32G                 | Storage reclaimed      |
| Overall Reduction      | 38.78%                 | Compression ratio      |
+--------------------------------------------------------------------------+
| Time & Performance Metrics                                               |
+--------------------------------------------------------------------------+
| Total Processing Time  | 02:34:17               | Wall clock time        |
| Average per File       | 00:15:26               | Processing efficiency  |
+--------------------------------------------------------------------------+
| Mission Accomplished!                                                    |
+--------------------------------------------------------------------------+
| Excellent compression! You've freed up serious storage space.            |
+--------------------------------------------------------------------------+

Lifetime Statistics

After the batch summary, if you have previous run history, a lifetime statistics section is displayed:

+--------------------------------------------------------------------------+
| LIFETIME STATISTICS (All Runs)                                           |
+--------------------------------------------------------------------------+
| Metric                 | Value                  | Details                |
+--------------------------------------------------------------------------+
| Total Runs             | 47                     | Since 2024-01-15       |
| Files Processed        | 523                    | All time               |
| Space Saved            | 892.45G                | Cumulative             |
| Total Input            | 2.34T                  | Original sizes         |
| Total Output           | 1.47T                  | Encoded sizes          |
| Overall Reduction      | 37.18%                 | Average compression    |
| Total Encode Time      | 156:23:47              | Cumulative             |
+--------------------------------------------------------------------------+

Lifetime statistics are stored in ~/.cache/av1conv/global_stats.txt and persist across all runs. At startup, if you have previous history, you'll see:

Loaded lifetime stats: 523 files, 892.45G saved across 47 runs

Intelligent Commentary

The script provides contextual feedback based on results:

• Excellent compression (>50% reduction): Celebrates significant space savings
• Good compression (30-50% reduction): Acknowledges solid results
• Modest gains (10-30% reduction): Notes worthwhile improvements
• Minimal changes (<10%): Suggests files were already well compressed
• Size increases (negative reduction): Advises on encoding setting adjustments

File Size Handling

Default Behaviour (Revert on Size Increase)

By default, when AV1 encoding results in larger files (common with VP9→AV1 or already efficient sources), the script automatically reverts to the original file:

AV1 encode is of a higher physical size, reverting
Space Saved: 0 (file reverted)

Allow Larger Files Option

Use --allow-larger or set allow_larger_files=true in config when you want to keep AV1 files regardless of size:

# Keep AV1 files even if larger
av1conv.sh -d /mnt/movies --allow-larger

# Results in honest reporting of size increases:
Space Saved: +672.29K  (file grew larger)
AV1 encode is larger (+672.29K) but keeping as requested

When to use --allow-larger:

• Codec migration: Moving from VP9/HEVC to AV1 for future compatibility

• Quality preservation: Maintaining maximum quality despite size increases

• Hardware compatibility: Ensuring playback on AV1-optimised devices

• Archive standardisation: Converting entire libraries to a single codec

• Clear indication when files increase in size

• Honest reporting of actual storage impact with positive/negative space tracking

Examples

# Basic conversion with automatic grain detection and full analytics
# Note: CAM/WORKPRINT/TELESYNC files are filtered by default
av1conv.sh -d /mnt/movies --detect-grain -v

# Fast batch conversion with progress monitoring
av1conv.sh -d /mnt/tv -p 10 -c 35 -f

# Process ALL files including CAM/WORKPRINT/TELESYNC (disable filtering)
av1conv.sh -d /mnt/movies -i -f

# High quality conversion with custom grain and detailed statistics
av1conv.sh -d /mnt/films -p 6 -c 30 --svt-film-grain 5

# Batch convert and remove originals with comprehensive reporting
av1conv.sh -d /mnt/archive -r -f --resize-1080p

# Force AV1 conversion even when files grow larger (codec migration)
av1conv.sh -d /mnt/library --allow-larger -f

# Spin up two encodes in parallel with live progress panels
av1conv.sh -d /mnt/tv -J 2 -f

# Process only files modified after 14 Oct 2025 (great for weekly cron jobs)
av1conv.sh -d /mnt/tv --since-date 251014 -f

# VMAF quality testing - automatically find optimal CRF for target quality
av1conv.sh -d /mnt/movies --test-vmaf --target-vmaf 95 -f

# Quick VMAF test with fewer samples (faster but less accurate)
av1conv.sh -d /mnt/tv --test-vmaf --target-vmaf 93 --vmaf-samples 3 -f

# High quality encode with VMAF validation (for archival/preservation)
av1conv.sh -d /mnt/archive --test-vmaf --target-vmaf 98 -p 4 -f

# Enable Telegram notifications for batch monitoring
av1conv.sh -d /mnt/movies -f \
  --webhook-url "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>" \
  --webhook-level summary

# Minimal notifications (batch start/end + errors only)
av1conv.sh -d /mnt/tv -f --webhook-url "<YOUR_WEBHOOK_URL>" --webhook-level minimal

# Test webhook configuration before running batch
av1conv.sh --webhook-url "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>" \
  --test-webhook

Automated Processing with Cron

The --since-date option is perfect for scheduled batch processing:

# Weekly cron job: process files from the last week
# Runs every Sunday at 2am
0 2 * * 0 /usr/local/bin/av1conv.sh -d /mnt/media --since-date $(date -d "7 days ago" +\%y\%m\%d) -f -r

## Webhook Notifications

The script includes comprehensive webhook support for real-time monitoring of batch encoding operations. Perfect for unattended cron jobs or long-running batches.

### Supported Platforms
- **Telegram Bot API** - Native integration with automatic payload formatting
- **Discord** - Standard webhook endpoints
- **Slack** - Incoming webhook URLs
- **Microsoft Teams** - Connector webhooks
- **ntfy.sh** - Simple HTTP notifications
- **Custom endpoints** - Any service accepting JSON POST with `{"text": "message"}` format

### Notification Levels

**`none`** (default):
- No notifications sent
- Webhook functionality disabled

**`minimal`**:
- Batch start notification with file count
- Batch completion summary with statistics
- Error notifications only
- Perfect for cron jobs where you only want to know start/finish status

**`summary`**:
- All minimal notifications
- File start notifications (filename, size, codec)
- File completion notifications (size saved, time taken)
- Skipped file notifications (Dolby Vision, errors)
- Ideal for monitoring progress without excessive detail

**`all`**:
- All summary notifications
- Detailed encoding steps (grain detection, VMAF testing, etc.)
- Audio/subtitle processing updates
- Most verbose - useful for debugging or very small batches

### Configuration Methods

**Command-line arguments** (highest priority):
```bash
av1conv.sh -d /mnt/movies \
  --webhook-url "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>" \
  --webhook-level summary \
  --webhook-retries 5 \
  -f

Edit script defaults (lines 109-116):

webhook_url="https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>"
webhook_level="summary"
webhook_retries=3

Telegram Setup

1. Create a bot: Message @BotFather on Telegram
2. Get your token: BotFather will provide a token like 123456789:ABCdefGHIjklMNOpqrsTUVwxyz
3. Get your chat ID: Message @userinfobot to get your chat ID
4. Test the webhook:

   av1conv.sh --webhook-url "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<CHAT_ID>" \
     --test-webhook
   

Error Handling

• Startup protection: Webhooks disabled during dependency checks to prevent notification spam
• Retry logic: Configurable retry attempts (1-10) with 2-second delays between attempts
• Graceful failure: Failed deliveries logged to /tmp/encode.log without stopping the batch
• Safe defaults: Empty webhook_url is perfectly safe - script continues normally
• No impact on encoding: Webhook failures never affect video processing

Dolby Vision Handling

The script automatically detects Dolby Vision content and applies the safest possible path for each profile. Profiles 7 and 8 are converted to HDR10 by stripping the RPU metadata, while unsupported profiles (such as profile 5) or DV sources without an HDR10-compatible base layer are skipped. You can force skipping all Dolby Vision titles by editing the skip_dolby_vision variable at the top of the script.

• Accurate Detection: Inspects metadata, side data, and base layer compatibility
• HDR10 Conversion: Profiles 7/8 retain HDR tone mapping while removing dynamic metadata
• Clean Skipping: Unsupported profiles or forced skips are logged without failed attempts
• Statistics Integration: DV skips and conversions are clearly surfaced in batch summaries

Audio Processing

• Automatic conversion to Opus codec with optimal bitrates:
  • Mono: 64 kbps
  • Stereo: 128 kbps
  • 2.1 (3-channel): Copied as-is (Opus doesn't support 2.1, preserves original codec/quality)
  • 5.1 Surround: 512 kbps
  • 7.1 Surround: 640 kbps
• Fallback to AAC if Opus encoder is unavailable
• Language preservation with automatic English track prioritization
• Multi-track support for films with multiple audio options

Subtitle Handling

• Intelligent subtitle selection: Prioritises English text-based subtitles
• Format support: SRT, PGS (Blu-ray), and other common formats
• Text conversion: Converts compatible formats to SRT for broad compatibility
• PGS preservation: Maintains PGS subtitles when text alternatives aren't available

Output Format

• Container: Matroska (.mkv) format for maximum compatibility
• Filename: original_name, AV1.mkv (cleanly removes codec references)
• Metadata: Comprehensive encoding statistics and source information
• Chapters: Preserved from source material

Statistics and Reporting

Individual File Statistics

Each processed file generates a detailed statistics table showing:

• Codec comparison (original vs encoded)
• File size analysis with precise compression ratios
• Bitrate optimization results
• Processing time for performance analysis
• Encoding parameters used for reproducibility

Batch-Level Analytics

• Aggregate processing time and efficiency metrics
• Total storage impact across all processed files
• Success/failure rates with clear categorization
• Performance averaging for planning future batches

Intelligent Optimization Suggestions

Based on results, the script provides actionable advice:

• CRF adjustments for better compression
• Preset recommendations for speed vs quality balance
• Source format insights (when to skip certain codecs)
• Storage planning data for large libraries

Performance Optimizations

• Single ffprobe call per file for comprehensive video analysis
• Parallel processing for file filtering and analysis
• Efficient HDR detection without redundant metadata queries
• Intelligent thread management for optimal CPU utilisation
• Automatic ffmpeg thread capping to available cores (hard ceiling of 16)
• Memory-conscious temporary file handling
• Optimized progress tracking with minimal overhead

Dependencies

The script requires these tools to be installed and available in PATH:

Required

• ffmpeg (with SVT-AV1 support) - Video encoding engine
• ffprobe - Video analysis (usually bundled with ffmpeg)

NEW: Latest FFmpeg snapshot binaries (N-122073-g7b773aba82, Dec 9 2025) with SVT-AV1 3.x and VMAF support are bundled in the ffmpeg/ directory! The script automatically uses these if no system FFmpeg is found, ensuring consistent behaviour across all systems. Includes 15-25% faster encoding and improved perceptual quality.

• mkvpropedit - Matroska metadata editing
• jq - JSON processing for ffprobe output
• GNU parallel - Parallel file processing
• curl - HTTP requests for webhook notifications

Standard Linux Tools

• nice - Process priority management
• numfmt - Number formatting (coreutils)
• find - File discovery
• awk - Text processing
• bc - Calculator for grain detection and statistics
• column - Tabular output (required when using parallel mode)
• stdbuf - Stream buffering control (coreutils; required when using parallel mode)
• mktemp - Secure temporary file creation (coreutils)
• stat - File information (coreutils)
• grep - Pattern matching (coreutils)
• cut - Text column extraction (coreutils)
• head/tail - Text file operations (coreutils)
• sed - Stream editing (coreutils)
• tr - Character translation (coreutils)
• readlink - Symbolic link resolution (coreutils)

Optional (for parallel processing)

• tmux - Terminal multiplexer (required when using -J/--parallel > 1)

When running more than one concurrent job (-J/--parallel > 1), the script requires tmux to provide a clean split-pane display for monitoring multiple encodes simultaneously.

Installation on Debian/Ubuntu

sudo apt update
sudo apt install ffmpeg mkvtoolnix jq parallel bc curl coreutils findutils gawk util-linux

# For parallel processing support:
sudo apt install tmux

Technical Notes

• Smart bit-depth encoding: 8-bit sources → 8-bit AV1, 10-bit sources → 10-bit AV1 for optimal efficiency
• HDR preservation: Appropriate HDR metadata preservation with intelligent pixel format selection
• Streaming optimisation: Framerate-aware GOP targeting ~4 seconds for better seeking in Jellyfin/Plex
• HDR-aware encoding: Automatically lowers CRF for HDR content to prevent banding in dark scenes
• Perceptual quality tuning: VQ (subjective quality) mode by default for optimal visual results
• Thread safety: Uses proper locking for parallel operations
• Error handling: Comprehensive error checking and recovery
• Logging: Detailed timestamped logs written to /tmp/encode.log
• Temporary files: Secure temporary directory handling with cleanup
• Statistics: Complete encoding metrics with bitrate analysis
• Memory management: Efficient tracking of batch-level statistics

Configuration

The script uses a simple configuration system:

1. Built-in defaults - Edit variables at the top of the script (lines 54-123)
2. Command-line arguments - Override defaults at runtime (highest priority)

Key configuration variables include:

• directory - Default directory to scan
• crf, preset, gop - Encoding parameters
• preferred_audio_language, preferred_subtitle_language - Language preferences
• skip_dolby_vision - Skip Dolby Vision content
• max_parallel_jobs - Parallel encoding jobs
• since_date - Only process files modified after this date (YYMMDD format)
• webhook_url, webhook_level, webhook_retries - Webhook notification settings

Troubleshooting

• No files found: Check directory path and file size thresholds
• Encoding failures: Enable verbose mode (-v) for detailed error information
• Slow performance: Reduce film grain levels or use faster presets
• Out of space: Monitor available disk space in both source and temp directories
• Statistics errors: Ensure all dependencies are installed and accessible
• Files reverting to original: Default behaviour when AV1 is larger; use --allow-larger if needed
• Webhook not working: Use --test-webhook to validate configuration; check /tmp/encode.log for delivery errors
• Script aborting unexpectedly: Ensure curl is installed if using webhooks; failed webhook deliveries are safely logged without stopping the batch

Performance Tips

• Use faster presets (9-12) for quick conversions
• Disable film grain (--svt-film-grain 0) for maximum speed
• Enable lazy mode (-l) to reduce system impact during encoding
• Use higher CRF values (32-36) for animated or already-compressed content
• Monitor batch summaries to optimize settings for your library

Understanding Results

• Positive space savings: Files compressed successfully
• Files reverted: AV1 was larger than source, automatically reverted (default behaviour)
• Negative space savings: Files grew larger but kept (when using --allow-larger)
• Processing time variations: Depends on source complexity, settings, and hardware
• Skipped files: Dolby Vision content or corrupted files automatically excluded

Support

While this is primarily a personal tool, feel free to open issues for:

• Bug reports with detailed error information
• Feature requests for additional functionality
• Performance issues or optimisation suggestions
• Statistics or reporting enhancements

For general encoding questions, the FFmpeg and AV1 communities are excellent resources.

Contributing

Contributions are welcome! Please:

• Follow shellcheck guidelines for script quality
• Test thoroughly on various file types
• Update documentation for any new features
• Submit pull requests with clear descriptions

Authors and Acknowledgment

Created as a personal project. Special thanks to:

• FFmpeg developers for the incredible multimedia framework
• SVT-AV1 team for the excellent AV1 encoder
• Matroska developers for the flexible container format

License

Free and open source software. Use, modify, and distribute as you wish under the principles of open source development.

Project Status

Actively maintained and regularly updated with:

• Performance improvements
• Enhanced analytics and reporting features
• Bug fixes and reliability enhancements
• Support for new encoding features