mirror of
https://github.com/duplicati/duplicati.git
synced 2026-05-06 07:16:38 -04:00
Kenneth Skovhede
3ec07aaee1
This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms. Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks. The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents. To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files. This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing. To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes. The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up. The ZStandard compression is from `ZstdSharp.Port` which has excellent performance. When we move to .NET11 this can easily be changed to the new built-in module. Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing. Co-authored-by: Copilot <copilot@github.com>
7 lines
160 B
JSON
7 lines
160 B
JSON
{
|
|
"name": "ZstdSharp.Port",
|
|
"description": "Port of zstd compression library to C#",
|
|
"link": "https://github.com/oleg-st/ZstdSharp",
|
|
"license": "MIT"
|
|
}
|