Commit Graph

129 Commits

Author SHA1 Message Date
Kenneth Skovhede 6362dd3cf8 Updated query to remove warning from SQLite logging 2024-04-16 14:19:15 +02:00
Kenneth Skovhede 40dd3c6816 Updated all license mentions to MIT 2024-02-28 15:45:30 +01:00
tariandeath 396119aa40 Add indexes to improve backup query performance. 2022-02-26 20:24:14 -07:00
Daniel Gehriger d143f8b3f9 Improved speed of AppendFilesFromPreviousSetWithPredicate, making USN backups usable when there are large file sets 2020-08-18 11:43:23 +02:00
Daniel Gehriger 82c54dde31 Fixing AppendFilesFromPreviousSetWithPredicate 2020-08-17 23:01:20 +02:00
Daniel Gehriger c5bef5c2e0 Improved Improved speed of AppendFilesFromPreviousSetWithPredicate, making USN backups usable when there are large file sets 2020-08-17 18:06:38 +02:00
Daniel Gehriger 8e8c758c65 Improved speed of AppendFilesFromPreviousSetWithPredicate, making USN backups usable when there are large file sets 2020-08-17 17:27:45 +02:00
Kenneth Skovhede e45b8c4b2e Merge pull request #4049 from seantempleton/filesetdifference
Fix unexpected difference in fileset when using Stop Now
2020-03-25 07:11:04 +01:00
Kenneth Hsu 13e1f0dac6 Remove unused Database members. 2020-03-14 16:45:08 -07:00
Sean Templeton d5e064ae45 Fix a few more issues with stop now
When a backup is interrupted it will now show as a partial backup in the list of backups on the restore files page.

Temporary filelist volumes are protected from deletion in several places. All but the RestoreHandler
only protect the most recent filelist. The RestoreHandler protects all temporary filelist volumes as the user
might be restoring from one that would be deleted.

Prevent the CoCoL BackendRequests channel is retired exception from appearing when the backup is stopped by checking
if the channel is retired before trying to add the FlushRequest to the channel. Any exceptions from the BackendUploader
task will still be thrown.
2020-03-01 21:19:07 -06:00
Sean Templeton da08be84a6 Convert VolumeID to long instead of int 2020-02-29 16:43:10 -06:00
Sean Templeton 446120ae3e A synthetic filelist could not be uploaded due to the logic in UploadSyntheticFilelist. The code passing in the fileset id to the class would get the id based on if the volume was in the Temporary or Uploading states. If the volume was in the Uploading state, present on the remote backend, and of the right size it would get changed to Uploaded.
One of the first things UploadSyntheticFilelist does is throw an exception if the volume is not in the Uploaded state. Or if it is in the Uploaded state, the next check would prevent a synthetic filelist from being created if the volume was not in the Uploading or Temporary states.

Removed the check for the volume not being in the Uploaded state as the warning in the second check handles that case.

Removed the using statement as it would dispose of the FilesetVolumeWriter after passing it to the UploadChannel. The UploadChannel could not use it since it had been disposed.

Fixed the GetRemoteVolumeFromIDAsync taking a fileset id and matching it up against a volume id. Now it will use fileset id to get the volume that matches the fileset.

Change the StopNow disruption test to not use a synthetic file list. The test was written with it not working and therefore fails a check on the number of filesets after stopping a backup.

Bonus fix for a possible null exception in Dispose for VolumeWriterBase.
2020-02-23 16:50:36 -06:00
BlueBlock 488aed3823 fix merge 2019-09-07 15:02:10 -04:00
BlueBlock c8bb889f6d Merge remote-tracking branch 'upstream/master' into fix-stop-after-current-file 2019-09-07 11:27:20 -04:00
Kenneth Hsu 0a390bd31a Move NormalizeDateTimeToEpochSeconds method to Utility class. 2019-09-01 12:38:23 -07:00
BlueBlock c8df8c5354 move method to base 2019-09-01 13:12:03 -04:00
BlueBlock d82d146322 fix case of parameter 2019-08-30 11:16:12 -04:00
BlueBlock 08e2c18185 add handling of a partial backup
- add backup state to DB table 'fileset' (job database upgrade to version 10)
- modify the Restore page dropdown to display if a backup is "partial"
- modify retention logic to remove partial backups only when the next recent full-backup has been removed
2019-08-05 20:14:05 -04:00
Rune Henriksen f5f353fe19 Merge branch 'master' into feature/fix_path_storage2 2019-01-25 23:37:57 +01:00
Rune Henriksen f696842877 remove unsed method and call 2019-01-05 00:30:43 +01:00
Kenneth Skovhede 03722fcba1 Merge branch 'master' into feature/fix_path_storage2
# Conflicts:
#	Duplicati/Library/Main/Database/LocalRecreateDatabase.cs
2018-10-23 11:33:35 +02:00
Kenneth Hsu f4f676a1de Remove unnecessary variable assignment.
These variables are assigned values that are never read by any
subsequent instruction.
2018-09-28 20:13:45 -07:00
Kenneth Skovhede fd87bf6de5 Fixed a missing GetMetadataHashAndSizeForFile method and added transactions to GetFileHash 2018-09-07 14:46:32 +02:00
Kenneth Skovhede a8cb4af389 Merge branch 'master' into feature/fix_path_storage2
# Conflicts:
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Operation/Backup/BackupDatabase.cs
#	Duplicati/Library/Main/Operation/Backup/MetadataPreProcess.cs
2018-09-07 09:44:23 +02:00
Kenneth Skovhede 809ba9a856 Changed the default query to be v==2, as multiple users reported this to be fastest.
The previous default query is now v==4.

Added an additional v==3, and a `LIMIT 1` clause to v==4 as suggested here:
https://forum.duplicati.com/t/release-2-0-3-10-canary-2018-08-30/4497/25
2018-09-05 10:12:28 +02:00
Kenneth Skovhede d0026d8028 Reworked the file-path query, and enabled experimentally changing it by setting the environment variable TEST_QUERY_VERSION=1 or TEST_QUERY_VERSION=2.
Added check to the unittests to catch cases where it scans too many files.
2018-08-13 13:23:59 +02:00
Kenneth Skovhede 4d1bea86a6 Attempt at making a leaner version of getting the previous file's information 2018-08-11 14:39:03 +02:00
Kenneth Skovhede 5507496352 Fixed the query to correctly extract metadata for a file, using the Metadataset indirection 2018-08-11 14:37:23 +02:00
Kenneth Skovhede 52cb9fb1f9 Removed the deprecated --store-metadata option.
Rewrote the logic for pre-filtering files based on timestamps/metadata to be more verbose and well documented.
Fixed an issue where `--check-filetime-only` also checked the file size, which was not present due to the reduced query used to grab only the lastmodified information.
Fixed a problem where the check for existing metadata was wrong and produced a warning.

This fixes #3204
This handles some of #3194
2018-08-11 11:48:19 +02:00
Kenneth Skovhede 61060e05b8 Merge branch 'master' into feature/fix_path_storage2
# Conflicts:
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
2018-08-07 22:44:41 +02:00
Kenneth Skovhede e9d7c47602 Disabled logging block and file database access to reduce overhead in these queries.
Added an option to explicitly re-enable logging of the database queries (`--profile-all-database-queries`)
2018-08-07 09:40:43 +02:00
Kenneth Skovhede cbc18970d5 Re-applied the path-storage fix as the merge failed somehow 2018-06-14 10:12:24 +02:00
Kenneth Skovhede d042e70ca2 Changed the returned IEnumerable values to be fully created, which avoids issues with locks and transactions being prematurely released.
This might help with #3202
2018-06-04 11:56:54 +02:00
Kenneth Hsu 20f2f37806 Remove unnecessary calls to String.Format. 2018-06-02 12:02:55 -07:00
Daniel Gehriger 8236cae390 USN change journal based backup now correctly reporting source size 2018-05-02 11:24:24 +02:00
Daniel Gehriger 7bf0a25912 Revert "Merge remote-tracking branch 'upstream/feature/fix_path_storage' into feature/usn"
This reverts commit 10d6b4c31c, reversing
changes made to 1af7f9e0ca.
2018-04-23 23:02:52 +02:00
Daniel Gehriger 10d6b4c31c Merge remote-tracking branch 'upstream/feature/fix_path_storage' into feature/usn 2018-04-23 22:57:32 +02:00
Kenneth Skovhede a8a32ea115 Added refactoring to improve the storage requirements for the local database, and speed up various queries.
This fixes #1283
2018-04-23 22:31:47 +02:00
Daniel Gehriger 1af7f9e0ca Merge remote-tracking branch 'upstream/master' into feature/usn 2018-04-23 22:28:27 +02:00
Daniel Gehriger 2c1a0ddb4e Implemented backup based on changes recorded in NTFS USN journal 2018-04-19 22:59:28 +02:00
Kenneth Skovhede 79f7b22e0f Merge branch 'master' into concurrent_processing
Changed a few things on the way, removing the log process

# Conflicts:
#	Duplicati/Library/Main/Controller.cs
#	Duplicati/Library/Main/Database/ExtensionMethods.cs
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/Operation/FilelistProcessor.cs
#	Duplicati/Library/Main/Operation/PurgeBrokenFilesHandler.cs
#	Duplicati/Library/Main/Operation/TestFilterHandler.cs
#	Duplicati/Library/Main/Options.cs
#	Duplicati/Library/Main/ResultClasses.cs
#	Duplicati/UnitTest/BasicSetupHelper.cs
2018-04-11 23:02:47 +02:00
Kenneth Skovhede 842fd96543 Implemented a new logging system that is more transparent and allows a more granular way of picking log messages.
Added ID's to each log message and each exception to allow later introduction of a Knowledgebase service that explains each error in more detail.
2018-03-15 09:12:34 +01:00
Kenneth Skovhede 9093603be1 Added support for disabling the file scanner during backups.
This fixes #2963
2018-02-11 20:57:38 +01:00
Kenneth Skovhede 75bf6857c1 Merge branch 'master' into concurrent_processing
# Conflicts:
#	.travis.yml
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/ResultClasses.cs
#	Duplicati/Library/Main/Volumes/VolumeWriterBase.cs
#	Duplicati/Library/Utility/TempFile.cs
2018-01-20 15:02:41 +01:00
Kenneth Hsu ab489d9a28 Make parameter names in XML documentation match code. 2017-12-09 18:42:30 -08:00
Kenneth Hsu 1c7391399d Remove XML documentation for unmatched parameters. 2017-12-09 18:30:01 -08:00
Kenneth Skovhede d57c789f4f Merge branch 'master' into concurrent_processing
# Conflicts:
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Main/Options.cs
2017-09-18 11:56:35 +02:00
Kenneth Skovhede 7d3ac25426 Re-introduced the in-memory block-cache. This time keeping it simple with a check for size-induced collisions. 2017-09-16 18:44:25 +02:00
Dan Stahr 3b422ff058 Fixed some of the failing tests in concurrent_processing branch 2017-08-06 11:32:11 +01:00
Kenneth Skovhede 76139f207e Merge branch 'master' into concurrent_processing 2017-05-30 14:00:24 +02:00