Removing deleted files from a backup set


Author
Message
falexc
falexc
New Member
New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)
Group: Awaiting Activation
Posts: 2, Visits: 4
If I use an incrementals forever plan, will files that are backed up and later deleted from the drive stay in the backup chain forever?  Is there any way to purge deleted files from a backup chain after some amount of time?
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)
Group: Forum Members
Posts: 2.8K, Visits: 19K
I use this strategy myself. Smile  I'll assume you're using Incrementals Forever with Synthetic Fulls enabled and address that scenario first, then afterward I'll address what works differently when that's disabled.

Data would only remain in the chain forever if you did not specify a retention policy.  If you do have a retention policy, whenever a new Inc causes the total number of Incs to exceed that policy, the oldest Inc is merged into the root Full and then deleted.  So for example let's say you capture a Full backup, then you delete 10GB worth of data from your source drive, then you capture an Incremental (call it Inc #1).  Let's also say for simplicity that your retention policy specifies that you want to keep 2 Incrementals.  Your next job generates Inc #2, and the job after that generates Inc #3 -- but since that's too many for your retention policy, Inc #1, which was captured after you deleted that 10GB of data, gets merged into the original Full.  As a result of that operation:
  • Inc #1 no longer exists as an independent entity in your backup set.  Instead, the root Full backup was "carried forward" to match the state of Inc #1, as if you had actually captured a Full on the day you captured Inc #1, even though you didn't -- hence the name "Synthetic Full".
  • It's no longer possible to restore back to the state of the Full you originally did capture, since again your Full has now been carried forward.
  • That 10GB of data you previously deleted no longer exists in your backup set, since there's no longer a backup in your set from the time that that data existed.
If you were running Incrementals Forever with Synthetic Fulls disabled, then the root Full remains frozen in time, and the merge operations will instead consolidate the second oldest Incremental into the oldest Incremental (rather than the oldest Incremental into the Full).  In this case, any data deleted after you captured your original Incremental would still eventually be purged from the set once the last backup that contains that data gets "merged out", but data contained in the Full backup that was deleted afterward will not ever be deleted from the set, since the Full is static. The only real reason to disable Synthetic Fulls would be if you wanted to replicate your backups somewhere else and having to replicate an updated Full after every backup would be impractical for bandwidth reasons.  But if that scenario applies to you, it can be a good idea to manually capture a new Full every now and then, even though that's technically not "Incrementals Forever" anymore.

Finally, note that with either strategy, the file size of the root Full (or oldest Incremental, if Synthetic Fulls are disabled) will always be the size of the backup it contains plus the size of the largest backup that has ever been consolidated into it.  There are technical reasons for this, but I mention it because in the scenario I described above, technically after that first merge operation you would see the file size of your root Full backup grow rather than shrink by ~10GB (depending on compression), even though the backup it contains is now smaller -- but that's only because it was the first merge operation.  The root Full (or oldest Inc) will not keep growing like that forever, unless of course the amount of data you're backing up continues to grow

Incidentally, Macrium has some nice animated demos of both of these variants:
Synthetic Fulls enabled -- https://www.youtube.com/watch?v=6aeWSmT2TP0
Synthetic Fulls disabled -- https://www.youtube.com/watch?v=1hhne4HHTKs

Edited 27 October 2017 7:01 PM by jphughan
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)
Group: Forum Members
Posts: 2.8K, Visits: 19K
One more technical note.  With Incrementals Forever, regardless of whether you're using Synthetic Fulls, do not rely on the Windows "Date Created" and "Date Modified" timestamps of your backup files to figure out when the backup they contain was captured.  Instead, look at the "Backup Date" shown in Reflect under the Restore tab, or right-clicking the file in Windows, select Properties, and select the Macrium Reflect tab.  Here's why the Windows timestamps won't be reliable:
  • The "Date Created" timestamp will always be the date the root Full (or oldest Inc) was originally created, which after a merge has occurred will always be older than the backup it actually contains.
  • As a result of the merge operations, the Date Modified timestamp on the file being "merged into", i.e .either the root Full or the oldest Incremental, will be the date the merge occurred, not the date of the backup that file now contains.  The merge date will of course be newer.
  • If you have Delta Incremental Indexes disabled (the default for Reflect V6), then every time a merge occurs, all backups in the chain after the backup that was just merged will need their indexes updated.  As a result, with Synthetic Fulls enabled, all of your backups will show the same Date Modified timestamp, specifically the date of the last merge. (If you have Synthetic Fulls disabled, this will still be true except for the root Full.)  If you have Delta Incremental Indexes enabled (the default for V7), then this bullet point would not apply, but in that case, the merge operation will still result in your oldest (or second oldest) backup in the set having the most recent Date Modified timestamp.  If you need to disable Synthetic Fulls because you're using a replication strategy like I described above, then it would be a good idea to also enable Delta Incremental Indexes, otherwise you'd have to replicate every Incremental in the set after every backup that triggered a merge operation.

Edited 27 October 2017 7:29 PM by jphughan
falexc
falexc
New Member
New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)New Member (4 reputation)
Group: Awaiting Activation
Posts: 2, Visits: 4
jphughan - 27 October 2017 6:15 PM
I use this strategy myself. Smile  I'll assume you're using Incrementals Forever with Synthetic Fulls enabled and address that scenario first, then afterward I'll address what works differently when that's disabled.

Data would only remain in the chain forever if you did not specify a retention policy.  If you do have a retention policy, whenever a new Inc causes the total number of Incs to exceed that policy, the oldest Inc is merged into the root Full and then deleted.  So for example let's say you capture a Full backup, then you delete 10GB worth of data from your source drive, then you capture an Incremental (call it Inc #1).  Let's also say for simplicity that your retention policy specifies that you want to keep 2 Incrementals.  Your next job generates Inc #2, and the job after that generates Inc #3 -- but since that's too many for your retention policy, Inc #1, which was captured after you deleted that 10GB of data, gets merged into the original Full.  As a result of that operation:
  • Inc #1 no longer exists as an independent entity in your backup set.  Instead, the root Full backup was "carried forward" to match the state of Inc #1, as if you had actually captured a Full on the day you captured Inc #1, even though you didn't -- hence the name "Synthetic Full".
  • It's no longer possible to restore back to the state of the Full you originally did capture, since again your Full has now been carried forward.
  • That 10GB of data you previously deleted no longer exists in your backup set, since there's no longer a backup in your set from the time that that data existed.
If you were running Incrementals Forever with Synthetic Fulls disabled, then the root Full remains frozen in time, and the merge operations will instead consolidate the second oldest Incremental into the oldest Incremental (rather than the oldest Incremental into the Full).  In this case, any data deleted after you captured your original Incremental would still eventually be purged from the set once the last backup that contains that data gets "merged out", but data contained in the Full backup that was deleted afterward will not ever be deleted from the set, since the Full is static. The only real reason to disable Synthetic Fulls would be if you wanted to replicate your backups somewhere else and having to replicate an updated Full after every backup would be impractical for bandwidth reasons.  But if that scenario applies to you, it can be a good idea to manually capture a new Full every now and then, even though that's technically not "Incrementals Forever" anymore.

Finally, note that with either strategy, the file size of the root Full (or oldest Incremental, if Synthetic Fulls are disabled) will always be the size of the backup it contains plus the size of the largest backup that has ever been consolidated into it.  There are technical reasons for this, but I mention it because in the scenario I described above, technically after that first merge operation you would see the file size of your root Full backup grow rather than shrink by ~10GB (depending on compression), even though the backup it contains is now smaller -- but that's only because it was the first merge operation.  The root Full (or oldest Inc) will not keep growing like that forever, unless of course the amount of data you're backing up continues to grow

Incidentally, Macrium has some nice animated demos of both of these variants:
Synthetic Fulls enabled -- https://www.youtube.com/watch?v=6aeWSmT2TP0
Synthetic Fulls disabled -- https://www.youtube.com/watch?v=1hhne4HHTKs
jphughan - 27 October 2017 7:21 PM
One more technical note.  With Incrementals Forever, regardless of whether you're using Synthetic Fulls, do not rely on the Windows "Date Created" and "Date Modified" timestamps of your backup files to figure out when the backup they contain was captured.  Instead, look at the "Backup Date" shown in Reflect under the Restore tab, or right-clicking the file in Windows, select Properties, and select the Macrium Reflect tab.  Here's why the Windows timestamps won't be reliable:
  • The "Date Created" timestamp will always be the date the root Full (or oldest Inc) was originally created, which after a merge has occurred will always be older than the backup it actually contains.
  • As a result of the merge operations, the Date Modified timestamp on the file being "merged into", i.e .either the root Full or the oldest Incremental, will be the date the merge occurred, not the date of the backup that file now contains.  The merge date will of course be newer.
  • If you have Delta Incremental Indexes disabled (the default for Reflect V6), then every time a merge occurs, all backups in the chain after the backup that was just merged will need their indexes updated.  As a result, with Synthetic Fulls enabled, all of your backups will show the same Date Modified timestamp, specifically the date of the last merge. (If you have Synthetic Fulls disabled, this will still be true except for the root Full.)  If you have Delta Incremental Indexes enabled (the default for V7), then this bullet point would not apply, but in that case, the merge operation will still result in your oldest (or second oldest) backup in the set having the most recent Date Modified timestamp.  If you need to disable Synthetic Fulls because you're using a replication strategy like I described above, then it would be a good idea to also enable Delta Incremental Indexes, otherwise you'd have to replicate every Incremental in the set after every backup that triggered a merge operation.

Sorry for the late reply
jphughan - 27 October 2017 6:15 PM
I use this strategy myself. Smile  I'll assume you're using Incrementals Forever with Synthetic Fulls enabled and address that scenario first, then afterward I'll address what works differently when that's disabled.

Data would only remain in the chain forever if you did not specify a retention policy.  If you do have a retention policy, whenever a new Inc causes the total number of Incs to exceed that policy, the oldest Inc is merged into the root Full and then deleted.  So for example let's say you capture a Full backup, then you delete 10GB worth of data from your source drive, then you capture an Incremental (call it Inc #1).  Let's also say for simplicity that your retention policy specifies that you want to keep 2 Incrementals.  Your next job generates Inc #2, and the job after that generates Inc #3 -- but since that's too many for your retention policy, Inc #1, which was captured after you deleted that 10GB of data, gets merged into the original Full.  As a result of that operation:
  • Inc #1 no longer exists as an independent entity in your backup set.  Instead, the root Full backup was "carried forward" to match the state of Inc #1, as if you had actually captured a Full on the day you captured Inc #1, even though you didn't -- hence the name "Synthetic Full".
  • It's no longer possible to restore back to the state of the Full you originally did capture, since again your Full has now been carried forward.
  • That 10GB of data you previously deleted no longer exists in your backup set, since there's no longer a backup in your set from the time that that data existed.
If you were running Incrementals Forever with Synthetic Fulls disabled, then the root Full remains frozen in time, and the merge operations will instead consolidate the second oldest Incremental into the oldest Incremental (rather than the oldest Incremental into the Full).  In this case, any data deleted after you captured your original Incremental would still eventually be purged from the set once the last backup that contains that data gets "merged out", but data contained in the Full backup that was deleted afterward will not ever be deleted from the set, since the Full is static. The only real reason to disable Synthetic Fulls would be if you wanted to replicate your backups somewhere else and having to replicate an updated Full after every backup would be impractical for bandwidth reasons.  But if that scenario applies to you, it can be a good idea to manually capture a new Full every now and then, even though that's technically not "Incrementals Forever" anymore.

Finally, note that with either strategy, the file size of the root Full (or oldest Incremental, if Synthetic Fulls are disabled) will always be the size of the backup it contains plus the size of the largest backup that has ever been consolidated into it.  There are technical reasons for this, but I mention it because in the scenario I described above, technically after that first merge operation you would see the file size of your root Full backup grow rather than shrink by ~10GB (depending on compression), even though the backup it contains is now smaller -- but that's only because it was the first merge operation.  The root Full (or oldest Inc) will not keep growing like that forever, unless of course the amount of data you're backing up continues to grow

Incidentally, Macrium has some nice animated demos of both of these variants:
Synthetic Fulls enabled -- https://www.youtube.com/watch?v=6aeWSmT2TP0
Synthetic Fulls disabled -- https://www.youtube.com/watch?v=1hhne4HHTKs

Sorry for the late reply - I'm just getting back to this.  Thank you for your very clear and concise response - exactly answering my question.
Hendrick99
Hendrick99
Junior Member
Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)Junior Member (54 reputation)
Group: Forum Members
Posts: 40, Visits: 191
@falexc

As I understand it, the index section on the incremental file is leading. I think the index is a sort of 'photo' reflecting the content of the partitons
at the time of the backup. This index is especially important during merge operations and also during restore jobs. So when user files get
deleted between backup jobs, it is important to know on which incremental the index will show those deleted files. You can use the standard
Windows Explorer to browse through the backup files.
This is one reason why I use Incremental Merge in stead of Synthetic Full. I would prefer the baseline to be untouched.

Good Luck!
 
Hendrick / Amsterdam
Edited 19 November 2017 9:30 AM by Hendrick99
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)
Group: Forum Members
Posts: 2.8K, Visits: 19K
Hendrick99 - 19 November 2017 7:16 AM
@falexc

As I understand it, the index section on the incremental file is leading. I think the index is a sort of 'photo' reflecting the content of the partitons
at the time of the backup. This index is especially important during merge operations and also during restore jobs. So when user files get
deleted between backup jobs, it is important to know on which incremental the index will show those deleted files. You can use the standard
Windows Explorer to browse through the backup files.
This is one reason why I use Incremental Merge in stead of Synthetic Full. I would prefer the baseline to be untouched.

Good Luck!
 
Hendrick / Amsterdam

If you use Incremental Merge rather than Synthetic Fulls, then as mentioned above (and for the reasons mentioned above) it's generally wise to capture a new Full manually every now and then, although it isn't technically required.  However, there's actually a case to be made that touching the baseline is better.  Regular merging into the root Full can cause corruption introduced after the Full was originally captured to be detected.  There was a thread here recently where a user's Synthetic Full consolidation operations started to fail with an error that the Full was corrupt.  Macrium says that their consolidation process wouldn't have caused that issue, so the theory was that the hard drive storing the image experienced a problem at either the hardware or file system level that rendered part of that Full corrupt or unreadable.  That user was able to resolve the issue by simply capturing a new Full (although all existing backups became worthless), but if that user had NOT been running Synthetic Fulls, then the Full never would have been checked automatically, so the user could conceivably have gone on forever without with backups without realizing that they were all useless, and perhaps not discovered that until they needed to recover their system after a major incident.

The above is a risk of strategies that only have one backup set, such as Incrementals Forever with or without Synthetic Fulls, and it's why I don't recommend using that type of strategy unless you are also replicating those backups elsewhere or using a disk rotation, since in either case you now have more than one set.  Or at the very least, it would be wise to perform manual verifications of all parent backups in the set periodically, but that is tedious and time-consuming task, and it still doesn't rule out the possibility of corruption being introduced between manual verifications, assuming the user is even diligent about performing them.

Edited 20 November 2017 7:06 PM by jphughan
GTK48
GTK48
New Member
New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)
Group: Forum Members
Posts: 32, Visits: 170
With todays update I can no longer remover old backups with MIG turned off


jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)Most Valuable Professional (3.9K reputation)
Group: Forum Members
Posts: 2.8K, Visits: 19K
Did you restart your PC after updating?

GTK48
GTK48
New Member
New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)
Group: Forum Members
Posts: 32, Visits: 170
jphughan - 20 November 2017 8:36 PM
Did you restart your PC after updating?

Yes I did

GTK48
GTK48
New Member
New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)New Member (41 reputation)
Group: Forum Members
Posts: 32, Visits: 170
Now I cannot even do a repair

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search