Incrementals forever count seems wrong


Author
Message
Telos
Telos
New Member
New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)
Group: Forum Members
Posts: 17, Visits: 35
I run a daily incrementals forever... with "create a synthetic full backup if possible checked".

Recently I changed the definition file from 30 incremental images to 25. After that odd things happened. I now have 39 incremental images including the baseline and they are numbered...
00, 26, 27, 35, 36, 39, 41-43, 48-77

I manually ran another incremental... and the log said...

Incremental Backups: 25 found
Consolidation: Merging '6C5E256DA474FE92-48-48.mrimg' to '6C5E256DA474FE92-49-49.mrimg'
Merge completed successfully in 00:00:23
Deleted '6C5E256DA474FE92-49-49.mrimg'
Renamed '6C5E256DA474FE92-48-48.mrimg' to '6C5E256FD474FE92-49-49.mrimg'

So I now have...
00, 26, 27, 35, 36, 39, 41-43, 49-78
So are all the incrementals from 26 to 39 no longer needed? 
And the sequential ones, 49-78 add up to 30 (coincidentally, my original incremenatls setting.

Do I need to start over and toss this chain?
What's happening here?
Nick
Nick
Macrium Representative
Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)Macrium Representative (2.7K reputation)
Group: Administrators
Posts: 1.5K, Visits: 8.5K
Telos - 5 March 2018 4:06 PM
I run a daily incrementals forever... with "create a synthetic full backup if possible checked".

Recently I changed the definition file from 30 incremental images to 25. After that odd things happened. I now have 39 incremental images including the baseline and they are numbered...
00, 26, 27, 35, 36, 39, 41-43, 48-77

I manually ran another incremental... and the log said...

Incremental Backups: 25 found
Consolidation: Merging '6C5E256DA474FE92-48-48.mrimg' to '6C5E256DA474FE92-49-49.mrimg'
Merge completed successfully in 00:00:23
Deleted '6C5E256DA474FE92-49-49.mrimg'
Renamed '6C5E256DA474FE92-48-48.mrimg' to '6C5E256FD474FE92-49-49.mrimg'

So I now have...
00, 26, 27, 35, 36, 39, 41-43, 49-78
So are all the incrementals from 26 to 39 no longer needed? 
And the sequential ones, 49-78 add up to 30 (coincidentally, my original incremenatls setting.

Do I need to start over and toss this chain?
What's happening here?

Thanks for posting.

Do you have a Differential image in the chain? Are all the images in the same backup set '6C5E256DA474FE92'?

If you open a support ticket we can request further info to explain what's happened.

https://www.macrium.com/support


Kind Regards

Nick - Macrium Support

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)
Group: Forum Members
Posts: 3.5K, Visits: 26K
EDIT: Nick beat me to it.

Edited 5 March 2018 4:28 PM by jphughan
Telos
Telos
New Member
New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)
Group: Forum Members
Posts: 17, Visits: 35
Nick - 5 March 2018 4:24 PM
Telos - 5 March 2018 4:06 PM
I run a daily incrementals forever... with "create a synthetic full backup if possible checked".

Recently I changed the definition file from 30 incremental images to 25. After that odd things happened. I now have 39 incremental images including the baseline and they are numbered...
00, 26, 27, 35, 36, 39, 41-43, 48-77

I manually ran another incremental... and the log said...

Incremental Backups: 25 found
Consolidation: Merging '6C5E256DA474FE92-48-48.mrimg' to '6C5E256DA474FE92-49-49.mrimg'
Merge completed successfully in 00:00:23
Deleted '6C5E256DA474FE92-49-49.mrimg'
Renamed '6C5E256DA474FE92-48-48.mrimg' to '6C5E256FD474FE92-49-49.mrimg'

So I now have...
00, 26, 27, 35, 36, 39, 41-43, 49-78
So are all the incrementals from 26 to 39 no longer needed? 
And the sequential ones, 49-78 add up to 30 (coincidentally, my original incremenatls setting.

Do I need to start over and toss this chain?
What's happening here?

Thanks for posting.

Do you have a Differential image in the chain? Are all the images in the same backup set '6C5E256DA474FE92'?

If you open a support ticket we can request further info to explain what's happened.

https://www.macrium.com/support

Thanks for your prompt response.

I think you hit upon it with the differentials... Several weeks ago I added a desktop shortcut to my incrementals forever definition file, in the event I wanted to take a snapshot prior to a software addition. What I didn't realize until now, was that the shortcut was set (my oversight apparently) to "differential". I did notice the larger "incrementals" but attributed them to moving programs between partitions.

So my question now is, how can I discern which of the images in the backup set are differentials... and can I simply delete them without affecting the integrity of the incrementals chain?

EDIT1: OK, I figured out how to check which are differentials vs incrementals. Can I delete the differentials without breaking the incrementals chain integrity?

Edited 5 March 2018 4:51 PM by Telos
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)
Group: Forum Members
Posts: 3.5K, Visits: 26K
Unfortunately if you delete a Differential, any Incrementals built from that Differential will become useless.  It also isn't possible to consolidate a Differential into a Full because if that were allowed, then that operation would render any other Differentials in your set useless because you would have just modified the Full they were all based on, and that would then render the child Incrementals of those Diffs useless as well.  And any Incs that may have been built directly off of that Full would also be ruined.

At this stage it might be easiest to just start over with a new Full and manually delete the older backups when you decide you no longer need them, but if you want to see what deletions ARE possible, the easiest way to do so safely would be to check within Reflect.  Go to the Restore tab (this is also an easy way to identify Full/Diff/Inc for a given backup, by the way), choose any image, click Other Actions, and click Delete.  A popup dialog will appear, and you'll notice that if you select a backup, any child backup that depends on the selected backup will also be selected for deletion.  This accomplishes two things -- it gives you a view into the impact of your planned deletion, and it also prevents "orphaned" backups from lingering on your disk uselessly taking up space.

Edited 5 March 2018 5:18 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)
Group: Forum Members
Posts: 3.5K, Visits: 26K
@Nick, thinking about this a bit especially as a similar issue was just mentioned in another thread, would it be possible to update the standalone Consolidate.exe utility to allow consolidating a Diff into a Full if both of the below conditions are satisfied?

- There is only one Diff in the set.
- Any Incs that exist in the set are children of that Diff, i.e. there are no Incs that are direct children of the Full.

I don't think it would be good to support automatic Synthetic Full creation with Diffs as part of Reflect's regular operations even when the above conditions exist, but having a manually executed tool that supported this would offer a cleanup mechanism for cases like this without having to start over, which may be problematic because users may be forced to balance storage constraints against the desire to retain older backups for a while

Edited 5 March 2018 5:27 PM by jphughan
Telos
Telos
New Member
New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)
Group: Forum Members
Posts: 17, Visits: 35
jphughan - 5 March 2018 5:24 PM
@Nick, thinking about this a bit especially as a similar issue was just mentioned in another thread, would it be possible to update the standalone Consolidate.exe utility to allow consolidating a Diff into a Full if both of the below conditions are satisfied?

- There is only one Diff in the set.
- Any Incs that exist in the set are children of that Diff, i.e. there are no Incs that are direct children of the Full.

I don't think it would be good to support automatic Synthetic Full creation with Diffs as part of Reflect's regular operations even when the above conditions exist, but having a manually executed tool that supported this would offer a cleanup mechanism for cases like this without having to start over, which may be problematic because users may be forced to balance storage constraints against the desire to retain older backups for a while

Thank you. I cleaned up a good bit of dead differentials and created a new full backup. What I found interesting was that when I ran the incrementals forever after creating the new full, that it pruned one of the old chain's incrementals, yet added the latest incremental to the new chain. I didn't realize that would happen. Quite nice. So I can let the old chain slowly dissipate with each new incremental, and delete the differentials when they no longer have incrementals children.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)
Group: Forum Members
Posts: 3.5K, Visits: 26K
Excellent, glad you've been able to find a workaround. Smile  Yes, when Reflect evaluates the retention policy, it counts the totals across all "matching" backups, not just those created from the most recent Full.  The definition of "matching" depends on your retention policy settings.  For image backups, by default "matching" means that backups must contain exactly the same partition set what's contained in the currently running backup.  For F&F, there are other options.  And for either type, the matching policy can be switched to "All backups within the destination folder".  That mode can be handy for Windows 10 users due to behavior described in the first post of this thread that can cause the default retention policy matching policy not to act on older backups, but switching to "All backups" does mean you'll want to dedicate a destination folder specifically to backups created by a given definition file, because if you have multiple definition files sending backups to the same folder, Job A's retention policy could end up purging backups created by Job B.

And incidentally, if you use Incrementals Forever to capture images of a Windows 10 system, you may want to specify a Full backup retention policy of either 1 or 2 in order to account for the behavior described in the thread I linked, since it can cause you to unexpectedly get a new Full, and you therefore may want to have old Fulls automatically purged.  Note that if you set the policy to just 1 Full though, ALL previous backups will be deleted whenever a new Full runs.  That can be especially dangerous if you have "Run purge before backup" enabled, since the system might fail after the purge but before the new backup completes.  If you have a disk rotation, that's less of a risk, and I actually recommend a disk rotation for anyone using Incrementals Forever because there are risks associated with only having a single backup set.  For example, if the drive storing your backups later has some file system or hardware-level issue that renders your Full backup partially corrupt/unreadable, then all of your backups are now unusable because they all depend on that one Full.  That actually happened to someone here.  He was alerted to the problem because his Synthetic Full consolidations started failing, but technically it's possible for the problem to occur before you run another consolidation to have the chance to discover it, and I don't know this for certain, but if the consolidation never had to update data in the damaged/unreadable portion of the Full, maybe even a consolidation operation wouldn't always discover this problem.

At the very least, if you continue running Incrementals Forever without a disk rotation, I would recommend periodically verifying at least your Full backup.  Even if backups were verified when they were originally created, that does not guarantee they are still readable today.  Or consider just using a disk rotation.  Incrementals Forever is the easiest strategy of all to implement one since every job is always an Incremental, and Reflect's optional GUID-based disk discovery option makes it easy to implement multiple backup destinations.  I have a disk rotation of 9 disks all running from the same definition file.

Edited 5 March 2018 8:49 PM by jphughan
Telos
Telos
New Member
New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)New Member (33 reputation)
Group: Forum Members
Posts: 17, Visits: 35
My backup strategy is somewhat simple... I use the incrementals forever chain (nominally 30 days) primarily as a precaution against accidental stupidity. It is automatically taken daily, and manually whenever I'm about to add a software package or make system changes such as driver updates.

Alongside the incrementals forever images, I keep a separate monthly full with weekly differentials. This, along with a monthly USB full and an HDD clone, is my catastrophic backup.

Images from both backup methods are initially stored on a dedicated partition of the main drive. The monthly full/weekly differential image series is then pushed to a server, and then onto cloud storage. So I'm reasonably confident that redundancy is sufficient... maybe even overdone.

Your post has given me a lot to think about in terms of retention policy. I'll need to review this more carefully. Also... I never thought to verify the full after all this time. I am doing that now. Thanks for all that.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)Macrium Evangelist (5K reputation)
Group: Forum Members
Posts: 3.5K, Visits: 26K
You’re welcome! Feel free to ask if you have any other questions, but you definitely have more than one backup set in your strategy, and even better they’re spread across multiple disks, so you’re in good shape in terms of resiliency. If I were in your position with multiple fallbacks, I probably wouldn’t bother with verifying the Full in my Incs Forever set unless maybe it was small enough not to take much time or losing up to a week’s worth of backups (by having to resort to a weekly Diff) would be a major inconvenience, but it certainly can’t hurt!
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search