Macrium Reflect 7.3.5555 miscounting Differential image backups


Author
Message
EatableFever2
EatableFever2
New Member
New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)
Group: Forum Members
Posts: 7, Visits: 10
Macrium Reflect Licensed and paid for version. 7.3.5555
Windows 10 20H2 build 19042.804
Lenovo Laptop.

Source is an Optane enabled Western Digital 1TB Hard disk.
Destination is a USB3 attached Seagate Expansion 1 TB Hard Disk.

I ran a Full image of all 4 Windows partitions.
746 GB in 9hours 15 mins.
   Saving Index
New File: 746 GB    AE57A3DE604842BA-00-00.mrimg
Verifying 'AE57A3DE604842BA-00-00.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    0 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    0 found
I/O Performance:    Read 261.9 Mb/s - Write 912.0 Mb/s
    Image and Verification Completed Successfully in 09:15:14

I made a few updates, mostly new MP3s from iTunes.
I ran a Differential image.  1GB in 4 minutes or so.
Saving Index
New File: 1 GB    AE57A3DE604842BA-01-01.mrimg
Verifying 'AE57A3DE604842BA-01-01.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    1 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    0 found
I/O Performance:    Read 103.8 Mb/s - Write 700.1 Mb/s
    Image and Verification Completed Successfully in 00:04:38

NOTE - MACRIUM HAS FOUND A DIFFERENTIAL image.

I installed KB4023057 and KB4577586 neither of which required a restart.

I ran an Incremental image.  452MB in 3 mins.

Saving Index
New File: 452 MB    AE57A3DE604842BA-02-02.mrimg
Verifying 'AE57A3DE604842BA-02-02.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    0 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    1 found
I/O Performance:    Read 134.1 Mb/s - Write 515.7 Mb/s
    Image and Verification Completed Successfully in 00:03:03

Where is my Differential Backup?  It was there when the Differential ended, but after the Incremental it isn't listed.  I have run 3 further incrementals and they all don't show the Differential.
It is shown by Windows Explorer as a Differential in the "Backup Method' column as are the Full and Incremental backups.

If I go to the restore tab and try to delete the Full backup Macrium correctly prompts me that there are a Differential and 4 Incrementals that will also be deleted.


So, that part of Macrium Reflect recognizes the Differential but when looking at the retention rules it does not!

I have changed the retention rules in the XML file from  "Apply to matching backup sets" to "Apply to all backup sets in the matching folder" and it makes no difference.  There are no other backup sets on the destination, although it has been used for earlier Macrium backups.  These old backups were deleted from inside Macrium to avoid the Image Guardian.

Is this is a bug, or am I missing something?

The HTML versions of the logs are available if required, once I have edited out my license number.

Thankyou.
EatableFever2
EatableFever2
New Member
New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)
Group: Forum Members
Posts: 7, Visits: 10
EatableFever2 - 23 February 2021 3:45 PM
Macrium Reflect Licensed and paid for version. 7.3.5555
Windows 10 20H2 build 19042.804
Lenovo Laptop.

Source is an Optane enabled Western Digital 1TB Hard disk.
Destination is a USB3 attached Seagate Expansion 1 TB Hard Disk.

I ran a Full image of all 4 Windows partitions.
746 GB in 9hours 15 mins.
   Saving Index
New File: 746 GB    AE57A3DE604842BA-00-00.mrimg
Verifying 'AE57A3DE604842BA-00-00.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    0 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    0 found
I/O Performance:    Read 261.9 Mb/s - Write 912.0 Mb/s
    Image and Verification Completed Successfully in 09:15:14

I made a few updates, mostly new MP3s from iTunes.
I ran a Differential image.  1GB in 4 minutes or so.
Saving Index
New File: 1 GB    AE57A3DE604842BA-01-01.mrimg
Verifying 'AE57A3DE604842BA-01-01.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    1 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    0 found
I/O Performance:    Read 103.8 Mb/s - Write 700.1 Mb/s
    Image and Verification Completed Successfully in 00:04:38

NOTE - MACRIUM HAS FOUND A DIFFERENTIAL image.

I installed KB4023057 and KB4577586 neither of which required a restart.

I ran an Incremental image.  452MB in 3 mins.

Saving Index
New File: 452 MB    AE57A3DE604842BA-02-02.mrimg
Verifying 'AE57A3DE604842BA-02-02.mrimg'
Ok. Continuing...
Retention Rules
Rules will be applied to all matching backup sets in the destination folder
Full:    Retain 1 full images
Linked incremental and differential images will also be deleted
Backup Sets:    1 sets found
Nothing to delete
Differential:    Retain 2 differential images
Linked incremental images will also be deleted
Differential Backups:    0 found
Nothing to delete
Incremental:    Retain 10 incremental images
The oldest incremental images may be consolidated
Incremental Backups:    1 found
I/O Performance:    Read 134.1 Mb/s - Write 515.7 Mb/s
    Image and Verification Completed Successfully in 00:03:03

Where is my Differential Backup?  It was there when the Differential ended, but after the Incremental it isn't listed.  I have run 3 further incrementals and they all don't show the Differential.
It is shown by Windows Explorer as a Differential in the "Backup Method' column as are the Full and Incremental backups.

If I go to the restore tab and try to delete the Full backup Macrium correctly prompts me that there are a Differential and 4 Incrementals that will also be deleted.


So, that part of Macrium Reflect recognizes the Differential but when looking at the retention rules it does not!

I have changed the retention rules in the XML file from  "Apply to matching backup sets" to "Apply to all backup sets in the matching folder" and it makes no difference.  There are no other backup sets on the destination, although it has been used for earlier Macrium backups.  These old backups were deleted from inside Macrium to avoid the Image Guardian.

Is this is a bug, or am I missing something?

The HTML versions of the logs are available if required, once I have edited out my license number.

Thankyou.

https://1drv.ms/u/s!Ai2JEB1xZewtgoIyCZAb7sWKwYX3oQ?e=bo8bur

should be a link to my Onedrive compressed folder of the Full Differential and Incremental logs showing what Macrium Reflect recorded and what the delete side of things shows.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
I reproduced this and am narrowing down the problem conditions....

EatableFever2
EatableFever2
New Member
New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)
Group: Forum Members
Posts: 7, Visits: 10
jphughan - 23 February 2021 5:12 PM
I reproduced this and am narrowing down the problem conditions....

Thankyou - so it wasn't my imagination!
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Ok, @Nick I've got a problem report that builds off the initial report above.

When an Incremental is created as a child of a Differential, the specific parent Differential of that backup is not counted in the number of Diffs found under the retention policy.  So in this scenario, the number of Differentials found, as recorded in the log, will always be N-1 relative to the actual number of matching Differentials.  When creating Fulls, other Diffs, or any Incs that are NOT children of a Differential (such as Incs created as direct children of a Full when there's also an older matching Full that has a Diff), the Differentials are counted correctly.

@EatableFever2 It's possible that this is a deliberate miscount, because the parent Diff of the backup you just created should never be considered for purging under the retention policy.  If it were, then Reflect might delete it, and if it did that, then it would also end up deleting the Incremental backup it had just created as a child of that Differential.  So in this case, the number of backups "found" and the number of backups "sensibly eligible for retention policy purging" are not the same.

Edited 23 February 2021 5:48 PM by jphughan
Nick
Nick
Macrium Representative
Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)Macrium Representative (4.1K reputation)
Group: Administrators
Posts: 2.3K, Visits: 14K
jphughan - 23 February 2021 5:41 PM
Ok, @Nick I've got a problem report that builds off the initial report above.

When an Incremental is created as a child of a Differential, the specific parent Differential of that backup is not counted in the number of Diffs found under the retention policy.  So in this scenario, the number of Differentials found, as recorded in the log, will always be N-1 relative to the actual number of matching Differentials.  When creating Fulls, other Diffs, or any Incs that are NOT children of a Differential (such as Incs created as direct children of a Full when there's also an older matching Full that has a Diff), the Differentials are counted correctly.

@EatableFever2 It's possible that this is a deliberate miscount, because the parent Diff of the backup you just created should never be considered for purging under the retention policy.  If it were, then Reflect might delete it, and if it did that, then it would also end up deleting the Incremental backup it had just created as a child of that Differential.  So in this case, the number of backups "found" and the number of backups "sensibly eligible for retention policy purging" are not the same.

Thanks for the post. As you have noted, the parent Diff is not eligible for deletion because deleting it would delete the current backup therefore it's excluded from the count. This was a change made quite recently.  We need to change the text in the log to include the words 'eligible for deletion' to avoid confusion, or perhaps include the non-eligible parent in the displayed count. 

Kind Regards

Nick - Macrium Support

Next Webinar


EatableFever2
EatableFever2
New Member
New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)
Group: Forum Members
Posts: 7, Visits: 10
jphughan - 23 February 2021 5:41 PM
Ok, @Nick I've got a problem report that builds off the initial report above.

When an Incremental is created as a child of a Differential, the specific parent Differential of that backup is not counted in the number of Diffs found under the retention policy.  So the number of Differentials found as recorded in the log in this scenario will always be N-1 relative to the actual number of matching Differentials.  When creating Fulls, Diffs, or Incs that are NOT children of Differentials (such as direct children of a separate Full where a previous matching Full has a Diff), the Differentials are counted correctly.

@EatableFever2 It's possible that this is a deliberate miscount, because the parent Diff of the backup you just created should never be considered for purging under the retention policy.  If it were, then Reflect might delete it, and if it did that, then it would also end up deleting the Incremental backup it had just created as a child of that Differential.  So in this case, the number of backups "found" and the number of backups "sensibly eligible for retention policy purging" are not the same.

Thankyou for your reply.  I have to think about the logic of retention.

If my retention policy for Differentials is 1 then creating an Incremental as a child of that one Differential isn't a problem. No Differential will be deleted by the retention policy so the newly created Incremental isn't affected.  If I then create a 2nd Differential the retention policy can quite safely delete the 1st Differential and any Incremental children.
If my retention policy for Differentials is 2 then creating a third Differential should only result in the deletion of the 1st Differential and it's Incremental children.
To summarise, if my retention policy for Differentials is n, then creating the n+1 Differential can safely result in the deletion of the earliest Differential and its Incremental children.
 
I don't think the Differential parent of a newly created Incremental can EVER be deleted by the retention policy.

I think the 'deliberate miscounting' is confusing to say the least.  The retention policy analysis at the end of a backup should truly reflect the state of my backup images.

I have an email notification from you of an update to this post as I type.  I'll post this reply and read your reply.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
You're correct that it would never make sense to delete the newly created Incremental's parent Differential, since that would result in the deletion of the backup you had just created.  That behavior actually existed until recently, and it was found and reported as a bug by yours truly over in this thread.  Macrium updated Reflect to ensure that the parent Diff was always excluded from consideration for retention policy purging, even in cases like Scenario #2 that I described in that linked thread where doing so would technically violate the retention policy, since that seems preferable to a scenario where Reflect would immediately delete a backup that it had just created.  But as Nick said above, the update that added the new logic for this scenario did not take into account the impact on the language of the log.  But as Nick also just said above, they're going to fix that. Smile  If you're new to Macrium, the Macrium reps on this forum are actual developers, not minimally trained customer support reps as you might find elsewhere. Nick in addition to being a developer is also Macrium's founder and CEO.  That level of engagement here means that Macrium often responds very quickly to bug reports and even some enhancement requests here.  I personally have seen several of my own bug reports and enhancement suggestions over the years result in Reflect updates to fix/implement them, and those updates often don't take very long to arrive if the bug is well described or the enhancement would not take very much effort to implement.

As to the best practice question of your retention policy, that would probably depend on how often you plan to create Differentials.  As you say, retaining only 1 Differential means that when you create a new one, the previous Diff and all of its child Incs will be deleted in one fell swoop.  So for example if you're only creating one Diff per month, you might not want a situation where creating one new Diff causes you to lose a month's worth of backups all at once, leaving you with only your brand new Diff and its parent Full.  That's a sharp drop in the number of backups you have available.

EatableFever2
EatableFever2
New Member
New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)New Member (8 reputation)
Group: Forum Members
Posts: 7, Visits: 10
EatableFever2 - 23 February 2021 6:17 PM
jphughan - 23 February 2021 5:41 PM
Ok, @Nick I've got a problem report that builds off the initial report above.

When an Incremental is created as a child of a Differential, the specific parent Differential of that backup is not counted in the number of Diffs found under the retention policy.  So the number of Differentials found as recorded in the log in this scenario will always be N-1 relative to the actual number of matching Differentials.  When creating Fulls, Diffs, or Incs that are NOT children of Differentials (such as direct children of a separate Full where a previous matching Full has a Diff), the Differentials are counted correctly.

@EatableFever2 It's possible that this is a deliberate miscount, because the parent Diff of the backup you just created should never be considered for purging under the retention policy.  If it were, then Reflect might delete it, and if it did that, then it would also end up deleting the Incremental backup it had just created as a child of that Differential.  So in this case, the number of backups "found" and the number of backups "sensibly eligible for retention policy purging" are not the same.

Thankyou for your reply.  I have to think about the logic of retention.

If my retention policy for Differentials is 1 then creating an Incremental as a child of that one Differential isn't a problem. No Differential will be deleted by the retention policy so the newly created Incremental isn't affected.  If I then create a 2nd Differential the retention policy can quite safely delete the 1st Differential and any Incremental children.
If my retention policy for Differentials is 2 then creating a third Differential should only result in the deletion of the 1st Differential and it's Incremental children.
To summarise, if my retention policy for Differentials is n, then creating the n+1 Differential can safely result in the deletion of the earliest Differential and its Incremental children.
 
I don't think the Differential parent of a newly created Incremental can EVER be deleted by the retention policy.

I think the 'deliberate miscounting' is confusing to say the least.  The retention policy analysis at the end of a backup should truly reflect the state of my backup images.

I have an email notification from you of an update to this post as I type.  I'll post this reply and read your reply.

Hi Nick,
 I can't envisage a scenario where the Differential policy would try and delete the parent Differential of the Incremental I have just created. I am always creating an Incremental as a child of the parent Differential (or Full). If my Differential (or Full) retention policy is 1 or greater then my Incremental will always have a parent.  

If my Differential retention policy was1, but I amended my XML file to a retention policy of 0 then that would be another matter.  Does the Macrium wizard for creating the XML file disallow 0?  I could edit the XML file outside of Macrium (using Notepad say).

I've just used Notepad to look at my XML file and I see retention can be based on days or weeks rather than just number of files.  If my retention for Differential was, say 2 days and I hadn't done a backup for 3 days (or more) then you definitely wouldn't want the Differential parent of the Incremental just created to be deleted.  In that case the Differential shouldn't be considered eligible for deletion.
However, if the Differential retention policy is specified as a number of files (greater than 0) then the Differential parent of my newly created Incremental can never be deleted by the retention policy. So, under that circumstance I see no reason to not count it in the retention policy analysis at the end of a backup.

The situation is definitely if the retention policy is specified in days or weeks. In those circumstances the parent Differential of a newly created Incremental backup can not be deleted until a new Differential is created.

The logic is complicated - more so than I thought as I wasn't aware retention could be in days or weeks.

I have a new reply from @jphughan so I'll post this and read the reply.


jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
@EatableFever2 Everyone in here is in agreement.  The parent Diff of the current Incremental shouldn't be considered for deletion, since there's no point in having Reflect create a backup and then delete it immediately thereafter.  For the same reason, the current backup's parent Full should also not be considered for deletion, and Macrium has already ensured that.  But when I submitted the bug report that I linked above, there WERE circumstances where the current backup's parent Diff could be considered for deletion -- hence my bug report.  Macrium fixed that, since that was clearly not desirable behavior.  But that recent fix to exclude the current backup's parent Diff from retention policy consideration appears to have had the side effect of the log no longer being precisely accurate in its reporting of how many Differential backups were "found".  Nick has already acknowledged this side effect of that recently implemented fix, and it seems ANOTHER fix for this reporting anomaly will be forthcoming, which will continue to exclude the current backup's parent Diff from retention policy consideration but clarify the wording in the log.  But in the meantime, Reflect will still operate as expected/desired from a practical standpoint.  The only impact is a wording anomaly in the log that does not affect how backups will be or should be handled under your configured retention policy.

Edited 23 February 2021 7:11 PM by jphughan
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search