This a copy of my recent Full and Incremental Image Backups as they appear on my Backup Drive J:


1. The 4th entry 8E455.... is a full Backup made on Sunday, Aug. 4th . My image Backups are scheduled as FULL Backup every Sunday with Incremental Backups Mon - Sat.
     I see no incrementals below it.
2. The 5th entry 65607.... is a Full Backup made the following Sunday, as per schedule. Only 2 Incrementals to this Full Backup follow next as per schedule.
3. Rather than continuing with the remaining incrementals for the 65607.... Full Image Backup, Incrementals for the previous 8E455.... were created on 3 successive days, all  using      the same  date as the last incremental backup of the last 65607... incremental.
4. Can anyone tell me what's going on?

Short version: With Incremental backups, the Date Modified timestamp will not always be the same as the date the backup was actually created.  If you want to see the date of the actual backup data that it contains, right-click the backup file, select Properties, and go to the Macrium Reflect tab to check the Backup Time field.  Alternatively, look at your backups within Reflect itself under the Restore tab, where the Date field will be displayed all the time and will correspond to the actual backup date.

Long version: The Date Modified timestamp is being changed due to Incremental consolidation.  Let's say you have a schedule to create Incrementals Mon-Sat as you do, but your retention policy specifies to retain only 3 Incrementals.  Incrementals #1-3 runs on Monday-Wednesday, respectively. Then Incremental #4 runs on Thursday -- but your retention policy specifies to retain only 3 Incrementals.  At this point, what happens depends on whether you have any Differentials and whether you have "Create a Synthetic Full if possible" checked, but for the purposes of this explanation I'll assume that neither of those is the case.  After that fourth Incremental is created, the oldest Incremental has to go because of the retention policy -- but because of the nature of Incremental backups, you can't just delete the oldest Incremental, because that would invalidate all subsequent backups.  So instead, Incremental #1 and Incremental #2 get "consolidated".  You will see a note to this effect in the job log for Incremental #4, either at the beginning or end of the log, depending on whether you have "Run purge before backup" checked or unchecked.  The end result of the consolidation is that Incremental #1 from Monday is gone, with Incremental #2 from Tuesday now being the oldest remaining backup, but the backup set is still intact.  But because the consolidation counts as a file modification as far as Windows is concerned, the Date Modified timestamp of Incremental #2 will now correspond to either the beginning or end of the time that Incremental #4 was created, because that's when this consolidation operation happened -- even though the data it stores is still the data from the time Incremental #2 was originally created.  And if you have Delta Incremental Indexing disabled (it's disabled by default for Reflect V6 and enabled by default for V7), then the Date Modified timestamps of all subsequent Incrementals in the chain will ALSO be updated to that same date because their indexes will have needed to be updated as a result of consolidating the oldest two Incrementals in the set.

If you have Synthetic Fulls enabled, and there are no Diffs in your set that would prevent a Synthetic Full from being created, then a similar process occurs except that the oldest Incremental gets consolidated with the root Full, thereby "carrying the Full forward in time" rather than the oldest two Incrementals being consolidated into each other.  If you're running Synthetic Fulls, have Delta Incremental Indexing disabled, and have "Run purge before backup" unchecked, you actually end up with every backup in your set having the same Date Modified timestamp date whenever a consolidation occurs.  Based on your screenshot, it looks like you have Synthetic Fulls disabled, Delta Incremental Indexing disabled, and "Run purge before backup" unchecked (or that last Incremental was created extremely quickly)

+x

 

jphughan - 14 August 2019 5:19 PM

Short version: With Incremental backups, the Date Modified timestamp will not always be the same as the date the backup was actually created.  If you want to see the date of the actual backup data that it contains, right-click the backup file, select Properties, and go to the Macrium Reflect tab to check the Backup Time field.  Alternatively, look at your backups within Reflect itself under the Restore tab, where the Date field will be displayed all the time and will correspond to the actual backup date.

Long version: The Date Modified timestamp is being changed due to Incremental consolidation.  Let's say you have a schedule to create Incrementals Mon-Sat as you do, but your retention policy specifies to retain only 3 Incrementals.  Incrementals #1-3 runs on Monday-Wednesday, respectively. Then Incremental #4 runs on Thursday -- but your retention policy specifies to retain only 3 Incrementals.  At this point, what happens depends on whether you have any Differentials and whether you have "Create a Synthetic Full if possible" checked, but for the purposes of this explanation I'll assume that neither of those is the case.  After that fourth Incremental is created, the oldest Incremental has to go because of the retention policy -- but because of the nature of Incremental backups, you can't just delete the oldest Incremental, because that would invalidate all subsequent backups.  So instead, Incremental #1 and Incremental #2 get "consolidated".  You will see a note to this effect in the job log for Incremental #4, either at the beginning or end of the log, depending on whether you have "Run purge before backup" checked or unchecked.  The end result of the consolidation is that Incremental #1 from Monday is gone, with Incremental #2 from Tuesday now being the oldest remaining backup, but the backup set is still intact.  But because the consolidation counts as a file modification as far as Windows is concerned, the Date Modified timestamp of Incremental #2 will now correspond to either the beginning or end of the time that Incremental #4 was created, because that's when this consolidation operation happened -- even though the data it stores is still the data from the time Incremental #2 was originally created.  And if you have Delta Incremental Indexing disabled (it's disabled by default for Reflect V6 and enabled by default for V7), then the Date Modified timestamps of all subsequent Incrementals in the chain will ALSO be updated to that same date because their indexes will have needed to be updated as a result of consolidating the oldest two Incrementals in the set.

If you have Synthetic Fulls enabled, and there are no Diffs in your set that would prevent a Synthetic Full from being created, then a similar process occurs except that the oldest Incremental gets consolidated with the root Full, thereby "carrying the Full forward in time" rather than the oldest two Incrementals being consolidated into each other.  If you're running Synthetic Fulls, have Delta Incremental Indexing disabled, and have "Run purge before backup" unchecked, you actually end up with every backup in your set having the same Date Modified timestamp date whenever a consolidation occurs.  Based on your screenshot, it looks like you have Synthetic Fulls disabled, Delta Incremental Indexing disabled, and "Run purge before backup" unchecked (or that last Incremental was created extremely quickly)

+x

 

jphughan - 14 August 2019 5:19 PM

Short version: With Incremental backups, the Date Modified timestamp will not always be the same as the date the backup was actually created.  If you want to see the date of the actual backup data that it contains, right-click the backup file, select Properties, and go to the Macrium Reflect tab to check the Backup Time field.  Alternatively, look at your backups within Reflect itself under the Restore tab, where the Date field will be displayed all the time and will correspond to the actual backup date.

Long version: The Date Modified timestamp is being changed due to Incremental consolidation.  Let's say you have a schedule to create Incrementals Mon-Sat as you do, but your retention policy specifies to retain only 3 Incrementals.  Incrementals #1-3 runs on Monday-Wednesday, respectively. Then Incremental #4 runs on Thursday -- but your retention policy specifies to retain only 3 Incrementals.  At this point, what happens depends on whether you have any Differentials and whether you have "Create a Synthetic Full if possible" checked, but for the purposes of this explanation I'll assume that neither of those is the case.  After that fourth Incremental is created, the oldest Incremental has to go because of the retention policy -- but because of the nature of Incremental backups, you can't just delete the oldest Incremental, because that would invalidate all subsequent backups.  So instead, Incremental #1 and Incremental #2 get "consolidated".  You will see a note to this effect in the job log for Incremental #4, either at the beginning or end of the log, depending on whether you have "Run purge before backup" checked or unchecked.  The end result of the consolidation is that Incremental #1 from Monday is gone, with Incremental #2 from Tuesday now being the oldest remaining backup, but the backup set is still intact.  But because the consolidation counts as a file modification as far as Windows is concerned, the Date Modified timestamp of Incremental #2 will now correspond to either the beginning or end of the time that Incremental #4 was created, because that's when this consolidation operation happened -- even though the data it stores is still the data from the time Incremental #2 was originally created.  And if you have Delta Incremental Indexing disabled (it's disabled by default for Reflect V6 and enabled by default for V7), then the Date Modified timestamps of all subsequent Incrementals in the chain will ALSO be updated to that same date because their indexes will have needed to be updated as a result of consolidating the oldest two Incrementals in the set.

If you have Synthetic Fulls enabled, and there are no Diffs in your set that would prevent a Synthetic Full from being created, then a similar process occurs except that the oldest Incremental gets consolidated with the root Full, thereby "carrying the Full forward in time" rather than the oldest two Incrementals being consolidated into each other.  If you're running Synthetic Fulls, have Delta Incremental Indexing disabled, and have "Run purge before backup" unchecked, you actually end up with every backup in your set having the same Date Modified timestamp date whenever a consolidation occurs.  Based on your screenshot, it looks like you have Synthetic Fulls disabled, Delta Incremental Indexing disabled, and "Run purge before backup" unchecked (or that last Incremental was created extremely quickly)

Wow! You really gave me an education. Your long version was quite complete and explicit. Thank you.
I was able to follow and understand why the sequence of image backups appeared to be juxtaposed at first but I was unaware of incremental consolidation.That explained a lot.

One thing, however, still is not clear. My retention rules indicate that 6 incrementals should be retained with each Full image (corresponding to the 6 days of the week when incrementals are created). But the 60657... Full image only lists 2 incrementals (cosolidation??). If so, why wouldn't all 6 incrementals be listed since there didn't appear to have any need to consolidate at this point.
Similarly, why would the missing incrementals belonging to the 8E455...image appear now?

If all your FULL backups are of the same disk geometry/subsystem, the Incrementals will consolidate across the Full boundary.  If you want all the Incs produced between Fulls, you should unCheck the Inc retention and let them die when the Full dies.  Otherwise you need to specify how many Incs you want total at any given time (Days or Backups).

+x

 

MORTON NEWMAN - 14 August 2019 9:32 PM

Wow! You really gave me an education. Your long version was quite complete and explicit. Thank you.
I was able to follow and understand why the sequence of image backups appeared to be juxtaposed at first but I was unaware of incremental consolidation.That explained a lot.

One thing, however, still is not clear. My retention rules indicate that 6 incrementals should be retained with each Full image (corresponding to the 6 days of the week when incrementals are created). But the 60657... Full image only lists 2 incrementals (cosolidation??). If so, why wouldn't all 6 incrementals be listed since there didn't appear to have any need to consolidate at this point.
Similarly, why would the missing incrementals belonging to the 8E455...image appear now?

Froggie's answer is correct.  Setting your retention policy to 6 Incrementals does not mean 6 Incrementals per set (i.e. per Full).  It means 6 Incrementals across all "matching" sets (by default, or it would mean 6 Incrementals total in the entire destination folder if you change your retention policy matching setting).  So that means that after a Thursday backup, for example, you'll have 4 Incrementals built from the most recent Full (from Mon-Thurs) and then the Friday and Saturday backups built from the previous week's Full.  If you want to retain 6 Incrementals per set, then Froggie's suggestion to simply disable the Incremental retention policy entirely and then just let the Incrementals get deleted when the parent Full is purged is a good one.  The only catches are that a) you'll need to maintain enough storage to retain that many Incrementals, and b) you'll lose an entire week's worth of Incrementals in one fell swoop, whereas with the current strategy you only lose one at a time, because with your current setup of retaining 6 Incrementals, by the time an old Full is ready to be purged, it won't have any more child Incrementals left.  The other option would be to set your Incremental retention policy to something like 12 or 18.  That would allow you to retain 2-3 weeks' worth of Incementals while still only having them get purged one at a time rather than losing a week's worth when the Full gets purged.  That assumes that your Full will be retained longer than 2-3 weeks, of course.  If the Full retention policy purges a Full that still has child Incrementals, then the Incrementals are gone.  Reflect won't "spare" the Full based on an Incremental retention policy that would otherwise have kept those child Incrementals

+x

 

jphughan - 14 August 2019 9:56 PM

+x

 

MORTON NEWMAN - 14 August 2019 9:32 PM

Wow! You really gave me an education. Your long version was quite complete and explicit. Thank you.
I was able to follow and understand why the sequence of image backups appeared to be juxtaposed at first but I was unaware of incremental consolidation.That explained a lot.

One thing, however, still is not clear. My retention rules indicate that 6 incrementals should be retained with each Full image (corresponding to the 6 days of the week when incrementals are created). But the 60657... Full image only lists 2 incrementals (cosolidation??). If so, why wouldn't all 6 incrementals be listed since there didn't appear to have any need to consolidate at this point.
Similarly, why would the missing incrementals belonging to the 8E455...image appear now?

Froggie's answer is correct.  Setting your retention policy to 6 Incrementals does not mean 6 Incrementals per set (i.e. per Full).  It means 6 Incrementals across all "matching" sets (by default, or it would mean 6 Incrementals total in the entire destination folder if you change your retention policy matching setting).  So that means that after a Thursday backup, for example, you'll have 4 Incrementals built from the most recent Full (from Mon-Thurs) and then the Friday and Saturday backups built from the previous week's Full.  If you want to retain 6 Incrementals per set, then Froggie's suggestion to simply disable the Incremental retention policy entirely and then just let the Incrementals get deleted when the parent Full is purged is a good one.  The only catches are that a) you'll need to maintain enough storage to retain that many Incrementals, and b) you'll lose an entire week's worth of Incrementals in one fell swoop, whereas with the current strategy you only lose one at a time, because with your current setup of retaining 6 Incrementals, by the time an old Full is ready to be purged, it won't have any more child Incrementals left.  The other option would be to set your Incremental retention policy to something like 12 or 18.  That would allow you to retain 2-3 weeks' worth of Incementals while still only having them get purged one at a time rather than losing a week's worth when the Full gets purged.  That assumes that your Full will be retained longer than 2-3 weeks, of course.  If the Full retention policy purges a Full that still has child Incrementals, then the Incrementals are gone.  Reflect won't "spare" the Full based on an Incremental retention policy that would otherwise have kept those child Incrementals

I'm still confused ...It doesn't take much :-)
I've copied my backup drive as of today. It shows a Full backup on 8/18/2019. Below that it lists incrementals for the past 6 days (Mon - Sat).
Tonight, Sunday, it should generate a new Full image with incrementals for the next 6 days thereafter.
My retention is still set for a Full every Sunday with 6 incrementals Mon-Sat). Retain for 5 weeks.
When the next Full image is generated tonight (Sunday), I assume that last week's set (Full + 6 Incrementals shown) will be consolidated into 1 listing under the title of its Full designation (3836642FF7BEF1C3).
After next week's set of images is generated, I suspect that the uppermost (oldest) set 8E4559D... which should consist of a Full with its 6 incrementals consolidated, will be deleted as per my retention plan.
This is my understanding. Please verify or dispute any portion.
If, indeed , I am correct, then if I wish to recover an image in the future, will I only see the Full designation or will all the incrementals which are consolodated be displayed as well?

+x

 

MORTON NEWMAN - 25 August 2019 6:44 PM

I'm still confused ...It doesn't take much :-)
I've copied my backup drive as of today. It shows a Full backup on 8/18/2019. Below that it lists incrementals for the past 6 days (Mon - Sat).
Tonight, Sunday, it should generate a new Full image with incrementals for the next 6 days thereafter.
My retention is still set for a Full every Sunday with 6 incrementals Mon-Sat). Retain for 5 weeks.
When the next Full image is generated tonight (Sunday), I assume that last week's set (Full + 6 Incrementals shown) will be consolidated into 1 listing under the title of its Full designation (3836642FF7BEF1C3).
After next week's set of images is generated, I suspect that the uppermost (oldest) set 8E4559D... which should consist of a Full with its 6 incrementals consolidated, will be deleted as per my retention plan.
This is my understanding. Please verify or dispute any portion.
If, indeed , I am correct, then if I wish to recover an image in the future, will I only see the Full designation or will all the incrementals which are consolodated be displayed as well?

Tonight's Full will NOT cause all of last week's backups to be consolidated into a single file.  I'm not sure why you would assume that, because that would violate your retention policy of keeping 6 Incrementals.  In addition, your older Full backups like 8E4559D are just Full backups.  They are not "a Full with its 6 Incrementals consolidated".  That isn't even a thing.  It sounds like you're not understanding what consolidation is, so I'll try to explain a bit here.

First of all, be aware that if your retention policy specifies to retain 6 Incrementals, that does NOT mean "6 Incrementals per Full".  It means "6 Incrementals across all matching backups in that folder".  To avoid going down another rabbit hole, for the purposes of this reply, think of a matching backup as any backup files generated by the same job.  The reason this distinction exists is because some people might have backups generated by completely different jobs all stored in the same folder.

Anyhow, tonight (Sunday) you'll get a Full.  Whether or not the oldest Full that currently exists will get purged depends on your Full retention policy, which you did not specify.  Then on Monday, you'll get a new Incremental, and then in order for there to only be 6 Incrementals total, Reflect will either consolidate the oldest two Incrementals together (if you have Synthetic Fulls disabled) or will consolidate the oldest Incremental and its parent Full (if you have Synthetic Fulls enabled).  This behavior of getting one new Incremental under the latest Full and losing the oldest Incremental on the previous Full will continue throughout the week.

When backups are consolidated, the older backup is eliminated, so you wouldn't be able to see or restore from it anymore.  So for example let's say you have your Sunday Full and then Incrementals from Monday and Tuesday (plus newer backups) and Reflect needs to eliminate one Incremental to comply with your retention policy.  If you have Synthetic Fulls disabled, then the Monday and Tuesday Incrementals, as the oldest and second oldest, are consolidated together, and after that the Monday Incremental no longer exists as a separate entity that can be restored from.  Macrium has a nice animation of this concept here.  If you have Synthetic Fulls enabled, then the Full and Monday Incremental get consolidated together.  In this case, the Monday Incremental disappears, and your FULL will now be dated Monday because it is a Full that now reflects the state of your source data as of Monday, even though you never actually captured a Full on Monday -- hence the name "Synthetic Full".  Macrium has a nice animation of THIS concept here.  The sole purpose of consolidation is to allow older Incrementals to be purged without invalidating subsequent backups.  It basically involves preserving only the data from that backup that's necessary to keep the REMAINING backups in the chain usable.  But the backup that was actually consolidated is gone afterward.  It's not just stored inside in its entirety inside a Full and kept there to be used.  Having 6 Incrementals plus a bunch more Incrementals "hiding" inside a Full would violate your retention policy.

And as I already mentioned above, I would really recommend that you look at your backups using the Restore tab within Reflect rather than within Windows Explorer, especially if you're already a bit confused.  Being able to sort by Backup Date rather than Date Modified should help make things clearer, or at least avoid confusing you further.  And then you might consider checking Reflect each day to see what backups you still have after the previous night's backup ran in order to get a sense of what's happening

+x

 

jphughan - 25 August 2019 7:07 PM

+x

 

MORTON NEWMAN - 25 August 2019 6:44 PM

I'm still confused ...It doesn't take much :-)
I've copied my backup drive as of today. It shows a Full backup on 8/18/2019. Below that it lists incrementals for the past 6 days (Mon - Sat).
Tonight, Sunday, it should generate a new Full image with incrementals for the next 6 days thereafter.
My retention is still set for a Full every Sunday with 6 incrementals Mon-Sat). Retain for 5 weeks.
When the next Full image is generated tonight (Sunday), I assume that last week's set (Full + 6 Incrementals shown) will be consolidated into 1 listing under the title of its Full designation (3836642FF7BEF1C3).
After next week's set of images is generated, I suspect that the uppermost (oldest) set 8E4559D... which should consist of a Full with its 6 incrementals consolidated, will be deleted as per my retention plan.
This is my understanding. Please verify or dispute any portion.
If, indeed , I am correct, then if I wish to recover an image in the future, will I only see the Full designation or will all the incrementals which are consolodated be displayed as well?

Tonight's Full will NOT cause all of last week's backups to be consolidated into a single file.  I'm not sure why you would assume that, because that would violate your retention policy of keeping 6 Incrementals.  In addition, your older Full backups like 8E4559D are just Full backups.  They are not "a Full with its 6 Incrementals consolidated".  That isn't even a thing.  It sounds like you're not understanding what consolidation is, so I'll try to explain a bit here.

First of all, be aware that if your retention policy specifies to retain 6 Incrementals, that does NOT mean "6 Incrementals per Full".  It means "6 Incrementals across all matching backups in that folder".  To avoid going down another rabbit hole, for the purposes of this reply, think of a matching backup as any backup files generated by the same job.  The reason this distinction exists is because some people might have backups generated by completely different jobs all stored in the same folder.

Anyhow, tonight (Sunday) you'll get a Full.  Whether or not the oldest Full that currently exists will get purged depends on your Full retention policy, which you did not specify.  Then on Monday, you'll get a new Incremental, and then in order for there to only be 6 Incrementals total, Reflect will either consolidate the oldest two Incrementals together (if you have Synthetic Fulls disabled) or will consolidate the oldest Incremental and its parent Full (if you have Synthetic Fulls enabled).  This behavior of getting one new Incremental under the latest Full and losing the oldest Incremental on the previous Full will continue throughout the week.

When backups are consolidated, the older backup is eliminated, so you wouldn't be able to see or restore from it anymore.  So for example let's say you have your Sunday Full and then Incrementals from Monday and Tuesday (plus newer backups) and Reflect needs to eliminate one Incremental to comply with your retention policy.  If you have Synthetic Fulls disabled, then the Monday and Tuesday Incrementals, as the oldest and second oldest, are consolidated together, and after that the Monday Incremental no longer exists as a separate entity that can be restored from.  Macrium has a nice animation of this concept here.  If you have Synthetic Fulls enabled, then the Full and Monday Incremental get consolidated together.  In this case, the Monday Incremental disappears, and your FULL will now be dated Monday because it is a Full that now reflects the state of your source data as of Monday, even though you never actually captured a Full on Monday -- hence the name "Synthetic Full".  Macrium has a nice animation of THIS concept here.  The sole purpose of consolidation is to allow older Incrementals to be purged without invalidating subsequent backups.  It basically involves preserving only the data from that backup that's necessary to keep the REMAINING backups in the chain usable.  But the backup that was actually consolidated is gone afterward.  It's not just stored inside in its entirety inside a Full and kept there to be used.  Having 6 Incrementals plus a bunch more Incrementals "hiding" inside a Full would violate your retention policy.

And as I already mentioned above, I would really recommend that you look at your backups using the Restore tab within Reflect rather than within Windows Explorer, especially if you're already a bit confused.  Being able to sort by Backup Date rather than Date Modified should help make things clearer, or at least avoid confusing you further.  And then you might consider checking Reflect each day to see what backups you still have after the previous night's backup ran in order to get a sense of what's happening

Your attempts to "simplify" the concept of retention rules is sincerely appreciated. Perhaps in my younger days I would be able to follow and appreciate what you have designed into the  program. Unfortunately, it seems that my aging brain can no longer sort through most of it in spite of your efforts.

What I have been attempting to do is to set up a system which produces a Full backup each Sunday with separate individual Incremental backups each day for the next 6 days. This sequence, hopefully will be replicated each week, retaining the previous sets of Full plus all 6 incrementals from the previous week. Then, at the end of the 5th week of retaining these sets, when the 6th Sunday begins the next set, the oldest set of Full, together with its 6 incrementals, would be deleted. Thus I would always have the latest 5 sets of Full images with their 6 incrementals retained.

Any suggestions for producing this result would be welcome and appreciated.
Thank you!

Ok, focusing on what you want to achieve rather than what you've got configured and what you're seeing is indeed simpler.  Based on your post, it sounds like you don't want to retain just 6 Incrementals.  You want to retain 6 Incremetnals per Full, i.e. Incrementals created from a given Full should never be deleted until the Full itself is deleted.  In that case, the easiest way to achieve that is to simply disable the Incremental retention policy entirely, which you can do by unchecking it.  In that setup, Reflect will allow them to continue accumulating until their parent backup is deleted (in this case your Sunday Full), at which point they'll all be deleted together.  Another advantage to this setup is that consolidation won't ever occur, which means the Incremental jobs will be faster.

The only thing to be aware of is that on Sunday when your Full backup runs, you'll gain only one new backup for that Sunday, but you'll lose an entire week's worth of backups (the oldest Full and the entire week's worth of its Incrementals).  As a result, the length of "backup history" you have will vary depending on where you are in the week.  Right before your Sunday Full backup, you'll have 5 weeks' worth of backups stored, but after that backup, you'll only have 4 weeks' worth of backups left because again, you'll have just lost a week's worth of backups when you created that one new Full.  By comparison, with the consolidation strategy, you only lose one old backup for each new backup you create, so this variable history doesn't happen -- but the need for consolidation in this strategy means your Incrementals will take longer.  So there are pros and cons to each strategy; you just have to decide which you prefer.  And if you choose the former, make sure you're ok with sometimes having only 4 weeks' worth of backups.