Will Incremental Forever create a Synthetic Full from old Full and Differential?


Author
Message
techiejohn
techiejohn
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 5, Visits: 9
I was using Reflect Free with Full/Differential but now switched to Incremental Forever. The first incremental seems to be just that. Will this create a synthetic full from the old Full and Differential when it's time (e.g. 30 days)? I also enabled CBT.
Edited 2 January 2021 4:08 PM by techiejohn
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)
Group: Forum Members
Posts: 8.4K, Visits: 57K
A Synthetic Full is only created if the only backups in the set are a Full and Incrementals, with no Differentials.  The reason for this restriction is that if you had a Full with multiple child Differentials and then merged one of the Differentials into that Full, you would immediately render all of your other Differentials useless, as well as any Incrementals that were built off of them.  A Differential backup stores a backup state relative to the state of its parent Full.  But that Differential backup is only valid if the Full remains in that same state.  If you subsequently modify the Full, then the remaining Differentials are no longer relative to that new state of the parent Full and therefore can't be used.

If you want to use Synthetic Fulls, you'll need to either create a new Full or delete all of your existing Differentials (and any child Incrementals) so that your existing Full no longer has any Differential that depend on it.

techiejohn
techiejohn
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 5, Visits: 9
jphughan - 2 January 2021 4:14 PM
A Synthetic Full is only created if the only backups in the set are a Full and Incrementals, with no Differentials.  The reason for this restriction is that if you had a Full with multiple child Differentials and then merged one of the Differentials into that Full, you would immediately render all of your other Differentials useless, as well as any Incrementals that were built off of them.  A Differential backup stores a backup state relative to the state of its parent Full.  But that Differential backup is only valid if the Full remains in that same state.  If you subsequently modify the Full, then the remaining Differentials are no longer relative to that new state of the parent Full and therefore can't be used.

If you want to use Synthetic Fulls, you'll need to either create a new Full or delete all of your existing Differentials (and any child Incrementals) so that your existing Full no longer has any Differential that depend on it.

Ok, so... a couple of questions:
1. Was the incremental I created based off the last differential?
2. Should I simply remove all of the differentials (and the incremental if it was based off the differential), but leave the last full?

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)
Group: Forum Members
Posts: 8.4K, Visits: 57K
Incrementals are always created from the most recent backup, whether that's a Full, Diff, or another Incremental.  But if you have a Differential in your set, then yes it's virtually guaranteed that your new Incremental would somehow depend on that Differential.  (If you want to see how your backups are related, go to the Restore tab, select the Incremental, and choose Other Actions > Delete.  In the interface that pops up you'll see a hierarchical view of all backups in that set that illustrates dependencies.)

In terms of whether you should delete your Differentials and all child Incrementals, if you're comfortable losing all of those backups, then go ahead.  That's the most space-efficient way to start using Incrementals Forever.  But if you don't want to lose all of those backup states, you can instead create a new Full to preserve those existing backups for as long as you need to -- assuming you have the storage to do this, of course.

techiejohn
techiejohn
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 5, Visits: 9
Sounds good... I don't need those, so I'll delete all the differentials (which the incremental should be linked to) and keep the full so start the chain from. Thanks.

techiejohn
techiejohn
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 5, Visits: 9
How is the Synthetic full created? Specifically, I'm trying to figure out how efficient it will be. Does it require the read and re-write of the entire full when the incremental is merged? I'm backing up over a wireless network and trying to get better efficiency with the incremental since I have a hard time finishing full backups without some network error.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)
Group: Forum Members
Posts: 8.4K, Visits: 57K
Here's what happens.  You start with a Full that has never "absorbed" an Incremental before.  The first time an Incremental is consolidated into it, the size of the Full grows basically by the amount of the Incremental it's absorbing.  The purpose of this is so that if the consolidation fails partway through, all of the original data blocks of the Full are still there in that file and therefore the Full remains usable rather than becoming corrupted.  Once the Incremental is successfully consolidated into the Full, that becomes the new state of the Full, and any data blocks pertaining to the original state that are no longer needed to support the new state get marked as scratch space.  Those data blocks are therefore eligible to be overwritten during the NEXT consolidation operation.  If the next Incremental you merge doesn't require any more space than the available scratch space, the Full doesn't grow any further.  Otherwise, the Full will use its scratch space during the consolidation and then grow as needed to absorb the next Incremental.  This process goes on.  The Full will never shrink, but it doesn't grow by the amount of every Incremental that gets consolidated into it either.

techiejohn
techiejohn
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 5, Visits: 9
jphughan - 3 January 2021 3:24 PM
Here's what happens.  You start with a Full that has never "absorbed" an Incremental before.  The first time an Incremental is consolidated into it, the size of the Full grows basically by the amount of the Incremental it's absorbing.  The purpose of this is so that if the consolidation fails partway through, all of the original data blocks of the Full are still there in that file and therefore the Full remains usable rather than becoming corrupted.  Once the Incremental is successfully consolidated into the Full, that becomes the new state of the Full, and any data blocks pertaining to the original state that are no longer needed to support the new state get marked as scratch space.  Those data blocks are therefore eligible to be overwritten during the NEXT consolidation operation.  If the next Incremental you merge doesn't require any more space than the available scratch space, the Full doesn't grow any further.  Otherwise, the Full will use its scratch space during the consolidation and then grow as needed to absorb the next Incremental.  This process goes on.  The Full will never shrink, but it doesn't grow by the amount of every Incremental that gets consolidated into it either.

Great thanks, I was hoping it was something like this (which seems similar to sparse disk and bundle on Mac).
Charles Godwin
Charles Godwin
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 4, Visits: 11
I don't know if this is a followup or a new question. If I should make a new question, tell me and I'll delete this and repost.

I am running Full/incremental on 3 home PCs to a local NAS. I do daily incremental with a 30 day retention so the Full gets 'synthetic full backup' after 30 days. I want to add an automated remote duplication of the backup files for added redundancy. I plan to use rsync from my local NAS to a remote system. The daily incrementals are about 3-4 GB which is a tolerable amount. But what about this synthetic full backup that will be changed every day. These are each in the 500GB and larger category. The rsync program is supposed to only update changed blocks in files so the best case scenario would be a daily change about equal to the daily increment, or another 3-4GB daily. so far, so good. But if the synthetic full backup file is so changed that rsync thinks it's new, then I have a big problem as the daily transmission would be over a terabyte and my plan won't be practical.

Comments?


jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)Macrium Evangelist (12K reputation)
Group: Forum Members
Posts: 8.4K, Visits: 57K
Charles Godwin - 12 January 2021 4:12 PM
I don't know if this is a followup or a new question. If I should make a new question, tell me and I'll delete this and repost.

I am running Full/incremental on 3 home PCs to a local NAS. I do daily incremental with a 30 day retention so the Full gets 'synthetic full backup' after 30 days. I want to add an automated remote duplication of the backup files for added redundancy. I plan to use rsync from my local NAS to a remote system. The daily incrementals are about 3-4 GB which is a tolerable amount. But what about this synthetic full backup that will be changed every day. These are each in the 500GB and larger category. The rsync program is supposed to only update changed blocks in files so the best case scenario would be a daily change about equal to the daily increment, or another 3-4GB daily. so far, so good. But if the synthetic full backup file is so changed that rsync thinks it's new, then I have a big problem as the daily transmission would be over a terabyte and my plan won't be practical.

Comments?


Your understanding is correct.  I can't speak to how RSync will respond to the updated Synthetic Full since I haven't used those items together.  But I would say that in general, if you plan to use replication, you may want to consider using Incremental Merge instead, where the oldest two Incrementals are consolidated together rather than the oldest Incremental being consolidated into the Full.  That will keep your Full intact.  Of course that also means it becomes increasingly outdated, so this strategy would require you to create a new Full on some sort of scheduled basis.  However, that's somewhat inevitable anyway under the current setup because Synthetic Full files become increasingly fragmented internally over time, which is NOT resolved by defragmenting the underlying volume.  That fragmentation imposes performance penalties.  I have a client that runs a backup destination disk rotation such that no single disk is seen more often than once per week.  I have a Synthetic Full strategy set up for all of those disks, and each Incremental is about 150 GB.  Early on, an Incremental that takes an hour to be created would take about an hour to be consolidated later on.  After several dozen such consolidations, the fragmentation within the Synthetic Full means that an Incremental that took an hour to create might take 8+ hours to consolidate.  Macrium has acknowledged this issue and has said they're looking at solutions, but I haven't seen anything about that in a while.  So at the moment I'm just manually creating a Full when consolidation times become unreasonable.  The reason it's manual in my case is that formally scheduling a periodic Full when you have a rotation of 9 destination disks gets tricky, even before accounting for human error that causes the wrong disk to be connected on a given day.  But my point is that true "Incrementals Forever" doesn't seem to be feasible today, so you may want to consider scheduling periodic Fulls -- and if you do that, then disabling Synthetic Fulls sidesteps the replication traffic issue you've called out.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search