Incremental Forever (Synthetic Full Backup)


Author
Message
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 8K, Visits: 55K
Ok, this post is going to be a mixture of clarifications in terms of how Reflect and synthetic fulls work, and also some of my own best practice recommendations.

First, the only time Reflect ever generates a synthetic full is when it creates a new Incremental and the resulting number of Incrementals in the chain exceeds the number you've specified in your retention period.  For example, in my "Incrementals Forever with Synthetic Fulls" job, I specify to retain 7 Incrementals.  The first time my job executed, it generated a Full.  Then each of the next 7 times, Incrementals were generated.  The time after that, an 8th Incremental was generated, but since that exceeds my retention policy, at that point Reflect automatically merged Incremental #1 into my Full backup, thereby "carrying the Full forward" so that I have a Full equivalent to having actually captured a Full on the day Incremental #1 ran, even though I didn't actually run a Full that day -- hence the name "synthetic full".  Incremental #1 is deleted after the Full is updated with its contents (thereby keeping my backup set compliant with my retention policy), and of course I lose the ability to restore all the way back to the day I ACTUALLY ran that Full.  From that point on, this process of merging the oldest Incremental into the root Full continues each time a new Incremental is generated, thereby keeping your backup set current without the number of Incrementals ever exceeding the retention policy you specify.  But if you set a retention policy of 30 Incrementals, you won't ever get a synthetic full until Incremental #31 is being created.  The strategy you're describing above sounds like you'd only ever have a Full followed by 30 Incrementals, and then you'd manually capture a new Full -- nothing at all wrong with that strategy, but you'd never get a Synthetic Full out of that.

Now, a few best practice recommendations.  If you've got 2 targets, I would recommend that you instead rotate them back and forth on a weekly basis rather than keeping both connected all the time and backing up to one of them for 2 months before switching to the other.  Weekly rotations will allow you to keep some of your backups offline (safe from ransomware), which additionally gives you the option to store them off-site for protection against break-ins, natural disasters, etc.  And by rotating them weekly, you'll ensure that even if you do lose the drive with the most recent backup, the "fallback" drive is no more than a week out of date rather than 2 months.  Finally, you wouldn't even need 2 separate schedules.  If only one drive is ever connected at a time and it's assigned a consistent drive letter (there are tools to enforce this), you could just set up a single schedule and Reflect will simply back up to whichever set is available at the time a backup runs.  Backup sets on different disks are wholly independent from each other (i.e. you would never have a situation where an Incremental chain requires files from two different disks).  Alternatively, you can use Reflect's built-in support for specifying additional destinations and targeting them by Disk ID rather than drive letter, in which case Reflect will again use whichever drive is available at any given time when the backup runs, or the first available drive in the list if multiple targets are available, all regardless of drive letter assignment.  Easier scheduling and better resilience -- what's not to love? Smile

However, if you do want to temporarily pause a scheduled task, you can do so under the Scheduled Tasks tab by right-clicking it and choosing "Disable" -- although this will disable ALL scheduled backup types for that definition file.  If you want to stop all scheduled jobs for a task permanently, you'd just go to Edit Schedule and remove all scheduled backup types.  The definition file would still stay there in case you wanted to run it manually.

Yes, you can delete older backup sets if you want, but the idea of "Incrementals Forever with Synthetic Fulls" is that you don't ever have more than one set on a given disk.  Instead, you just decide how far back you'd ever need to restore, and set your retention period as appropriate to achieve that based on the frequency of backups.  For example, if you wanted 2 months of history on each of your 2 disks and you were rotating them weekly with backups running every day, you'd just set your retention policy to 27.  The Full plus those 27 Incrementals is 28 backups, or 4 weeks' worth -- but since each disk is only being used every other week, that allows you to go 8 weeks back in time, with each disk having alternate weeks' backups on it.   Does that make sense?

Edited 29 May 2017 11:42 PM by jphughan
phrab
phrab
Proficient Member
Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)
Group: Forum Members
Posts: 148, Visits: 321
Thank you so much for your explanation.  To answer your question, "Yes!  That makes a lot of sense."  While it will take me awhile to get used to Macrium, your explanation goes a long way.  I really appreciate the time you've taken!
--
Phil


Phil
Windows 10 Pro
Windows XP- SP-3

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 8K, Visits: 55K
phrab - 29 May 2017 11:40 PM
Thank you so much for your explanation.  To answer your question, "Yes!  That makes a lot of sense."  While it will take me awhile to get used to Macrium, your explanation goes a long way.  I really appreciate the time you've taken!
--
Phil

Glad to help!  I made a few minor edits after that initial post.  Anyhow, you may also find this synthetic full animated demo that Macrium staff created useful: https://www.youtube.com/watch?v=6aeWSmT2TP0

And of course feel free to keep asking questions, since there are several people on these forums that are knowledgeable and happy to help. Good luck!



CraigS26
CraigS26
Talented Member
Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)
Group: Forum Members
Posts: 93, Visits: 449
[ If you've got 2 targets, I would recommend that you instead rotate them back and forth on a weekly basis rather than keeping both connected all the time and backing up to one of them for 2 months before switching to the other.  Weekly rotations will allow you to keep some of your backups offline (safe from ransomware), which additionally gives you the option to store them off-site for protection against break-ins, natural disasters, etc.  And by rotating them weekly, you'll ensure that even if you do lose the drive with the most recent backup, the "fallback" drive is no more than a week out of date rather than 2 months. 

Finally, you wouldn't even need 2 separate schedules.  If only one drive is ever connected at a time and - it's  - assigned a consistent drive letter (there are tools to enforce this), you could just set up a single schedule and Reflect will simply back up to whichever set is available at the time a backup runs. ]
=================================================================
Hopefully this clarification I'm asking will help the Phrab, too.
For [ One Schedule for 2 Drives ....one drive connected at a time and it's (vs. THEY) assigned a consistent drive letter ] ....to an Amateur like me .... can mean Drive 1 is always E and Drive 2 is always J -- OR -- BOTH Drives are Always -E-, which would require a Manual Renaming of one of them (  I THINK )  in Disk Mgt. ..........What do you really mean?

Also, I seem to recall bailing out of Scheduling BECAUSE Macrium relies on Task Scheduler, and Task Scheduler wanted a Profile Password (don't use one) to exist for it to work.
AM I remembering right and is THAT Still True?
Thanks for all of your expertise provided to us.





Macrium V7.2.4971 - WIN 10 1909 Hm


Edited 30 May 2017 1:16 PM by CraigS26
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 8K, Visits: 55K
You'd want both disks to be assigned E, or whatever letter you pick.  I don't think Disk Management will work for this, because if you manually assign a letter to one disk, and then later manually assign it to another disk, Windows will forget the special assignment for the first disk even if they're never connected at the same time.  But you can try this; I don't remember that for sure.  I use a tool called USB Drive Letter Manager (free for personal use, inexpensive for commercial use) in a mode where I have a little DriveLetter.ini file on each of my disks containing the letter I want them to use, with all drives specifying the same letter.  That works well enough, but if you want to use an "all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily, then use the "Add additional destinations" feature to add all of them, using whatever drive letter they're currently assigned.  Reflect will actually store their Disk IDs rather than drive letters in its definition file (even though it continues to show the currently assigned drive letter), and from that point on it will just use whatever disk is connected.

As for the scheduling user, V7 now by default uses the SYSTEM account, which does NOT require storing passwords (unless you need to store passwords to access network shares), and also means that you don't risk breaking your scheduled backups by changing the schedule user's password.  But yes, if you choose to use the V6 method of specifying a specific user to run the backup, you will have to store the password, although Windows is very good about protecting that type of content.  It stores passwords for Wi-Fi networks and such, after all.  It's not just a plaintext file or registry entry somewhere. Smile

Edited 30 May 2017 1:32 PM by jphughan
phrab
phrab
Proficient Member
Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)
Group: Forum Members
Posts: 148, Visits: 321
jphughan - 29 May 2017 12:14 AM
The Verify option that applies to backup creation jobs, I believe, will only verify the most recently created backup file.  It will not go back and verify the entire set every single time, since that could take forever and would result in all of your data constantly being re-verified.  I believe the idea there is that there would be "transitive trust", i.e. presumably if you had that option set from the beginning, then you've already verified those earlier backups anyway. However, I'd be curious to know if that Verify option will verify a newly generated synthetic full after a consolidation operation occurs. But the option I was talking about above is one you can check immediately before initiating a restore.  That option WILL verify as much of the set is necessary to execute the restore, but again that's something that only occurs immediately prior to a restore, not after a backup is created.

I checked with Macrium support & you are absolutely right.  They said: "With the option to "Auto Verify" after creation, Reflect will verify just the latest created file. When selecting the option to "Verify before Restore", this will verify the full chain before the restore as the chain will need to be valid in order to successfully restore any point in that chain."

Thanks again, Phil


Phil
Windows 10 Pro
Windows XP- SP-3

Edited 30 May 2017 3:05 PM by phrab
CraigS26
CraigS26
Talented Member
Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)Talented Member (138 reputation)
Group: Forum Members
Posts: 93, Visits: 449
Thanks for the Reply.
I've setup (2) Non-synthetic Schedules ( [Western Dig] Ext Drives E & J ) just to get my feet wet and we'll see how things go.

Fulls ea Monday 5 p.m. starting Mon June 5, retain 1 Full, and 6 Daily Incrementals starting Tuesday June 6 @ 5 p.m..
The Next Mon Full is Made and THEN the Prior Full is Deleted with attached 6 Incr's ( minus any missed days).
IF a Failure occurs during the NEXT Monday Full I still have the Last Monday Full because I didn't Check "Purge Before Image".

I'll alternate Drives Weekly as you suggested prior -

You lost me on the [ all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily ] .... but it sounds worth reading the Manual for.
My first look in the Manual didn't reveal anything about Destination by Disk ID  But Don't bother explaining here as I don't mind trying Two Rules Sets and others need your help more, I'm sure..

Thanks again!



Macrium V7.2.4971 - WIN 10 1909 Hm


jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 8K, Visits: 55K
CraigS26 - 30 May 2017 3:27 PM
Thanks for the Reply.
I've setup (2) Non-synthetic Schedules ( [Western Dig] Ext Drives E & J ) just to get my feet wet and we'll see how things go.

Fulls ea Monday 5 p.m. starting Mon June 5, retain 1 Full, and 6 Daily Incrementals starting Tuesday June 6 @ 5 p.m..
The Next Mon Full is Made and THEN the Prior Full is Deleted with attached 6 Incr's ( minus any missed days).
IF a Failure occurs during the NEXT Monday Full I still have the Last Monday Full because I didn't Check "Purge Before Image".

I'll alternate Drives Weekly as you suggested prior -

You lost me on the [ all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily ] .... but it sounds worth reading the Manual for.
My first look in the Manual didn't reveal anything about Destination by Disk ID  But Don't bother explaining here as I don't mind trying Two Rules Sets and others need your help more, I'm sure..

Thanks again!


It's under Edit Defaults > Advanced > Destination Drive Discovery.  Note that this will affect ALL definition files you may configure, and if you make this change after setting up your definition file, edit the definition file to make any kind of change (even if you save it and then edit it again to reverse that change immediately thereafter) to ensure that the definition file is updated to store Disk IDs rather than drive letters.  You can also verify that that's the case by looking at the XML view of your definition file Smile

Edited 30 May 2017 7:42 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 8K, Visits: 55K
A few further thoughts.  Weekly Fulls and daily Incrementals is a perfectly valid and widely used backup strategy, but there are some benefits to the Incrementals Forever with Synthetic Fulls strategy that you may want to consider after you get your feet wet:

- Retaining a given amount of history takes much less storage space since there's only ever one Full.
- Backups always happen more quickly since you're only ever capturing Incrementals rather than capturing a new Full from scratch every week.
- You only ever lose one backup to your retention policy every time a new backup runs, i.e. if you run daily backups, you only ever lose the oldest day's backup each day a new backup runs. By comparison, with weekly Fulls and daily Incrementals, on the day a weekly Full runs and the Full retention policy is triggered, that will delete the oldest Full AND all of its child Incrementals (since those are useless without the Full anyway), which means that you lose a week's worth of history in a single moment.  Consequently, if you want to always make sure you have a minimum amount of history, you need to make sure you have enough storage to retain an EXTRA week's worth in order to offset this. Otherwise, you'd have to have the attitude of, "I'd like to have up to 3 weeks of history since that's all the storage I can allocate to backups, but I accept that I'll only ever have 3 weeks AT MOST, but sometimes I'll only have 2 weeks of history."
Edited 30 May 2017 7:43 PM by jphughan
phrab
phrab
Proficient Member
Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)Proficient Member (252 reputation)
Group: Forum Members
Posts: 148, Visits: 321
Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated.

Is that correct?

Phil
Windows 10 Pro
Windows XP- SP-3

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search