Macrium Support Forum

Incremental Forever (Synthetic Full Backup)

https://forum.macrium.com/Topic14392.aspx

By phrab - 28 May 2017 4:39 AM

I'm fairly new to Macrium Reflect, but not new to backups, having used Acronis True Image for over a decade.  My plan was to use the Incremental Forever option, backing up every other day for a month or two.  I always have options set to verify immediately.  Then I switch to another external drive (each of which is 2TB) & do the same thing.  My questions are these:
1.  When MR creates a Synthetic Full backup, what's the danger that this full backup becomes corrupt?  I'm worried that after, say, a month & 1/2, the original full backup, or even the first incremental, becomes corrupt.  Does my plan seem reasonable.
2.  If I want to delete an older backup set, can I do this after the fact or do I have to decide on "retention rules" before I run my backup set?
Thank you in advance,
Phil
By Arvy - 28 May 2017 5:08 AM

RE #1: The is always some risk of corruption with any long chain of incremental backups and that risk is not greatly affected by choosing the Incrementals Forever option.  In fact, that option's consolidation function will "fail safe" (i.e. not alter the existing state) if it encounters a problem.  (See this KB article for more details and Macrium also provides some good YouTube videos.)  Immediate verification is a wise precaution, but it doesn't eliminate the possibility of post-operative corruption due to drive deterioration or other such causes.  Multiple backups on multiple drives with off-site storage for at least one of them is the generally recommended procedure for protecting critical data.

RE #2: Reflect allows manual deletion of old backup sets at any time you choose regardless of your retention rule choices.  You should always use the Reflect user interface for that and other backup set management and not any other methods such as using the Windows File Explorer to delete backup files. That's especially important if the same destination folder is used for more than one backup set.  For some general information on Reflect's management of backup sets, see this KB article.
By phrab - 28 May 2017 5:34 AM

Thank you so much.  I appreciate your quick & thorough reply!
By Arvy - 28 May 2017 5:49 AM

You're welcome.  I fully understand that familiarizing oneself with any new application and its multiple options takes some considerable time and effort.  Be assured that you've made a good choice to become a member of a large group of "converts" here. Smile
By jphughan - 28 May 2017 7:17 PM

Fwiw, I use a very similar strategy to you, mixing Incrementals Forever with Synthetic Fulls and disk rotation.  Macrium assures me that they have extremely robust integrity checking on the consolidation operations, and that consolidations that fail midway through for whatever reason can be recovered from, which is confirmed by logs indicating exactly that.  I'm not sure if the Verify option will actually verify the newly consolidated Full, however; I think it only verifies the new Incremental, but Macrium would be able to give you a definitive answer there.  That said, I've done manual verifications on various files in my sets and so far they've all passed, and I've been running Incrementals Forever with Synthetic Fulls for several months now.  I don't verify backups as part of the job because while I did so while I was using tapes, I never really saw the point on disks, and given that Macrium only added a verification option to Reflect as of V6 (or maybe V5?), it seems they didn't for a long time either.  I don't manually check file hashes every time I copy a file from one disk to another, after all, and the verification time might cause my backups to run unacceptably long.

For #2, yes you can delete old backup sets, but in an Incrementals Forever backup, there's only ever one set on a given destination in the first place, so that doesn't really mean anything.  If you want to reduce the size of your Incremental chain, however, you obviously can't accomplish that by simply deleting the old ones.  Macrium does offer a standalone Consolidate.exe file to reduce the chain while maintaining its integrity, however, or you can simply reduce your retention policy later and Reflect will consolidate multiple files the next time the backup job runs if that's necessary to comply with your new rules.
By phrab - 28 May 2017 9:37 PM

Thank you jphughan for this information.  For #2, I was actually thinking that I would keep a Synthetic Full on Drive D: & then do a Synthetic Full on Drive E: & switch back & forth.  At some point, I would want to delete an old backup chain to make room.  I feel better about not having the chains too long.  After all, if you do 60-70 backups (full followed by incrementals), & something happens to the 2nd incremental, your last 60 or so backups are illusory.

With True Image, I found that sometimes after numerous incremental backups, I would get a message that it couldn't find a previous incremental & therefore couldn't create a new one.  By keeping the backup chain shorter, that chance was lessened.  And, by the way, I'm not trying to knock True Image.  It served me well for over a decade.  But their last version, 2017, has caused me nothing but problems for almost 6 months.  Their support team tried to help, but I had numerous issues.  The only thing I use it for is their "Try & Decide" function, which you start & then can download or try out any app you want.  If you don't like it, just reject all the changes...it's faster than doing a backup & then restoring.

By the way, regarding verifying, I checked the MR user manual & on p.186 it said: "The entire backup set required to restore the selected backup file will be verified. To just verify the integrity of the selected Differential or Incremental select the option 'Verify just this Differential/Incremental."
Also, I found in their knowledge base for Auto verify:
Auto VerifyAutomatically verify the resulting backup file. This will add more time to the backup process but confirms if the resulting backup can be restored from. 
Thanks again!!  This seems like a wonderful group!



By jphughan - 28 May 2017 9:52 PM

^ That note about verification applies to when you perform a RESTORE, where there is an option (NOT enabled by default) to verify the backup before beginning the restore. It does not mean the entire set is verified at every backup/consolidation operation.

I agree with keeping chains shorter, but I still don't understand what you mean about deleting old chains if you're using a Synthetic full. There's only one chain in that strategy on the disk. I personally use a rotation with a different disk for each day of the week and 7 Incrementals per disk, so on any given disk I have weekly backups going back 2 months with fairly few files. I use a tool called USB Drive Letter Manager to make sure each disk is always assigned the same letter since only one disk is ever connected at s time (the others are offline and/or off-site to guard against ransomware and natural disaster), though Reflect does have other ways to accomplish this native to it. Good luck!
By phrab - 28 May 2017 11:34 PM

Hi jphughan,
Auto Verify Automatically verify the resulting backup file. This will add more time to the backup process but
confirms if the resulting backup can be restored from."  I guess I assumed that that would mean all the previous backups were OK, Otherwise, you couldn't restore from the last one.  However, I will check with support to be sure.

As far as what I meant by deleting old chains is that what I was planning to do is create Synthetic Full #1 on Drive D, then create Synthetic Full #2 on Drive E, Synthetic Full #3 on Drive D, etc.  At some point, I would go back & delete Synthetic Full #1.  I was thinking that this would keep the chains shorter than continuing with just one chain per external drive.

Thanks again!
Phil
By jphughan - 29 May 2017 12:14 AM

Ok, I was thinking of a different option.  The Verify option that applies to backup creation jobs, I believe, will only verify the most recently created backup file.  It will not go back and verify the entire set every single time, since that could take forever and would result in all of your data constantly being re-verified.  I believe the idea there is that there would be "transitive trust", i.e. presumably if you had that option set from the beginning, then you've already verified those earlier backups anyway. However, I'd be curious to know if that Verify option will verify a newly generated synthetic full after a consolidation operation occurs. But the option I was talking about above is one you can check immediately before initiating a restore.  That option WILL verify as much of the set is necessary to execute the restore, but again that's something that only occurs immediately prior to a restore, not after a backup is created.

You can't span a chain across multiple drives.  You might want to more clearly detail your intended schedule, settings, and disk rotation, because it sounds like you may be about to attempt something that won't work as you want it to.  You can't delete a Synthetic Full without killing all of the backups that were built off of it, and Synthetic Fulls aren't something that are typically created manually in the first place; they're created (and then updated) over the course of scheduled backups in order to comply with your specified retention policy.  The only way to keep your chains shorter is to reduce the number of Incrementals you retain, and if you want to do that while retaining the option to go back a given amount of time in the past, you'd either need to reduce the frequency of backups or add more disks to your rotation.  Like I said, I have a strategy that in aggregate gives me daily backups going back 2 months, but I only have a chain of 7 Incrementals on any given disk.
By phrab - 29 May 2017 7:10 PM

jphughan - 29 May 2017 12:14 AM
You can't span a chain across multiple drives.  You might want to more clearly detail your intended schedule, settings, and disk rotation, because it sounds like you may be about to attempt something that won't work as you want it to.

Thank you again for your reply.  This is what I thought I could do, but please tell me if this won't work.  I have 2 external drives: D & E.  I wanted to be able to start with a Full backup & then do incrementals every other day.  I would make the decision how many incrermentals to keep flexible, which is why I chose the Synthetic Full backup.  My plan was to start on Drive D, choosing Incrementals Forever as the backup plan, backing up every 2nd day until I have about 30 incrementals (~2 months worth).  Then I would stop the schedule (although I don't know how to do that yet).  Then I would do the same thing on Drive E (~2 months of backups, doing it every other day).

Then I wanted to repeat those steps by creating a new Synthetic Full backup on Drive D, independent  of the previous Synthetic Full backup & do that for about 2 months.  Then I would repeat on Drive E.  My idea was to do this many times, having a few complete Synthetic Full backup sets on each external drive.  Actually, some of them might now be Synthetic Full because if there's space on the drive, it would just be a Full backup followed by ~30 incrementals.

At some point, I would want to delete an entire old backup set.  Am I able to do this or just confused?

By the way, I'm not sure how to edit the schedule of an existing backup plan to stop future backups.  I couldn't find a "Do not schedule" option.
By jphughan - 29 May 2017 11:30 PM

Ok, this post is going to be a mixture of clarifications in terms of how Reflect and synthetic fulls work, and also some of my own best practice recommendations.

First, the only time Reflect ever generates a synthetic full is when it creates a new Incremental and the resulting number of Incrementals in the chain exceeds the number you've specified in your retention period.  For example, in my "Incrementals Forever with Synthetic Fulls" job, I specify to retain 7 Incrementals.  The first time my job executed, it generated a Full.  Then each of the next 7 times, Incrementals were generated.  The time after that, an 8th Incremental was generated, but since that exceeds my retention policy, at that point Reflect automatically merged Incremental #1 into my Full backup, thereby "carrying the Full forward" so that I have a Full equivalent to having actually captured a Full on the day Incremental #1 ran, even though I didn't actually run a Full that day -- hence the name "synthetic full".  Incremental #1 is deleted after the Full is updated with its contents (thereby keeping my backup set compliant with my retention policy), and of course I lose the ability to restore all the way back to the day I ACTUALLY ran that Full.  From that point on, this process of merging the oldest Incremental into the root Full continues each time a new Incremental is generated, thereby keeping your backup set current without the number of Incrementals ever exceeding the retention policy you specify.  But if you set a retention policy of 30 Incrementals, you won't ever get a synthetic full until Incremental #31 is being created.  The strategy you're describing above sounds like you'd only ever have a Full followed by 30 Incrementals, and then you'd manually capture a new Full -- nothing at all wrong with that strategy, but you'd never get a Synthetic Full out of that.

Now, a few best practice recommendations.  If you've got 2 targets, I would recommend that you instead rotate them back and forth on a weekly basis rather than keeping both connected all the time and backing up to one of them for 2 months before switching to the other.  Weekly rotations will allow you to keep some of your backups offline (safe from ransomware), which additionally gives you the option to store them off-site for protection against break-ins, natural disasters, etc.  And by rotating them weekly, you'll ensure that even if you do lose the drive with the most recent backup, the "fallback" drive is no more than a week out of date rather than 2 months.  Finally, you wouldn't even need 2 separate schedules.  If only one drive is ever connected at a time and it's assigned a consistent drive letter (there are tools to enforce this), you could just set up a single schedule and Reflect will simply back up to whichever set is available at the time a backup runs.  Backup sets on different disks are wholly independent from each other (i.e. you would never have a situation where an Incremental chain requires files from two different disks).  Alternatively, you can use Reflect's built-in support for specifying additional destinations and targeting them by Disk ID rather than drive letter, in which case Reflect will again use whichever drive is available at any given time when the backup runs, or the first available drive in the list if multiple targets are available, all regardless of drive letter assignment.  Easier scheduling and better resilience -- what's not to love? Smile

However, if you do want to temporarily pause a scheduled task, you can do so under the Scheduled Tasks tab by right-clicking it and choosing "Disable" -- although this will disable ALL scheduled backup types for that definition file.  If you want to stop all scheduled jobs for a task permanently, you'd just go to Edit Schedule and remove all scheduled backup types.  The definition file would still stay there in case you wanted to run it manually.

Yes, you can delete older backup sets if you want, but the idea of "Incrementals Forever with Synthetic Fulls" is that you don't ever have more than one set on a given disk.  Instead, you just decide how far back you'd ever need to restore, and set your retention period as appropriate to achieve that based on the frequency of backups.  For example, if you wanted 2 months of history on each of your 2 disks and you were rotating them weekly with backups running every day, you'd just set your retention policy to 27.  The Full plus those 27 Incrementals is 28 backups, or 4 weeks' worth -- but since each disk is only being used every other week, that allows you to go 8 weeks back in time, with each disk having alternate weeks' backups on it.   Does that make sense?
By phrab - 29 May 2017 11:40 PM

Thank you so much for your explanation.  To answer your question, "Yes!  That makes a lot of sense."  While it will take me awhile to get used to Macrium, your explanation goes a long way.  I really appreciate the time you've taken!
--
Phil
By jphughan - 29 May 2017 11:44 PM

phrab - 29 May 2017 11:40 PM
Thank you so much for your explanation.  To answer your question, "Yes!  That makes a lot of sense."  While it will take me awhile to get used to Macrium, your explanation goes a long way.  I really appreciate the time you've taken!
--
Phil

Glad to help!  I made a few minor edits after that initial post.  Anyhow, you may also find this synthetic full animated demo that Macrium staff created useful: https://www.youtube.com/watch?v=6aeWSmT2TP0

And of course feel free to keep asking questions, since there are several people on these forums that are knowledgeable and happy to help. Good luck!


By CraigS26 - 30 May 2017 1:15 PM

[ If you've got 2 targets, I would recommend that you instead rotate them back and forth on a weekly basis rather than keeping both connected all the time and backing up to one of them for 2 months before switching to the other.  Weekly rotations will allow you to keep some of your backups offline (safe from ransomware), which additionally gives you the option to store them off-site for protection against break-ins, natural disasters, etc.  And by rotating them weekly, you'll ensure that even if you do lose the drive with the most recent backup, the "fallback" drive is no more than a week out of date rather than 2 months. 

Finally, you wouldn't even need 2 separate schedules.  If only one drive is ever connected at a time and - it's  - assigned a consistent drive letter (there are tools to enforce this), you could just set up a single schedule and Reflect will simply back up to whichever set is available at the time a backup runs. ]
=================================================================
Hopefully this clarification I'm asking will help the Phrab, too.
For [ One Schedule for 2 Drives ....one drive connected at a time and it's (vs. THEY) assigned a consistent drive letter ] ....to an Amateur like me .... can mean Drive 1 is always E and Drive 2 is always J -- OR -- BOTH Drives are Always -E-, which would require a Manual Renaming of one of them (  I THINK )  in Disk Mgt. ..........What do you really mean?

Also, I seem to recall bailing out of Scheduling BECAUSE Macrium relies on Task Scheduler, and Task Scheduler wanted a Profile Password (don't use one) to exist for it to work.
AM I remembering right and is THAT Still True?
Thanks for all of your expertise provided to us.



By jphughan - 30 May 2017 1:30 PM

You'd want both disks to be assigned E, or whatever letter you pick.  I don't think Disk Management will work for this, because if you manually assign a letter to one disk, and then later manually assign it to another disk, Windows will forget the special assignment for the first disk even if they're never connected at the same time.  But you can try this; I don't remember that for sure.  I use a tool called USB Drive Letter Manager (free for personal use, inexpensive for commercial use) in a mode where I have a little DriveLetter.ini file on each of my disks containing the letter I want them to use, with all drives specifying the same letter.  That works well enough, but if you want to use an "all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily, then use the "Add additional destinations" feature to add all of them, using whatever drive letter they're currently assigned.  Reflect will actually store their Disk IDs rather than drive letters in its definition file (even though it continues to show the currently assigned drive letter), and from that point on it will just use whatever disk is connected.

As for the scheduling user, V7 now by default uses the SYSTEM account, which does NOT require storing passwords (unless you need to store passwords to access network shares), and also means that you don't risk breaking your scheduled backups by changing the schedule user's password.  But yes, if you choose to use the V6 method of specifying a specific user to run the backup, you will have to store the password, although Windows is very good about protecting that type of content.  It stores passwords for Wi-Fi networks and such, after all.  It's not just a plaintext file or registry entry somewhere. Smile
By phrab - 30 May 2017 3:05 PM

jphughan - 29 May 2017 12:14 AM
The Verify option that applies to backup creation jobs, I believe, will only verify the most recently created backup file.  It will not go back and verify the entire set every single time, since that could take forever and would result in all of your data constantly being re-verified.  I believe the idea there is that there would be "transitive trust", i.e. presumably if you had that option set from the beginning, then you've already verified those earlier backups anyway. However, I'd be curious to know if that Verify option will verify a newly generated synthetic full after a consolidation operation occurs. But the option I was talking about above is one you can check immediately before initiating a restore.  That option WILL verify as much of the set is necessary to execute the restore, but again that's something that only occurs immediately prior to a restore, not after a backup is created.

I checked with Macrium support & you are absolutely right.  They said: "With the option to "Auto Verify" after creation, Reflect will verify just the latest created file. When selecting the option to "Verify before Restore", this will verify the full chain before the restore as the chain will need to be valid in order to successfully restore any point in that chain."

Thanks again, Phil
By CraigS26 - 30 May 2017 3:27 PM

Thanks for the Reply.
I've setup (2) Non-synthetic Schedules ( [Western Dig] Ext Drives E & J ) just to get my feet wet and we'll see how things go.

Fulls ea Monday 5 p.m. starting Mon June 5, retain 1 Full, and 6 Daily Incrementals starting Tuesday June 6 @ 5 p.m..
The Next Mon Full is Made and THEN the Prior Full is Deleted with attached 6 Incr's ( minus any missed days).
IF a Failure occurs during the NEXT Monday Full I still have the Last Monday Full because I didn't Check "Purge Before Image".

I'll alternate Drives Weekly as you suggested prior -

You lost me on the [ all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily ] .... but it sounds worth reading the Manual for.
My first look in the Manual didn't reveal anything about Destination by Disk ID  But Don't bother explaining here as I don't mind trying Two Rules Sets and others need your help more, I'm sure..

Thanks again!

By jphughan - 30 May 2017 7:34 PM

CraigS26 - 30 May 2017 3:27 PM
Thanks for the Reply.
I've setup (2) Non-synthetic Schedules ( [Western Dig] Ext Drives E & J ) just to get my feet wet and we'll see how things go.

Fulls ea Monday 5 p.m. starting Mon June 5, retain 1 Full, and 6 Daily Incrementals starting Tuesday June 6 @ 5 p.m..
The Next Mon Full is Made and THEN the Prior Full is Deleted with attached 6 Incr's ( minus any missed days).
IF a Failure occurs during the NEXT Monday Full I still have the Last Monday Full because I didn't Check "Purge Before Image".

I'll alternate Drives Weekly as you suggested prior -

You lost me on the [ all-Reflect" solution, configure Reflect to identify targets by Disk ID rather than drive letter, then connect all disks in your rotation temporarily ] .... but it sounds worth reading the Manual for.
My first look in the Manual didn't reveal anything about Destination by Disk ID  But Don't bother explaining here as I don't mind trying Two Rules Sets and others need your help more, I'm sure..

Thanks again!


It's under Edit Defaults > Advanced > Destination Drive Discovery.  Note that this will affect ALL definition files you may configure, and if you make this change after setting up your definition file, edit the definition file to make any kind of change (even if you save it and then edit it again to reverse that change immediately thereafter) to ensure that the definition file is updated to store Disk IDs rather than drive letters.  You can also verify that that's the case by looking at the XML view of your definition file Smile
By jphughan - 30 May 2017 7:41 PM

A few further thoughts.  Weekly Fulls and daily Incrementals is a perfectly valid and widely used backup strategy, but there are some benefits to the Incrementals Forever with Synthetic Fulls strategy that you may want to consider after you get your feet wet:

- Retaining a given amount of history takes much less storage space since there's only ever one Full.
- Backups always happen more quickly since you're only ever capturing Incrementals rather than capturing a new Full from scratch every week.
- You only ever lose one backup to your retention policy every time a new backup runs, i.e. if you run daily backups, you only ever lose the oldest day's backup each day a new backup runs. By comparison, with weekly Fulls and daily Incrementals, on the day a weekly Full runs and the Full retention policy is triggered, that will delete the oldest Full AND all of its child Incrementals (since those are useless without the Full anyway), which means that you lose a week's worth of history in a single moment.  Consequently, if you want to always make sure you have a minimum amount of history, you need to make sure you have enough storage to retain an EXTRA week's worth in order to offset this. Otherwise, you'd have to have the attitude of, "I'd like to have up to 3 weeks of history since that's all the storage I can allocate to backups, but I accept that I'll only ever have 3 weeks AT MOST, but sometimes I'll only have 2 weeks of history."
By phrab - 31 May 2017 1:26 AM

Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated.

Is that correct?
By jphughan - 31 May 2017 2:27 AM

phrab - 31 May 2017 1:26 AM
Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated.

Is that correct?

Yes, that should work, but if you want to be sure, you can verify that Disk IDs rather than drive letters are being used in your definition file by selecting the "XML View" tab underneath the definition files list and looking through the contents.  If that looks good, from this point on, Reflect will send backups to whichever of those disks is connected at the time the job runs, even if the disk is assigned some other drive letter in the future.  If both disks are available, I believe whichever disk was assigned E when you set up the definition file will be used since that was specified as your primary target.  Other than that, technically you just created a Full backup; Synthetic Fulls are only ever created by Reflect automatically when needed to remain in compliance with your Incrementals retention policy, unless for some reason you use the standalone Consolidate.exe to perform that process manually.  But otherwise, it sounds like you understand the relevant concepts and should be completely good to go. Smile
By CraigS26 - 31 May 2017 12:05 PM

phrab - 31 May 2017 1:26 AM
Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  

Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated
Is that correct?

Phrab, Thanks for this as it's confirmed Most of what I thought was the correct way.
I think I'm with you up to here UNTIL the Red above in your Post (you may have to Clk the Quote Arrow to see it).

Then I went back to the backup definition tab
What does "Made a Change - OK, made the Change Back - OK ... mean. Dbl Clking it doesn't appear to offer anything about the Destination Drive Discovery selection, so I'm lost as to What you did and How.

HOW does My Setup Screen Below compare with your screen?
WHAT were your Retention Rule #s - OR - is the 1 /30 Default THE ONLY WAY? (1) Synthetic Full if Possible and 30 INCR is Default.
Purge BEFORE Image is Correct?
Save as XML left Checked?
Any clarification is appreciated.
I did notice the Following on manual Pg 368 ...........
Changing the option affects new backup definitions. To apply to an existing backup definition, the
definition XML file needs to be opened and re-saved by clicking 'Finish'.
I did this and presume until the June 5 Syn Full Start schedule I can Manually make Incrementals to an Existing FULL.
Still wondering, too, IF all who understand v7 aren't really just visiting from another galaxy ... TBD

http://forum.macrium.com/uploads/images/c099cb8e-6e98-4e03-b049-5d76.jpg





By jphughan - 31 May 2017 3:09 PM

- You don't have to make a change specifically to your destination configuration in order to trigger the definition file update after changing to Disk ID-based targeting.  You just need to change some preference somewhere in your definition file, then click OK.  Then you can open it right back up and undo that change if you want.  Again, you can check the XML View of your definition file after this to verify that you have Disk IDs rather than drive letters shown there.  Another test would be manually changing the drive letter of your target and seeing if Reflect still uses it.  The Reflect activity window you show will ALWAYS show the drive letters currently assigned to the target(s) simply for readability, so you may see them change depending on what letter they're assigned.

- You do NOT have to use the default 30 Incrementals to use this strategy.  I use 7 based on my disk rotation that involves a different disk for every weekday. That decision should be calculated based purely on how far back you want to be able to restore backups from and how often you perform them, and what your rotation strategy is like if you're using one.  And you obviously need to make sure your destination has enough storage to support that.  But for example if you want to keep 1 month of backups and you're performing daily backups to a rotation of 2 disks that you're swapping weekly, each disk will only need 14 backups on it, so you'd set your Incrementals to 13 (the root Full that's created the first time isn't counted in the Incremental retention period).  Note however that for Synthetic Fulls, I believe Reflect still requires you to specify the retention period in number of backups (the default), not number of days/weeks.

- Yes, you can manually make Incrementals if you want, and Reflect will still start making synthetic fulls whenever necessary in order to remain in compliance with your Incrementals retention period.  The only requirement is that your backup set can ONLY consist of an initial Full followed by Incrementals.  If there's a manually created Differential in the chain, synthetic fulls will never be created. Instead, Reflect will use Incremental Merge, which is beyond the scope of this post unless you're curious. Smile

- The purge option can be whichever you want.  Typically that's more important for people who are capturing multiple real Fulls and don't have storage to temporarily store an extra Full while it's being created before deleting an old one, so they need to delete the oldest one first, before running the new backup.  That of course creates a risk in that scenario, especially if you only have enough storage for a SINGLE Full, because then there's a period where you have no previous backup and your current backup hasn't completed yet.  But in an Incrementals Forever strategy, this option shouldn't matter very much. If you're running so close to your target's capacity that this option makes the difference between running out of storage and not, you need more capacity or a shorter retention period, especially since this strategy can't use the "low disk space purge" option below, so you simply need to make sure that you have enough capacity to support your retention period.  I personally choose to run the purge AFTER the backup simply because it means I always have at least the amount of history I expect to rather than prematurely purging an old backup before knowing that the new one was successful. It also means the backup (more important operation) happens FIRST, whereas if you run the purge first, consolidation occurs first.

We're not from another planet.  It's just that Reflect is a powerful and flexible piece of software, so it takes a bit of familiarization to utilize fully. But Macrium has great documentation, and fortunately they've resisted the temptation to do SO much that they end up with a bloated, complicated, and completely unintuitive solution like many enterprise backup solutions I've seen!
By CraigS26 - 31 May 2017 4:53 PM

I DO See IDs in the XML Tab so it appears all is well. ......Backup Def File Tab: ..... Hi-Lighting the Full Syn Schedule XML line that Starts June 5 ..... 
I'll run a Manual INCR late this afternoon and see what happens but the Monday June 5 Schedule Start will be the real Test.
I can never repay you for the help, and Thanks to Phrab for letting me join the discussion.;
The Retention # is very helpful and having made Only 3 Restores with Macrium I've Always Used the Most Recent data; Chasing a date you know you didn't have the current Virus you're fighting would be my best guess for specifying More INCR's. I'll start with 7, I think, and adjust as I learn or needs change.
This stuff isn't always intuitive to me but I'm willing to try and have learned a lot just in this Thread; I'll dig into the Manual more as I haven't found the specifics there yet that you have offered.
Pg 368 just mentions Unique Volume Identifier but no details there.
Thanks again! All the Best!
By phrab - 17 June 2017 4:29 PM

I thought I followed your advice, but I'm wondering if I made a mistake.  I have 2 external drives & opted to create a full backup, followed by 7 incrementals on Drive E.  After 7 incrementals, Reflect creates a synthetic backup.  Then I do the same with Drive D.  I am using Disk ID to select the drive, with E being the primary.

However, I think I'm missing something.  I started on 5/31 & I understand that 5/31 to 6/2 backups are gone.  But if I wanted to, could I really go back to 6/3 on my backup?  The attachment shows that my synthetic full is modified today, as is today's incremental backup.  Drive E contains a synthetic backup dated 6/8 & incrementals from 6/9 through 6/15.

If I wanted to restore to 6/3, could I?  If so, would I have to have both Drive D & E available or just Drive E?
Thank you in advance,
Phil 
By Arvy - 17 June 2017 6:11 PM

If I wanted to restore to 6/3, could I?
Yes.

If so, would I have to have both Drive D & E available or just Drive E?
Just "Drive E".  (And the recovery target drive, of course.)

Either of the two backup sets can be used for recovery to any of its "snapshots" that remain available after the most recent consolidation of that backup set.  In fact, if you wanted to, you could revert to the pre-6/3 status of the consolidated full in that set, but you'd need to look at it in Reflect's user interface (rather than Windows File Explorer) to determine exactly what date-time status that consolidation represents.

I'll leave it for you to ask if you want a more "long-winded" dissertation on the subject of backup set management, etc., but this KB article provides an overview.
By jphughan - 17 June 2017 6:49 PM

Do not rely on the Date Created or Date Modified timestamps in Windows when using Synthetic Fulls or Incremental Merge. As Arvy said, I can give more detail as to why if you're curious, or you can just look at the Backup Date in Reflect's Restore tab or the Reflect property tab that's available when you right-click a backup file and select Properties.
By jphughan - 17 June 2017 10:57 PM

Now that I'm not typing from a phone anymore, I'll provide the detail in case the OP or anyone else who happens on this thread later is curious:

The Date Created timestamp will be misleading on any backup file that has had a newer backup merged into it, i.e. the root Full in a Synthetic Fulls strategy. The reason is that the Date Created timestamp will always have the date that the file was originally created, and as the contents of that file are "carried forward", the Date Created timestamp is NOT updated -- after all, the merge simply updated an existing file as opposed to creating a brand new one. But this means that as soon as the first merge occurs, if you look at the Date Created timestamp in Windows, you'll appear to have a backup available from longer ago than you actually do. If you're using Incrementals Forever without Synthetic Fulls (called Incremental Merge), this will be observed on the oldest Incremental in the chain since it would be the file receiving merges rather than the root Full, which remains static.

The Date Modified timestamp can be problematic both for any backup file that has other backups merged into it and possibly ALL files in the backup chain; the latter case depends on whether you're using Delta Incremental Indexes, which is enabled by default in V7 and disabled by default in V6.  The first case is problematic because let's say you're using Synthetic Fulls and have a Full captured on May 1. It eventually gets the first Incremental in the chain from May 2 merged into it. The result is a Synthetic Full with a date of May 2 -- but the actual merge operation might have occurred on June 1, for example, and therefore the Full will have a Date Modified timestamp of June 1. Windows of course doesn't know or care that the file in question contains a backup from May 2. This leads to the strange result that if Delta Incremental indexes are enabled (more on this momentarily), your Full will appear to be the newest file in the set according to Windows (or second newest behind the most recent Incremental if you have "Run purge before backup" enabled), even though it of course contains the OLDEST backup.  If you're only using Incremental Merge rather than Synthetic Fulls, this again will be observed on the oldest Incremental instead.

The second issue revolves around Delta Incremental Indexes. In short, if you have them enabled, then the Date Modified timestamps on the Incrementals in your chain (except the first one in an Incremental Merge strategy) WILL correspond to the time that the backup jobs they contain actually completed. Reflect's Backup Date however shows the time the job started, so those will still always be a bit off. But if you have that option DISABLED, then each Incremental contains a full index of the data contained in the backup set up to the time of that Incremental, as opposed to Delta indexes where each Incremental only contains an index of the files contained in that specific Incremental.  The reason that becomes an issue is that when a merge occurs between older backup files in the chain, that of course changes the data in the set, and if all Incrementals contain a full index up to their own backup date, then all of them need to be modified with an updated index that reflects the changes made by the merge. Consequently, in a Synthetic Full strategy, as soon as the first merge occurs in your backup set, ALL files in the chain will always have a Date Modified timestamp of the most recent backup job's completion.  If you're using Incremental Merge instead of synthetic fulls, then this will still be true EXCEPT for the root Full, since that isn't being carried forward. Further reading on Delta Incremental Indexes if you'd like: http://knowledgebase.macrium.com/display/KNOW/Delta+Indexes+for+Incremental+Backups
By phrab - 18 June 2017 2:33 AM

Thank you again for all the information!!  Typing from a phone can suck.BigGrin