Incremental Forever (Synthetic Full Backup)


Author
Message
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 82K
phrab - 31 May 2017 1:26 AM
Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated.

Is that correct?

Yes, that should work, but if you want to be sure, you can verify that Disk IDs rather than drive letters are being used in your definition file by selecting the "XML View" tab underneath the definition files list and looking through the contents.  If that looks good, from this point on, Reflect will send backups to whichever of those disks is connected at the time the job runs, even if the disk is assigned some other drive letter in the future.  If both disks are available, I believe whichever disk was assigned E when you set up the definition file will be used since that was specified as your primary target.  Other than that, technically you just created a Full backup; Synthetic Fulls are only ever created by Reflect automatically when needed to remain in compliance with your Incrementals retention policy, unless for some reason you use the standalone Consolidate.exe to perform that process manually.  But otherwise, it sounds like you understand the relevant concepts and should be completely good to go. Smile

Edited 31 May 2017 2:28 AM by jphughan
CraigS26
CraigS26
Talented Member
Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)
Group: Forum Members
Posts: 131, Visits: 780
phrab - 31 May 2017 1:26 AM
Hi again jphughan,
Just to make sure I have this right, as I found the steps slightly different from what you said.  There are times when I have both drive D & E open, as drive D has some files I use.  So this is what I did.
1.  I made a Full Synthetic Backup for Drive E.  
2.  Then I went to the Backup definition tab, right clicked on the scheduled backup & chose Edit.
3.  Under Destination, I clicked on Alternative locations.
4.  Drive E was selected & I added Drive D.  I did not see any Disk ID, just the drive letters.

When that was done, I went to Other Tasks>Edit Defaults>Destination Drive Discovery & chose "Use the unique volume identifier to locate the backup drive".  

Then I went back to the backup definition tab, made a change>OK, made the change back>OK to make sure the backup was updated
Is that correct?

Phrab, Thanks for this as it's confirmed Most of what I thought was the correct way.
I think I'm with you up to here UNTIL the Red above in your Post (you may have to Clk the Quote Arrow to see it).

Then I went back to the backup definition tab
What does "Made a Change - OK, made the Change Back - OK ... mean. Dbl Clking it doesn't appear to offer anything about the Destination Drive Discovery selection, so I'm lost as to What you did and How.

HOW does My Setup Screen Below compare with your screen?
WHAT were your Retention Rule #s - OR - is the 1 /30 Default THE ONLY WAY? (1) Synthetic Full if Possible and 30 INCR is Default.
Purge BEFORE Image is Correct?
Save as XML left Checked?
Any clarification is appreciated.
I did notice the Following on manual Pg 368 ...........
Changing the option affects new backup definitions. To apply to an existing backup definition, the
definition XML file needs to be opened and re-saved by clicking 'Finish'.
I did this and presume until the June 5 Syn Full Start schedule I can Manually make Incrementals to an Existing FULL.
Still wondering, too, IF all who understand v7 aren't really just visiting from another galaxy ... TBD

http://forum.macrium.com/uploads/images/c099cb8e-6e98-4e03-b049-5d76.jpg







Macrium V8.1.7638 - WIN 10 Pro 22H2 - 256G SSD + 1 TB HDD - ESET EIS Prem - Mbam Prem - SuperAS Pro - Diskeeper Pro '15


Edited 31 May 2017 2:32 PM by CraigS26
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 82K
- You don't have to make a change specifically to your destination configuration in order to trigger the definition file update after changing to Disk ID-based targeting.  You just need to change some preference somewhere in your definition file, then click OK.  Then you can open it right back up and undo that change if you want.  Again, you can check the XML View of your definition file after this to verify that you have Disk IDs rather than drive letters shown there.  Another test would be manually changing the drive letter of your target and seeing if Reflect still uses it.  The Reflect activity window you show will ALWAYS show the drive letters currently assigned to the target(s) simply for readability, so you may see them change depending on what letter they're assigned.

- You do NOT have to use the default 30 Incrementals to use this strategy.  I use 7 based on my disk rotation that involves a different disk for every weekday. That decision should be calculated based purely on how far back you want to be able to restore backups from and how often you perform them, and what your rotation strategy is like if you're using one.  And you obviously need to make sure your destination has enough storage to support that.  But for example if you want to keep 1 month of backups and you're performing daily backups to a rotation of 2 disks that you're swapping weekly, each disk will only need 14 backups on it, so you'd set your Incrementals to 13 (the root Full that's created the first time isn't counted in the Incremental retention period).  Note however that for Synthetic Fulls, I believe Reflect still requires you to specify the retention period in number of backups (the default), not number of days/weeks.

- Yes, you can manually make Incrementals if you want, and Reflect will still start making synthetic fulls whenever necessary in order to remain in compliance with your Incrementals retention period.  The only requirement is that your backup set can ONLY consist of an initial Full followed by Incrementals.  If there's a manually created Differential in the chain, synthetic fulls will never be created. Instead, Reflect will use Incremental Merge, which is beyond the scope of this post unless you're curious. Smile

- The purge option can be whichever you want.  Typically that's more important for people who are capturing multiple real Fulls and don't have storage to temporarily store an extra Full while it's being created before deleting an old one, so they need to delete the oldest one first, before running the new backup.  That of course creates a risk in that scenario, especially if you only have enough storage for a SINGLE Full, because then there's a period where you have no previous backup and your current backup hasn't completed yet.  But in an Incrementals Forever strategy, this option shouldn't matter very much. If you're running so close to your target's capacity that this option makes the difference between running out of storage and not, you need more capacity or a shorter retention period, especially since this strategy can't use the "low disk space purge" option below, so you simply need to make sure that you have enough capacity to support your retention period.  I personally choose to run the purge AFTER the backup simply because it means I always have at least the amount of history I expect to rather than prematurely purging an old backup before knowing that the new one was successful. It also means the backup (more important operation) happens FIRST, whereas if you run the purge first, consolidation occurs first.

We're not from another planet.  It's just that Reflect is a powerful and flexible piece of software, so it takes a bit of familiarization to utilize fully. But Macrium has great documentation, and fortunately they've resisted the temptation to do SO much that they end up with a bloated, complicated, and completely unintuitive solution like many enterprise backup solutions I've seen!

Edited 31 May 2017 3:14 PM by jphughan
CraigS26
CraigS26
Talented Member
Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)Talented Member (199 reputation)
Group: Forum Members
Posts: 131, Visits: 780
I DO See IDs in the XML Tab so it appears all is well. ......Backup Def File Tab: ..... Hi-Lighting the Full Syn Schedule XML line that Starts June 5 ..... 
I'll run a Manual INCR late this afternoon and see what happens but the Monday June 5 Schedule Start will be the real Test.
I can never repay you for the help, and Thanks to Phrab for letting me join the discussion.;
The Retention # is very helpful and having made Only 3 Restores with Macrium I've Always Used the Most Recent data; Chasing a date you know you didn't have the current Virus you're fighting would be my best guess for specifying More INCR's. I'll start with 7, I think, and adjust as I learn or needs change.
This stuff isn't always intuitive to me but I'm willing to try and have learned a lot just in this Thread; I'll dig into the Manual more as I haven't found the specifics there yet that you have offered.
Pg 368 just mentions Unique Volume Identifier but no details there.
Thanks again! All the Best!


Macrium V8.1.7638 - WIN 10 Pro 22H2 - 256G SSD + 1 TB HDD - ESET EIS Prem - Mbam Prem - SuperAS Pro - Diskeeper Pro '15


Edited 31 May 2017 9:16 PM by CraigS26
phrab
phrab
Proficient Member
Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)
Group: Forum Members
Posts: 211, Visits: 453
I thought I followed your advice, but I'm wondering if I made a mistake.  I have 2 external drives & opted to create a full backup, followed by 7 incrementals on Drive E.  After 7 incrementals, Reflect creates a synthetic backup.  Then I do the same with Drive D.  I am using Disk ID to select the drive, with E being the primary.

However, I think I'm missing something.  I started on 5/31 & I understand that 5/31 to 6/2 backups are gone.  But if I wanted to, could I really go back to 6/3 on my backup?  The attachment shows that my synthetic full is modified today, as is today's incremental backup.  Drive E contains a synthetic backup dated 6/8 & incrementals from 6/9 through 6/15.

If I wanted to restore to 6/3, could I?  If so, would I have to have both Drive D & E available or just Drive E?
Thank you in advance,
Phil 

Phil
Windows 10 Pro
Windows XP- SP-3

Attachments
E Drive Backups.PNG (5 views, 31.00 KB)
Richard V.
Richard V.
Most Valuable Professional
Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)Most Valuable Professional (4.1K reputation)
Group: Forum Members
Posts: 2K, Visits: 8K
If I wanted to restore to 6/3, could I?
Yes.

If so, would I have to have both Drive D & E available or just Drive E?
Just "Drive E".  (And the recovery target drive, of course.)

Either of the two backup sets can be used for recovery to any of its "snapshots" that remain available after the most recent consolidation of that backup set.  In fact, if you wanted to, you could revert to the pre-6/3 status of the consolidated full in that set, but you'd need to look at it in Reflect's user interface (rather than Windows File Explorer) to determine exactly what date-time status that consolidation represents.

I'll leave it for you to ask if you want a more "long-winded" dissertation on the subject of backup set management, etc., but this KB article provides an overview.


Regards, Richard V. ("Arvy")
https://forum.macrium.com/uploads/images/afc5d4fe-5d25-4e25-be94-185e.png

Edited 17 June 2017 6:49 PM by Arvy
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 82K
Do not rely on the Date Created or Date Modified timestamps in Windows when using Synthetic Fulls or Incremental Merge. As Arvy said, I can give more detail as to why if you're curious, or you can just look at the Backup Date in Reflect's Restore tab or the Reflect property tab that's available when you right-click a backup file and select Properties.
Edited 17 June 2017 6:50 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 82K
Now that I'm not typing from a phone anymore, I'll provide the detail in case the OP or anyone else who happens on this thread later is curious:

The Date Created timestamp will be misleading on any backup file that has had a newer backup merged into it, i.e. the root Full in a Synthetic Fulls strategy. The reason is that the Date Created timestamp will always have the date that the file was originally created, and as the contents of that file are "carried forward", the Date Created timestamp is NOT updated -- after all, the merge simply updated an existing file as opposed to creating a brand new one. But this means that as soon as the first merge occurs, if you look at the Date Created timestamp in Windows, you'll appear to have a backup available from longer ago than you actually do. If you're using Incrementals Forever without Synthetic Fulls (called Incremental Merge), this will be observed on the oldest Incremental in the chain since it would be the file receiving merges rather than the root Full, which remains static.

The Date Modified timestamp can be problematic both for any backup file that has other backups merged into it and possibly ALL files in the backup chain; the latter case depends on whether you're using Delta Incremental Indexes, which is enabled by default in V7 and disabled by default in V6.  The first case is problematic because let's say you're using Synthetic Fulls and have a Full captured on May 1. It eventually gets the first Incremental in the chain from May 2 merged into it. The result is a Synthetic Full with a date of May 2 -- but the actual merge operation might have occurred on June 1, for example, and therefore the Full will have a Date Modified timestamp of June 1. Windows of course doesn't know or care that the file in question contains a backup from May 2. This leads to the strange result that if Delta Incremental indexes are enabled (more on this momentarily), your Full will appear to be the newest file in the set according to Windows (or second newest behind the most recent Incremental if you have "Run purge before backup" enabled), even though it of course contains the OLDEST backup.  If you're only using Incremental Merge rather than Synthetic Fulls, this again will be observed on the oldest Incremental instead.

The second issue revolves around Delta Incremental Indexes. In short, if you have them enabled, then the Date Modified timestamps on the Incrementals in your chain (except the first one in an Incremental Merge strategy) WILL correspond to the time that the backup jobs they contain actually completed. Reflect's Backup Date however shows the time the job started, so those will still always be a bit off. But if you have that option DISABLED, then each Incremental contains a full index of the data contained in the backup set up to the time of that Incremental, as opposed to Delta indexes where each Incremental only contains an index of the files contained in that specific Incremental.  The reason that becomes an issue is that when a merge occurs between older backup files in the chain, that of course changes the data in the set, and if all Incrementals contain a full index up to their own backup date, then all of them need to be modified with an updated index that reflects the changes made by the merge. Consequently, in a Synthetic Full strategy, as soon as the first merge occurs in your backup set, ALL files in the chain will always have a Date Modified timestamp of the most recent backup job's completion.  If you're using Incremental Merge instead of synthetic fulls, then this will still be true EXCEPT for the root Full, since that isn't being carried forward. Further reading on Delta Incremental Indexes if you'd like: http://knowledgebase.macrium.com/display/KNOW/Delta+Indexes+for+Incremental+Backups

Edited 17 June 2017 11:24 PM by jphughan
phrab
phrab
Proficient Member
Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)Proficient Member (351 reputation)
Group: Forum Members
Posts: 211, Visits: 453
Thank you again for all the information!!  Typing from a phone can suck.BigGrin


Phil
Windows 10 Pro
Windows XP- SP-3

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search