What does "Backup Set Matching" do?


Author
Message
Mintmag
Mintmag
Proficient Member
Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)
Group: Forum Members
Posts: 152, Visits: 540
File and Folder backup is not a feature I use often but since I have it I might as well learn about it. So what does, Could someone please give me a break down of how this retention rule works, because I read about it and still don't quite get it.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Backup Set Matching is how Reflect determines which backups in a destination folder are potential targets for deletion/consolidation when the retention policy is evaluated (and also whether a Diff/Inc backup can be created, but I'll address that later).  Let's say you had two completely separate backup jobs and you were using the same destination folder for both; not everyone uses a dedicated folder for each backup job they have.  You probably wouldn't want Job A's retention policy to touch backups created by Job B.  That's why Reflect has a concept of determining which backups that exist at a destination "match" the job that's running at any given time, to make sure that it's only purging/consolidating backups that were created by previous executions of that specific job.  Note that there IS an option where you can set a job’s retention policy to be applied to ALL backups at the destination, not just matching backups. There are some cases where that can be desirable in an image backup scenario as long as you make sure that your destination folder is dedicated to storing backups of one specific job, because there are situations where even backups created by the same definition file won't always be considered a match, but I won't delve into that here since you're asking about F&F backups.

With F&F jobs, under Advanced Options you have options for choosing what constitutes a match, which isn't possible with image backups.

- The default now is "Similar", which means that backups at a destination will be considered a match to the running backup as long as they contain a backup of at least one folder that is still included in the current backup job.  The benefit of this strategy is that if you ever decide to add or remove a folder from your backup job, any backups you already created would still be considered a match (again, as long as there was at least one folder common to the previous and current backups) and therefore those backups would still be purged/consolidated according to your retention policy.  This also allows you to continue creating Diff/Inc backups to existing backups after such a change.

- By comparison, the "Strict" option means that existing backups will only be considered a match to the current backup if they have EXACTLY the same folders as the current backup.  So if you ever decided to add or remove a folder from your backup job, those old backups would no longer be considered a match and your retention policy would not act on them anymore; you'd instead have to delete/consolidate them manually whenever you wanted.  In addition, after adding or removing a folder, you'd have to create a new Full backup rather than being able to add a Diff/Inc to an existing backup.

- The final option "All" means that every backup at the destination is considered a match, regardless of how the contents compare.  This can be dangerous for retention rules, and although it means you can technically add a Diff/Inc backup to any existing backup, that might become messy.

Edited 8 May 2019 4:28 AM by jphughan
dbminter
dbminter
Macrium Evangelist
Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)Macrium Evangelist (7.7K reputation)
Group: Forum Members
Posts: 4.8K, Visits: 51K
Oh, THAT'S what that option means!  Thanks!  I needed to know that information, too.

Mintmag
Mintmag
Proficient Member
Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)
Group: Forum Members
Posts: 152, Visits: 540
jphughan - 6 May 2019 9:35 PM
Backup Set Matching is how Reflect determines which backups in a destination folder are potential targets for deletion/consolidation when the retention policy is evaluated (and also whether a Diff/Inc backup can be created, but I'll address that later).  Let's say you had two completely separate backup jobs and you were using the same destination folder for both; not everyone uses a dedicated folder for each backup job they have.  You probably wouldn't want Job A's retention policy to touch backups created by Job B.  That's why Reflect has a concept of determining which backups that exist at a destination "match" the job that's running at any given time, to make sure that it's only purging/consolidating backups that were created by previous executions of that specific job.  Note that there IS an option where you can set a job’s retention policy to be applied to ALL backups at the destination, not just matching backups. There are some cases where that can be desirable in an image backup scenario as long as you make sure that your destination folder is dedicated to storing backups of one specific job, because there are situations where even backups created by the same definition file won't always be considered a match, but I won't delve into that here since you're asking about F&F backups.

With F&F jobs, under Advanced Options you have options for choosing what constitutes a match, which isn't possible with image backups.

- The default now is "Similar", which means that backups at a destination will be considered a match to the running backup as long as they contain a backup of at least one folder that is still included in the current backup job.  The benefit of this strategy is that if you ever decide to add or remove a folder from your backup job, any backups you already created would still be considered a match (again, as long as there was at least one folder common to the previous and current backups) and therefore those backups would still be purged/consolidated according to your retention policy.  This also allows you to continue creating Diff/Inc backups to existing backups after such a change.

- By comparison, the "Strict" option means that existing backups will only be considered a match to the current backup if they have EXACTLY the same folders as the current backup.  So if you ever decided to add or remove a folder from your backup job, those old backups would no longer be considered a match and your retention policy would not act on them anymore; you'd instead have to delete/consolidate them manually whenever you wanted.  In addition, after adding or removing a folder, you'd have to create a new Full backup rather than being able to add a Diff/Inc to an existing backup.

- The final option "All" means that every backup at the destination is considered a match, regardless of how the contents compare.  This can be dangerous for retention rules, and although it means you can technically add a Diff/Inc backup to any existing backup, that might become messy.

Sorry for the late reply and thanks for the detailed explanation but I'm afraid I still don't understand what it is you're saying. Won't a differential backup yield the same results?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K

Mintmag - 10 May 2019 10:08 PM
Sorry for the late reply and thanks for the detailed explanation but I'm afraid I still don't understand what it is you're saying. Won't a differential backup yield the same results?

What do you mean "same results"?  What results, and for that matter, same results as what?  You quoted a fairly long post and then asked a fairly unclear question.  But backup set matching and Differential backups are completely different things.  A Differential backup involves comparing the source data being backed up to what's already contained in an earlier Full backup in order to figure out what needs to be included in the new Differential you're creating.  Backup set matching identifies WHICH existing Full at the destination should be used to perform that comparison (among other things), because a Differential backup would be created as a child of the newest matching Full.  The backup set matching options allow you to change how Reflect decides what constitutes a match for its purposes.

Let's say on my system, I have two different File & Folder jobs configured.  Job A backs up my Documents folder, and Job B backs up a folder called D:\Stuff.  Both of those jobs store the backup files they generate at F:\MyBackups.  Let's say I run a Full backup for each of those jobs.  I now have two completely unrelated Full backups both stored in F:\MyBackups.  Now let's say I want to run a Differential backup of Job A (Documents).  Backup set matching is the logic that Reflect uses to determine which of the Fulls at the destination to use as the basis for this new Differential.  It's also what would prevent the retention policy configured on Job A from deleting any backups that were created by Job B and that are also stored in that same destination folder.  You might be thinking, "Well obviously the Job A Differential would append to the Job A Full rather than the Job B Full and only delete previous backups created by Job A", and in this very simple example that's understandable.  But when you start considering cases of users potentially adding or removing folders to back up in their F&F definition file, it gets more complicated, and backup set matching options give you a way to specify how you want Reflect to handle those situations.

If you still don't get it, unfortunately I don't think I can explain it more clearly than I did in my original post above, so if even rereading that doesn't help, then someone else might be better suited to help here.  Or you can just leave Reflect at its default backup set matching configuration for your job, because the odds are very good that that will give you the results you expect.  This is one of those cases where if you don't clearly know that you need to customize this kind of behavior, then you very likely don't.

Edited 10 May 2019 10:53 PM by jphughan
Mintmag
Mintmag
Proficient Member
Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)
Group: Forum Members
Posts: 152, Visits: 540
Sorry again for the late reply. Been very busy with health issues. You mentioned a job scenario involving two directories in two different locations. Job:A (documents) Job:B D/Stuff.

Could you please tell me a quick job situation I could make with just text files and folders to clearly show the difference between these two polices.

I understand how retention policies works, but I do tend to keep my backup jobs quite simple. I just need a situation where I can run a test and clearly see what this setting is doing exactly. I learn from experience more so then reading. So I'd really like to see what this thing does to a test backup before I use the file and folder backup on anything that I truly care about.
Edited 17 May 2019 1:57 PM by Mintmag
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Mintmag - 17 May 2019 1:53 PM
Sorry again for the late reply. Been very busy with health issues. You mentioned a job scenario involving two directories in two different locations. Job:A (documents) Job:B D/Stuff.

Could you please tell me a quick job situation I could make with just text files and folders to clearly show the difference between these two polices.

I understand how retention policies works, but I do tend to keep my backup jobs quite simple. I just need a situation where I can run a test and clearly see what this setting is doing exactly. I learn from experience more so then reading. So I'd really like to see what this thing does to a test backup before I use the file and folder backup on anything that I truly care about.

If you keep your jobs simple and Reflect is doing what you want it to (which is likely for simple setups given Reflect's default configuration), then as I said in my previous post, you very likely don't even need to think about this.  But here you go:

The impact of the "All" setting
1. Create two folders anywhere on your PC.  Call them "TestA" and "TestB".
2. Put a few files into each folder.  It doesn't matter what kind of files they are.
3. Configure two new File & Folder backup jobs named BackupA and BackupB, but don't run them right away, so uncheck the "Run now" option at the end of the wizard.  Configure BackupA to back up the TestA folder and BackupB to back up TestB folder, and have BOTH jobs use the same folder as their destination.  Then, on both jobs set the retention policy to 1 Full backup.  You can uncheck the Diff and Inc settings for the purposes of this demo.  Finally, go to the Advanced Options section and set the Backup Set Matching option to All. Again, verify that you do all of those things for both jobs.
4. Run a Full backup of BackupA.
5. Run a Full backup of BackupB.  Notice in the job log (and by checking in the destination folder) that BackupB's retention policy deleted the backup that was created by BackupA in the previous step.  That happened because the backup set matching setting is set to All.

The impact of the "Strict" vs. "Similar" settings
1. Create two new folders anywhere on your PC. Call them "TestC" and "TestD".
2. Put a few files into each folder. It doesn't matter what kind of files they are.
3. Create ONE new File & Folder backup jobs. Call it BackupC.  To begin, only include the TestC in the backup selection. Uncheck all of the retention policy options for this demo.  Finally, go to the Advanced Options section and set Backup Set Matching to Strict.
4. Run a Full backup of BackupC.
5. Edit the settings of that job to add the TestD folder to the backup selection.
6. Run an Incremental backup. Notice that Reflect will automatically switch to running a Full.  That's because your revised backup selection does not precisely match the selection that was used when that original Full was created, and when you use the "Strict" option, that precise match is required is required for the Full to be considered a "match" and therefore for Reflect to create an Incremental on it.
7. Delete that new Full you created just above. Leave the Full created in Step 4.
8. Edit the settings of the job to change the Backup Set Matching setting to the default Similar.  Keep both TestC and TestD in the backup selection.
9. Try running an Incremental of BackupC again.  This time, Reflect WILL create an Incremental on that original Full.  That's because the "Similar" option only requires that the existing backup and the currently running backup have at least one folder in common, and they do, namely the original TestC folder.  The fact that you added TestD to the backup selection since that original Full is ok when you're using Similar.  The Similar option therefore gives you some flexibility to add or remove folders from your backup selection without having to create a new Full every time you make that kind of change.

Notice that for the first demo I showed you the impact of the backup set matching choice on retention policy behavior, i.e. which backups get deleted, and for the second demo I showed you the impact on the ability to append new Diff/Inc backups to an existing backup.  But both behaviors apply to both cases.  For example, in Demo #1, having Backup Set Matching set to "All" would allow BackupB to append a Diff/Inc backup to the Full that was created by BackupA, even though those are two different jobs that have no backup selection folders in common.  And in Demo #2, keeping the "Strict" setting after changing the backup selection to include both TestC and TestD folder would cause the job's retention policy NOT to act on any existing backups that only contain the TestC folder, because under Strict standards, the backups that only contain TestC would not be considered a "match" to the current backup job that contains both TestC and TestD

Edited 17 May 2019 3:41 PM by jphughan
Mintmag
Mintmag
Proficient Member
Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)
Group: Forum Members
Posts: 152, Visits: 540
Hi jphughan. I can't thank you enough for your most recent post detailing the behavior of the "Backup Set Matching" feature of Macrium Reflect. Your example, has proved to be very helpful in understanding how this feature works. I think this post should be pinned or something. 

If I understand this correctly and correct me if I don't . "Backup Set Matching" determines when and how incremental and differential backups are conducted from a full backup; and is decided by the XML script that runs the backup. Is this correct?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Mintmag - 21 May 2019 9:38 AM
Hi jphughan. I can't thank you enough for your most recent post detailing the behavior of the "Backup Set Matching" feature of Macrium Reflect. Your example, has proved to be very helpful in understanding how this feature works. I think this post should be pinned or something. 

If I understand this correctly and correct me if I don't . "Backup Set Matching" determines when and how incremental and differential backups are conducted from a full backup; and is decided by the XML script that runs the backup. Is this correct?

I'm glad it helped! Smile  To your question, Backup Set Matching determines whether a Differential or Incremental can be appended to an existing backup, though not necessarily just a Full -- remember, Incrementals can be created from a Full, Differential, or Incremental.  You can only append a Differential or Incremental to a "parent" backup that is considered a match based on its contents and the Backup Set Matching setting you chose.  If there's no matching backup at the destination, then Reflect creates a Full even if you asked it to create something else.  Backup Set Matching also (by default) determines which backups at the destination are potential "targets" of the retention policy for purging.  In my first example above demonstrating the impact of the "All" setting, if you had left Backup Set Matching at its default "Similar" setting, then BackupB's retention policy would not have acted on the backups generated by BackupA, even though they were in the same folder.  The reason I said "by default" earlier is that in the backup settings wizard, just above the area where you specify your retention policy settings for each backup type, there's a dropdown option where you can choose to apply the retention policy to all backups in the destination folder rather than all matching backups, but again that isn't the default.

In terms of how matching is decided, for F&F backups the decision about which existing backups at the destination are a match for the currently running job is made by a) checking the Backup Set Matching setting, which is indeed in the XML definition file, in order to determine the "match criteria", and then b) analyzing the contents of the existing backups to see which ones are a match under those criteria.  For image backups, a match is always defined as a backup that contains exactly the same set of partitions and exactly the same partition size.  There's no Backup Set Matching setting to change the "match criteria".  (And on a minor note, the XML file isn't a script that actually runs the way a batch file does.  The XML file is just a configuration file or settings file that describes a particular job.  The reason I mention this is that Reflect also supports generating actual scripts for users who want to have things occur before and/or after the backup.  If you generate an actual script, the portion that runs Reflect is a line that basically says, "Call the Reflect application and have it run the job described in this particular XML file.")

Mintmag
Mintmag
Proficient Member
Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)Proficient Member (231 reputation)
Group: Forum Members
Posts: 152, Visits: 540
jphughan - 21 May 2019 1:28 PM
Mintmag - 21 May 2019 9:38 AM
Hi jphughan. I can't thank you enough for your most recent post detailing the behavior of the "Backup Set Matching" feature of Macrium Reflect. Your example, has proved to be very helpful in understanding how this feature works. I think this post should be pinned or something. 

If I understand this correctly and correct me if I don't . "Backup Set Matching" determines when and how incremental and differential backups are conducted from a full backup; and is decided by the XML script that runs the backup. Is this correct?

I'm glad it helped! Smile  To your question, Backup Set Matching determines whether a Differential or Incremental can be appended to an existing backup, though not necessarily just a Full -- remember, Incrementals can be created from a Full, Differential, or Incremental.  You can only append a Differential or Incremental to a "parent" backup that is considered a match based on its contents and the Backup Set Matching setting you chose.  If there's no matching backup at the destination, then Reflect creates a Full even if you asked it to create something else.  Backup Set Matching also (by default) determines which backups at the destination are potential "targets" of the retention policy for purging.  In my first example above demonstrating the impact of the "All" setting, if you had left Backup Set Matching at its default "Similar" setting, then BackupB's retention policy would not have acted on the backups generated by BackupA, even though they were in the same folder.  The reason I said "by default" earlier is that in the backup settings wizard, just above the area where you specify your retention policy settings for each backup type, there's a dropdown option where you can choose to apply the retention policy to all backups in the destination folder rather than all matching backups, but again that isn't the default.

In terms of how matching is decided, for F&F backups the decision about which existing backups at the destination are a match for the currently running job is made by a) checking the Backup Set Matching setting, which is indeed in the XML definition file, in order to determine the "match criteria", and then b) analyzing the contents of the existing backups to see which ones are a match under those criteria.  For image backups, a match is always defined as a backup that contains exactly the same set of partitions and exactly the same partition size.  There's no Backup Set Matching setting to change the "match criteria".  (And on a minor note, the XML file isn't a script that actually runs the way a batch file does.  The XML file is just a configuration file or settings file that describes a particular job.  The reason I mention this is that Reflect also supports generating actual scripts for users who want to have things occur before and/or after the backup.  If you generate an actual script, the portion that runs Reflect is a line that basically says, "Call the Reflect application and have it run the job described in this particular XML file.")

Oh right. I think I understood most of that. Though there is something that does trouble me. If the "Backup set Matching" is set to all and the Define Retention Rules is also set to "Apply retention rules to all backup sets in the same folder" isn't that basically the same thing?

Backup set matching determines how the current job matches other jobs in the destination and applies retention rules, as well as differential/incremental jobs accordingly. Is this correct? 

Edited 28 May 2019 9:28 PM by Mintmag
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search