"Grandfather-Father-Son" Backup Schedule Template: Help Needed


Author
Message
Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
I currently run a "Grandfather-Father-Son" backup schedule, most aspects of which have been left at the default settings.

The template is scheduled to run daily at 10:00 (24-hr Clock), as follows:

Full:    10:00 1st Sun of every Month
Diff:    10:00 1st Sun of every Week
Incr:    10:00 Every Day

The retention rules are as follows (Macrium Defaults):

Full:    Retain for 26 Weeks (linked Incr & Diff images will also be deleted)
Diff:    Retain for 4 Weeks (linked Incr images will also be deleted)
Incr:    Retain for 14 Days (oldest Incr images may be consolidated)

I have now been running this template since the 1st April 2020, (2020-04-01). This was not a Sunday, (the 1st Sunday of the month having been 2020-04-05), but I wanted to begin the whole series on the 1st April 2020.

On the 1st Sunday of May 2020, (2020-05-03), Macrium Reflect performed a new Full Backup, with a new file name, whilst also retaining the original Full Backup, along with its linked Diff and Incr images.

I now find that I have two sets of Diff and Incr images, and that the original Diff and Incr images are gradually being either purged or consolidated, whilst new Diff and Incr images are simultaneously being created and linked to the new Full Image.

The awful realisation has now dawned on me that by the time 26 weeks have passed, I could well have amassed 6 complete backup sets, (Full + Diff + Incr), on the same drive, and I am now worried that the amount of data will have almost grown beyond the capacity of my backup drive.

In the light of the situation outlined above, I have two questions:

1. Is it really necessary to retain a Full Backup for 26 weeks, considering that, (according to the template), a new Full Backup is being created monthly anyway?

2. If the answer to Question 1 is "No," would it be advisable for me to alter the retention rules for Full Backup and retain Full Images for 4 weeks only, and if not, what would be the best way to go about it?

My ideal situation is to have a Full Image to be created monthly on the 1st Sunday, with a linked Differential Image to be created weekly on each subsequent Sunday, and a linked Incremental Image to be created daily until the time comes for the creation of a new Full Image.

I simply cannot see why I would need to retain more than one complete Grandfather-Father-Son Backup Set, and I am hoping that somebody here might be able to offer me some suggestions as to how I could revise this plan in such a way as to avoid creating superfluous backup sets.


Best regards to all,
Christopher Souter
(Sydney, Australia)

Edited 10 May 2020 3:17 AM by Christopher Souter
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)
Group: Forum Members
Posts: 7K, Visits: 51K
Whether a given amount of retention is "necessary" doesn't have a universal answer.  That said, I would strongly recommend that you retain at least 2 Full backups, not just one complete set, for the simple reason that if you only have one set and even a portion of your Full backup becomes unreadable or corrupt, then all of your backups become useless.  Having at least one other Full gives you another backup (or backup set, if that Full also has child backups) to recover from.  Even better would be to have multiple sets stored in multiple locations, either by using a disk rotation or by replicating backups to another location, but that's a somewhat different topic.

In any case, you may well decide that you don't need to retain monthly Fulls for 26 weeks.  If you simply don't have the capacity, then you don't have the capacity.  But if you don't ever expect you'd need to back that far in time to recover an old version of a file or a file that you deleted long ago, then that might be fine.  You might want to consider whether you might be putting your destination storage capacity to better use by performing weekly Fulls and daily Incs (with no Diffs) in order to get more frequent Fulls.  That does mean that retaining a given length of backup "history" will cost you more storage, but the advantage is that if you ever have a damaged Full, then your next most recent usable backup will only be a week older rather than a month older.  So if you have the capacity to retain weekly Fulls going back as long as you think you'd ever want to "reach back in time", then that might be an alternative strategy worth considering.

But whatever you do, I would indeed suggest that you configure a retention policy that is actually achievable based on the size of your backups and the amount of capacity you have.  Reflect does have the option to purge older backup sets when free space at the destination drops below a threshold that you specify, and that can be executed mid-job if required in order to free up space, but that's sort of a "blunt instrument" because it will purge an entire older SET, i.e. a Full and ALL of its child backups.  And in some strategies, you might not want your oldest backups to be purged first.  For example, I have a strategy of creating monthly Fulls and daily Incs, and my retention specifies to retain 2 Fulls and 7 Incs.  So within a given month, the Incs created earlier in that month will get purged BEFORE the Full created the PREVIOUS month.  My rationale here is that if I have a restore situation where having choices at a daily "granularity" is useful, chances are I don't want to go back very far in time.  But I might also want an older backup if I want to go farther back in time, although in that case it may not matter precisely what older day the backup in question was captured.  The other reason for this design is that it saves space.  Capturing monthly Fulls and daily Incs while only retaining the most recent week's worth of Incs means I don't need as much space.

In terms of "your ideal situation", you can certainly schedule monthly Fulls, weekly Diffs, and daily Incs -- but that doesn't mean you always need to retain the entire set.  As I just alluded to above, you don't necessarily have to delete backups in the order in which they were created.  Just because you create daily Incs doesn't mean you need to retain every single daily Inc you've created since the oldest one you still have.  As with many things in life, there are pros and cons to various approaches, and different strategies will make sense for different use cases.  But if you give some thought to how far back in time you need to be able to go, whether you might want more frequent Fulls to protect against the scenario I described, and whether you really need to retain every "lower order" backup (Diff/Inc) you create until you delete their parent Full, I'd be happy to help you implement a different schedule and/or retention policy to meet your revised goals. Smile

Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
Many thanks for your quick reply. Smile

You have given me a great deal of food for thought, and it is well within the realms of possibility that I may take you up on your offer of assistance, and I'll get back in touch during the next few days.

Thanks once again.

Best regards to all,
Christopher Souter
(Sydney, Australia)

Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
jphughan - 10 May 2020 4:18 AM
Whether a given amount of retention is "necessary" doesn't have a universal answer.  That said, I would strongly recommend that you retain at least 2 Full backups, not just one complete set, for the simple reason that if you only have one set and even a portion of your Full backup becomes unreadable or corrupt, then all of your backups become useless.  Having at least one other Full gives you another backup (or backup set, if that Full also has child backups) to recover from.  Even better would be to have multiple sets stored in multiple locations, either by using a disk rotation or by replicating backups to another location, but that's a somewhat different topic.

In any case, you may well decide that you don't need to retain monthly Fulls for 26 weeks.  If you simply don't have the capacity, then you don't have the capacity.  But if you don't ever expect you'd need to back that far in time to recover an old version of a file or a file that you deleted long ago, then that might be fine.  You might want to consider whether you might be putting your destination storage capacity to better use by performing weekly Fulls and daily Incs (with no Diffs) in order to get more frequent Fulls.  That does mean that retaining a given length of backup "history" will cost you more storage, but the advantage is that if you ever have a damaged Full, then your next most recent usable backup will only be a week older rather than a month older.  So if you have the capacity to retain weekly Fulls going back as long as you think you'd ever want to "reach back in time", then that might be an alternative strategy worth considering.

But whatever you do, I would indeed suggest that you configure a retention policy that is actually achievable based on the size of your backups and the amount of capacity you have.  Reflect does have the option to purge older backup sets when free space at the destination drops below a threshold that you specify, and that can be executed mid-job if required in order to free up space, but that's sort of a "blunt instrument" because it will purge an entire older SET, i.e. a Full and ALL of its child backups.  And in some strategies, you might not want your oldest backups to be purged first.  For example, I have a strategy of creating monthly Fulls and daily Incs, and my retention specifies to retain 2 Fulls and 7 Incs.  So within a given month, the Incs created earlier in that month will get purged BEFORE the Full created the PREVIOUS month.  My rationale here is that if I have a restore situation where having choices at a daily "granularity" is useful, chances are I don't want to go back very far in time.  But I might also want an older backup if I want to go farther back in time, although in that case it may not matter precisely what older day the backup in question was captured.  The other reason for this design is that it saves space.  Capturing monthly Fulls and daily Incs while only retaining the most recent week's worth of Incs means I don't need as much space.

In terms of "your ideal situation", you can certainly schedule monthly Fulls, weekly Diffs, and daily Incs -- but that doesn't mean you always need to retain the entire set.  As I just alluded to above, you don't necessarily have to delete backups in the order in which they were created.  Just because you create daily Incs doesn't mean you need to retain every single daily Inc you've created since the oldest one you still have.  As with many things in life, there are pros and cons to various approaches, and different strategies will make sense for different use cases.  But if you give some thought to how far back in time you need to be able to go, whether you might want more frequent Fulls to protect against the scenario I described, and whether you really need to retain every "lower order" backup (Diff/Inc) you create until you delete their parent Full, I'd be happy to help you implement a different schedule and/or retention policy to meet your revised goals. Smile

As promised, here is my new backup definition file, and shown in Macrium Reflect.
I see no benefit in my situation to retain Full backups for 26 weeks, as all I need is the latest Full Backup, along with all its linked Differential and Incremental Backups.
The screenshot below shows the details.
As a result, I have now saved myself a considerable amount of space on my backup drive.
I am quite happy with this arrangement, but please feel free to make any comments you may feel necessary.



Best regards to all,
Christopher Souter
(Sydney, Australia)

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)
Group: Forum Members
Posts: 7K, Visits: 51K
I see some possible opportunities for improvement here, as well as some possible risks that should be checked:
  • Since you seem to be thinking about your needs in terms of number of backups, then why not set your retention policy that way?  You can set your Full retention policy to a certain number of backups rather than a certain number of days.  You can even mix and match these, e.g. saying "I want to retain X number of Fulls, but Y days' worth of Diffs and Incs."  Both are fine, and there are pros and cons to each when it comes to anomalous situations, such as backups that get missed due to your system being off vs. backups you might create "ad hoc" for special circumstances.
  • Setting the same retention period for all of your backup types is redundant.  When your Full gets purged, all of its child Diff and Inc backups get purged too.  There's only value in specifying a retention period for Diffs and Incs if you'll be setting them to a shorter period than their parent backup types.  Otherwise you may as well just uncheck the Diff and Inc retention policies and let them get deleted as a result of their parent Full getting deleted, which is what will happen anyway. (EDIT: Removed some incorrect information that I originally wrote here.  Apologies!)
  • You say that all you need is the latest Full backup, but with Fulls being captured every week and retained for 28 days, you're going to have 4 Full backups at any given time.  That said, for reasons I already gave in my earlier post, I would strongly recommend that you retain at least 2 Full backups at any given time, not just 1.  And if you decide to set a retention policy that will achieve that, i.e. either 2 Full backups or 14 days, then I would uncheck "Run purge before the backup".  Otherwise, Reflect will delete the oldest Full before it even starts creating the new one, which means that until that new Full completes, you'll only have one usable Full.
  • Your backup job is set to back up only the C partition.  Is that truly the only partition on your disk?  If there are other hidden partitions, you should absolutely include those in your backup, because on most systems these days, having just the C partition is not enough to get a bootable system if you ever have to restore onto an empty disk.  In general when backing up a disk that contains an operating system, you should only exclude partitions that you have a specific reason for excluding and that you know are safe to exclude, NOT include only the partitions that you think are necessary or important to you.  If you post a screenshot of your disk's partition map as shown in Reflect's "Create a backup" tab, I may be able to give you more precise guidance.  Or you can go there yourself and then in the Backup Tasks area in the upper-left corner, click "Create an image of the partitions necessary to back up and restore Windows."  Whatever partitions are pre-selected in the image backup wizard that appears should be considered your "minimum viable backup" for system restore purposes.  You can add partitions to that selection if you want, but you should NOT subtract any.  So if you see any additional partitions pre-selected there, you should edit your definition file to include those.  That will require you to create a new Full.

Edited 14 May 2020 1:58 AM by jphughan
Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
jphughan - 14 May 2020 1:47 AM
I see some possible opportunities for improvement here, as well as some possible risks that should be checked:
  • Since you seem to be thinking about your needs in terms of number of backups, then why not set your retention policy that way?  You can set your Full retention policy to a certain number of backups rather than a certain number of days.  You can even mix and match these, e.g. saying "I want to retain X number of Fulls, but Y days' worth of Diffs and Incs."  Both are fine, and there are pros and cons to each when it comes to anomalous situations, such as backups that get missed due to your system being off vs. backups you might create "ad hoc" for special circumstances.
  • Setting the same retention period for all of your backup types is redundant.  When your Full gets purged, all of its child Diff and Inc backups get purged too.  There's only value in specifying a retention period for Diffs and Incs if you'll be setting them to a shorter period than their parent backup types.  Otherwise you may as well just uncheck the Diff and Inc retention policies and let them get deleted as a result of their parent Full getting deleted, which is what will happen anyway. (EDIT: Removed some incorrect information that I originally wrote here.  Apologies!)
  • You say that all you need is the latest Full backup, but with Fulls being captured every week and retained for 28 days, you're going to have 4 Full backups at any given time.  That said, for reasons I already gave in my earlier post, I would strongly recommend that you retain at least 2 Full backups at any given time, not just 1.  And if you decide to set a retention policy that will achieve that, i.e. either 2 Full backups or 14 days, then I would uncheck "Run purge before the backup".  Otherwise, Reflect will delete the oldest Full before it even starts creating the new one, which means that until that new Full completes, you'll only have one usable Full.
  • Your backup job is set to back up only the C partition.  Is that truly the only partition on your disk?  If there are other hidden partitions, you should absolutely include those in your backup, because on most systems these days, having just the C partition is not enough to get a bootable system if you ever have to restore onto an empty disk.  In general when backing up a disk that contains an operating system, you should only exclude partitions that you have a specific reason for excluding and that you know are safe to exclude, NOT include only the partitions that you think are necessary or important to you.  If you post a screenshot of your disk's partition map as shown in Reflect's "Create a backup" tab, I may be able to give you more precise guidance.  Or you can go there yourself and then in the Backup Tasks area in the upper-left corner, click "Create an image of the partitions necessary to back up and restore Windows."  Whatever partitions are pre-selected in the image backup wizard that appears should be considered your "minimum viable backup" for system restore purposes.  You can add partitions to that selection if you want, but you should NOT subtract any.  So if you see any additional partitions pre-selected there, you should edit your definition file to include those.  That will require you to create a new Full.

Here is my latest Backup Definition file, modified to accommodate some, but not all, of your recommendations.
Point 1: The C: Partition is the only partition on this HDD, which appears as "Disk 0" in the Disk Management Console.  Please note that the Full Backup is scheduled to run only on the 1st Sunday of every month, not weekly as you stated.  The Differentials are set to run every Sunday, and the Incrementals are set to run every day.  From my previous experience with the scheduler, if a Full and a Differential are set to run on the same day, the Full would appear to take precedence, and the Differential does not run.  In the same way, if a Differential and an Incremental are set to run on the same day, the Differential will take precedence and run, and the Incremental will not run.  This would appear to be some kind of quirk with the scheduling module, but I was pleased to see that almost the whole of every Sunday was not going to be taken up with 2 or 3 different Backups all running either concurrently or sequentially.  I took your advice about backup retention policy, and removed the Differentials and Incremental therefrom. 
Point 2: I also have a D: Partition, but that is on a spanned volume, spread over a total of 3 HDDs, which appear as "Disk 1," "Disk 2," and "Disk 3" in the Disk Management Console. I use a different backup method with this volume, mainly because I once had trouble restoring a full image to a new disk arrangement, (I added a new HDD to increase capacity), and I found that it is easier for me to do a File and Folder Backup of this volume, because I can then restore the files and folders without getting strange and incomprehensible errors about volume size, etc.  I actually wanted to add a link to the forum discussion about these errors, but I have been unable to find it, and I think it's probably to be found on the old support forum.  If I can find the link, I'll post it here, but in summary, no solution was ever found. 
A screenshot of my definition file appears below:



The new scheme is set to commence from tomorrow, and I am waiting with bated breath to see the full result of the latest rejig by the end of next Monday's backup!


Best regards to all,
Christopher Souter
(Sydney, Australia)

Edited 15 May 2020 12:58 AM by Christopher Souter
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)
Group: Forum Members
Posts: 7K, Visits: 51K
My mistake on misreading your Full schedule before.  But you now have your retention policy set to retain only one Full backup, AND you have "Run purge before backup" enabled.  That means that whenever Reflect is scheduled to create a Full, it will delete all of your existing backups before even starting to create that new Full.  So there will be times during which you have no backups whatsoever.  That is obviously a very high-risk scenario.  This occurs because when Reflect evaluates its retention policy, the current backup is counted, and if you have "Run purge before backup" enabled, that means Reflect is counting a backup that has not yet been created.

If you're adamant about keeping only keeping one backup set, then at the very least uncheck "Run purge before backup" so that the previous backups don't get purged until the new Full was actually created successfully.  But even then, after the first Sunday of the month, the ONLY backup you will have will be that one Full that was just created, so if you want to retrieve data from even the previous day, you will be out of luck.

I'll say one last time that having only one backup set is highly inadvisable.  But if you're adamant, then I've done my best to warn you.

You're correct that when multiple backup types for the same backup job are scheduled to occur at the same time, then Reflect will only perform the "highest order" backup.  That is not a quirk; that was a deliberate design choice on Macrium's part.  Windows Task Scheduler still sends all of those tasks to Reflect, but it chooses to ignore the lower order backup job(s).

Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
jphughan - 15 May 2020 1:47 AM
My mistake on misreading your Full schedule before.  But you now have your retention policy set to retain only one Full backup, AND you have "Run purge before backup" enabled.  That means that whenever Reflect is scheduled to create a Full, it will delete all of your existing backups before even starting to create that new Full.  So there will be times during which you have no backups whatsoever.  That is obviously a very high-risk scenario.  This occurs because when Reflect evaluates its retention policy, the current backup is counted, and if you have "Run purge before backup" enabled, that means Reflect is counting a backup that has not yet been created.

If you're adamant about keeping only keeping one backup set, then at the very least uncheck "Run purge before backup" so that the previous backups don't get purged until the new Full was actually created successfully.  But even then, after the first Sunday of the month, the ONLY backup you will have will be that one Full that was just created, so if you want to retrieve data from even the previous day, you will be out of luck.

I'll say one last time that having only one backup set is highly inadvisable.  But if you're adamant, then I've done my best to warn you.

You're correct that when multiple backup types for the same backup job are scheduled to occur at the same time, then Reflect will only perform the "highest order" backup.  That is not a quirk; that was a deliberate design choice on Macrium's part.  Windows Task Scheduler still sends all of those tasks to Reflect, but it chooses to ignore the lower order backup job(s).

On further consideration, I think you might well be right about the "Purge" option., and I'll uncheck it now.
My reason for limiting the number of Full Backups is that I have limited space on my backup disk, and I can't afford to buy another disk at the moment, and I am also using that disk to store a number of other items until such time as I can afford to purchase another large HDD to add to my "Drive D:" spanned volume.
So, you see, in a nutshell, the main problem is, (as with many of us, I fear), lack of money!
My eventual plan is to build a totally new machine based around Windows 10, (I'm still running Windows 7, because I don't want to give up Windows Media Center), and I am slowly amassing the necessary hardware for this task.
When I finally manage to build this machine, I will have plenty of disk space, and running, say, two Full Backup sets will not be a problem.
BTW, thanks for the info about the scheduler.  I was thinking that the explanation might be something along those lines, but it's nice to hear that my suspicions have been confirmed.
May I say a big "thank you" to you for all your patient and diligent attempts to point me in the right direction!
I have to say that Macrium Reflect is no more difficult to configure than most other backup programs that I have used over the years, but I'm afraid to have to say that I have never come across any backup program that is by any means simple to configure, and Macrium Reflect is certainly not the most difficult that I have ever encountered.


Best regards to all,
Christopher Souter
(Sydney, Australia)

Edited 15 May 2020 10:07 AM by Christopher Souter
Christopher Souter
Christopher Souter
New Member
New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)New Member (29 reputation)
Group: Forum Members
Posts: 19, Visits: 34
FYI, definition has been adjusted WRT automatic purging policy.
Final Definition Settings Screen, schedule starts today:


Best regards to all,
Christopher Souter
(Sydney, Australia)

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)
Group: Forum Members
Posts: 7K, Visits: 51K
Ok, as long as you're ok with the fact that after your monthly Full completes, that will be the ONLY backup that you have available, then that looks fine.  And are you deliberately not running a Saturday Inc?  Originally you said you wanted an Incremental every day.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search