Backup scheduling advice to minimise HD space


Author
Message
TomP
TomP
Junior Member
Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)
Group: Forum Members
Posts: 61, Visits: 151
Having discussed this as part of another thread, jphughan suggested I open a separate topic for further discussion, having advised that Synthetic Full and Incrementals Forever was a risky process and sent me this  alternative suggestion...


To me it looks like there is a big gap not backed up, so it would help me to fully understand the relation / dates of the Incrementals in between the two Fulls
It would also help to understand what happens as the third new Full is backed up and the oldest full is removed - do I end up temporarily with 3 Fulls ( if the first is only deleted once the 3rd is validated) or temporarily with just the 2nd full as the first is deleted before the 3rd is completed?

Thanks in advance
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
The configuration above works out as follows:
  1. On Day 1 of a new month, I get a Full.
  2. On Days 2-8, I get a new Incremental.
  3. On Day 9, I get a new Incremental and my Day 2 Incremental gets merged out.  So now I have backups from Day 1 and Days 3-9.
  4. This carries on through the rest of the month, i.e. on any given day in the month, I have daily Incrementals going back for the last 7 days, plus the Full from the first day of the month.  So yes, in that sense there is a gap between my Full and my oldest remaining Incremental.
  5. On Day 1 of the SECOND month, I get another Full.
  6. On Day 2 of that next month, I get a new Incremental.  At this point I still have 6 Incrementals that are children of the PREVIOUS month's Full, which will be from the last 6 days of that previous month.
  7. As the new month progresses, I get a new Incremental each day and lose one Incremental from the previous month until all 7 of my Incrementals are children of my current month's Full.
  8. The concept of Step 4 repeats through this month.
  9. On Day 1 of the THIRD month, I get a new Full.  On that day or possibly a bit later, the Full from the first month will be deleted.  It depends on how many days are in the months involved.  If the first month was February and only and only had 28 days, for example, then at the end of March, the Full from February will only be 59 days old.  In that case, the February Full will get deleted when one of the daily Incrementals runs.
If you want to eliminate the "time gap" that occurs as you progress through the week, i.e. that your Full is from Day 1 of that month but toward the end of the month your oldest Incremental might be from Day 22 of that month, then you can enable Synthetic Fulls.  But the gap doesn't bother me, and since I replicate my backups to a secondary destination, I didn't want to have to replicate an updated Full backup each day

TomP
TomP
Junior Member
Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)
Group: Forum Members
Posts: 61, Visits: 151
Many thanks,
That helps make things a lot clearer. Just a few questions  (>>>...) on some of your points....

On Day 1 of a new month, I get a Full.
>>> Is there a way to seamlessly transition into this from my current Incrementals Forever + synthetic Full? or do I have to throw old backups away?
>>> does it make sense to start my 4 PCs on the 1st, 8th, 15th and 22nd of each month?

On Days 2-8, I get a new Incremental.
>>> can the first date be the 2nd rather than the first without making any difference?
>>> how many days incrementals can I retain without clashing with a next full at the end of Feb - 27 ? I was going to do 30 incrementals forever so would like to get close to that

If you want to eliminate the "time gap" that occurs as you progress through the week, i.e. that your Full is from Day 1 of that month but toward the end of the month your oldest Incremental might be from Day 22 of that month, then you can enable Synthetic Fulls. 
>>> would enabling Synthetic Full be a temporary manual schedule edit? or how would that be scheduled?

On Day 1 of the THIRD month, I get a new Full. On that day or possibly a bit later, the Full from the first month will be deleted. It depends on how many days are in the months involved. If the first month was February and only and only had 28 days, for example, then at the end of March, the Full from February will only be 59 days old. In that case, the February Full will get deleted when one of the daily Incrementals runs.
>>> in longer months, at what exact point is the first Full deleted after the third is created?
>>>  I'd like to avoid having third fulls lasting for any length of time, not just 3 days from Feb, but also 1 days more for 30 day months, so is the a sensible strategy for avoiding this?

Finally, more of an enhancement request. It seems it would help if scheduling options could permit setting on at 4-week rather than monthly cycle, then the issue of February or other month-length variations could be lessened. In almost all large companies for whom I've worked, their IT systems do work on 13 4-week periods to make things simpler.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
If you want to transition to the setup I have, just reconfigure your current schedule and retention policy to match what I've posted and either wait for the beginning of the month to get a new Full or create a new Full manually to start it mid-month, after which the schedule will make sure it runs normally afterward since the next Full will still be created at the beginning of the following month even though you manually created one mid-month.  You don't have to delete anything first.  As for start dates, you can pick whatever you want.  I picked the first day of the month just because it's the first of the month, but that is of course ultimately arbitrary.  You can make your monthly Full on whatever day of the month you want.  Or you can define your Full schedule such that it occurs on (for example) the first Saturday of the month, whatever day that happens to be within the month.  Or the third Thursday.  Whatever you want.  You can also of course create more than one Full per month.  It depends entirely on what you want to do.  You can run this with whatever frequency and quantity of Fulls and Incrementals you want.  The only restriction is that you want to use Synthetic Fulls, which I personally don't, you can't use Diffs at all.

To my knowledge there's no technical limit on the number of Incrementals you can have chained to a particular Full, so if you want to retain 30 Incs all on a single Full, that's fine.  Just keep in mind that if you do that, the ability to restore from a backup at the end of the month will rely on you having about 30 preceding backups all available and intact.  Some users might not want to have a "dependency chain" that long.  But I know that others are fine with it and have been using it successfully.

If you want to use Synthetic Fulls, you have to specify your Incremental retention policy in terms of number of backups, not a time period.  If you do that, you'll see a Synthetic Full checkbox appear underneath that dropdown, which is hidden in my screenshot because I'm using time.  Macrium has said there's no technical reason they couldn't support Synthetic Fulls with time-based Incremental retention policies, but it's just not implemented at the moment.

If you want to avoid having 3 Fulls for any period, set your Full retention policy to 2 backups rather than using time as I did, although I'd still recommend that you keep "Run purge before backup" unchecked.  In that case you'll have 3 Fulls for as long as it takes to successfully create the third Full, but once it succeeds, the first Full would be purged immediately.  However, this also means that if you ever wanted to create a new Full manually for some reason, that would cause an earlier than expected purge.  Time-based retention policies allow you to create additional "on-demand" backups without causing purges to occur earlier than they would have otherwise.

In terms of your enhancement request, what you're asking is already possible.  If you set a Weekly schedule, there's an "Every" field.  If you set that to 4, the schedule will run every 4 weeks, not every week. That would then give you the option of defining your Full retention as 56 days (28 x 2) rather than 2 backups.
Edited 2 December 2020 1:08 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
Quick add-on note: The reason I set my Full retention to be based on time is because when making image backups of a disk that contains a Windows 10 installation, you can end up with Full backups unexpectedly.  The reason is that Windows 10 feature updates can sometimes alter your partition layout to accommodate a need for a larger Recovery partition, and that type of change will require a Full backup afterward.  Reflect will automatically make a Full even if you ask for an Incremental in that situation, so your backups will still work, but if you specify retention policies in terms of number of backups, that unexpected Full can cause your oldest Full to get purged earlier than would have occurred according to your schedule.

But if you decide to schedule a Full for every 4 weeks rather than every month, then the best option would be to set your Full retention policy to 8 weeks (assuming you want retain 2 under normal circumstances).  As you said earlier, the issue with the "once per month" setup is that you'll sometimes get stuck holding onto 3 Fulls longer than needed because not all months have the same number of days and Reflect doesn't have a "month" option in its time-based retention options.  But if both your schedule and retention policy use weeks, then each time your "once every 4 week" Full schedule runs, it will delete the oldest Full right away, not a few days later -- AND any unexpected Fulls will NOT cause old Fulls to be purged earlier than expected.

Edited 2 December 2020 6:11 PM by jphughan
TomP
TomP
Junior Member
Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)
Group: Forum Members
Posts: 61, Visits: 151
jphughan - 2 December 2020 6:09 PM
Quick add-on note: The reason I set my Full retention to be based on time is because when making image backups of a disk that contains a Windows 10 installation, you can end up with Full backups unexpectedly.  The reason is that Windows 10 feature updates can sometimes alter your partition layout to accommodate a need for a larger Recovery partition, and that type of change will require a Full backup afterward.  Reflect will automatically make a Full even if you ask for an Incremental in that situation, so your backups will still work, but if you specify retention policies in terms of number of backups, that unexpected Full can cause your oldest Full to get purged earlier than would have occurred according to your schedule.

But if you decide to schedule a Full for every 4 weeks rather than every month, then the best option would be to set your Full retention policy to 8 weeks (assuming you want retain 2 under normal circumstances).  As you said earlier, the issue with the "once per month" setup is that you'll sometimes get stuck holding onto 3 Fulls longer than needed because not all months have the same number of days and Reflect doesn't have a "month" option in its time-based retention options.  But if both your schedule and retention policy use weeks, then each time your "once every 4 week" Full schedule runs, it will delete the oldest Full right away, not a few days later -- AND any unexpected Fulls will NOT cause old Fulls to be purged earlier than expected.

That sounds like a plan, thanks !! The only question then is when to start incrementals (day 1 or 2?) and  for how many days 26 or 27?) within the 4-week window?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
I'd schedule your Full to occur every 4 weeks starting whatever day you want.  Then for your Incremental schedule, just create a daily schedule set to start 1 day after the first Full, and schedule the Incremental for the same time of day as the Full.  That way, on days where both a Full and Incremental are scheduled, Reflect will ignore the Incremental and only make a Full.  In terms of retention, set Full retention to 8 weeks (again assuming you want 2 Fulls normally) and Incremental retention to however many days or weeks you want to retain Incrementals.  Or if you want to use Synthetic Fulls and therefore have to specify Incremental retention in number of backups, based on what you've described thus far it sounds like you'd want 27 Incrementals, which will give you 4 weeks' worth of Incrementals (since the first backup in your 28-day/4-week cycle will be a Full).  Then uncheck "Run purge before backup" so that you don't purge the old Full until you finish creating a new one, thereby making sure you have 2 good Fulls at all times.

And then if your image backup includes the Windows partition, I'd set the retention policy matching setting to apply to all backups in the destination, not all "matching backups".  Otherwise if you ever get one of those unexpected Fulls after a feature update, Reflect will stop purging the pre-update backups because they'll no longer be a match.  Just make sure that you're sending backups to a folder that is storing ONLY backups generated by this particular job.  Do not create multiple completely unrelated backup jobs and store them all in the same folder, otherwise changing that matching setting can cause Job A to purge unrelated backups created by Job B.

Edited 2 December 2020 8:38 PM by jphughan
TomP
TomP
Junior Member
Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)
Group: Forum Members
Posts: 61, Visits: 151
Great and comprehensive info, thanks !! Fortuitously, you also answered many questions / potential problems that I didn't even know to ask !!

Just one point of understanding. As I would be creating two Fulls and 27 synthetics, this is more than would be provided by Incrementals Forever, so do I actually need Synthetic fulls at all?

Presumably having those 2 unchanged Fulls, fixed / validated at the time of creation, would be less likely to fail than a Synthetic Full that is constantly recreating all the time?

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
Happy to help!  In terms of Synthetic Fulls, I had to sit down and look at a calendar for a little while to make sure I was simulating each scenario correctly before writing this answer.  But basically, with your setup of (approximately) monthly Fulls, daily Incrementals in between, and wanting to hold onto (approximately) a month's worth of Incrementals, it would indeed be better NOT to use Synthetic Fulls.  This then means you can set your Incremental retention policy to either 4 weeks or 27 backups.  If your schedule proceeds normally, the decision won't make a practical difference.  But it can make a difference in abnormal scenarios.  For example, if you want to create an "on-demand" Incremental, time-based might be preferable so that you don't cause early purges.  On the other hand, if you were to go on vacation for a week during which no backups occurred, then a time-based retention policy would cause a week's worth of backups to be purged when you only got one new backup upon your return.  That wouldn't happen with a retention policy based on number of backups.

But if you're curious, if you DID use Synthetic Fulls with your schedule, then in addition to the Full needing to be updated with each backup as you said, the fact that it was being "carried forward" with each backup would mean that it would essentially stay ahead of your time-based retention policy, when you specifically want it to "age" as a result of being kept static.  With Synthetic Fulls enabled, there would be a point at which your oldest Full would have been "carried forward" right up to the day before your next oldest Full (because all existing Incrementals at that point would be children of the newer Full) -- and then your second Full would itself start getting carried forward, leaving your first Full behind.  You could make that work, but with a time-based retention policy if you only wanted to have 2 Fulls, you'd have to make some adjustments to account for the carrying forward.

But again, in your setup I agree it's better to keep Synthetic Fulls disabled.  This variation is called "Incremental Merge", since instead of always merging the oldest Incremental into the Full to create a Synthetic Full, the oldest TWO Incrementals are merged with each other so that the Full can stay frozen.

And just to make sure you understand how your Incremental Merge setup will play out, I've laid it out below.  To understand WHY it will work like this, it's important to know that a retention policy of 27 Incrementals would mean 27 Incrementals total, not 27 Incrementals per Full.  (If you choose to use 4 weeks instead, it will work as I'm about to describe during normal circumstances.)

This example assumes you began your strategy with a Full created this past Monday, 30 November 2020:
  • On 30 November, you create your first Full.  Then you'd accumulate daily Incrementals up through Sunday, 27 December.
  • On 28 December, you create your second Full.
  • Fast forward to 11 January 2011, just after the backup for that day completed.
    • WITHOUT Synthetic Fulls enabled, at this point you would still have your Full from 30 November, and your oldest remaining Incremental would be from 15 December.  So you'll have a "time gap" of 2 weeks between your oldest Full and your oldest Incremental.  And since you're halfway through the month, you'll have half your Incrementals as children of the newer Full, and half as children of the older Full.  But in aggregate, you will have daily backups for the last 4 weeks, i.e. 15 December through 11 January, plus that old Full from 30 November.
    • WITH Synthetic Fulls enabled, the status at this point would be the same as above except your oldest Full would have been carried forward to 14 December rather than being frozen at 30 November, thus eliminating that "time gap" to your oldest remaining Incremental.
  • On 25 January, you get your third Full.  By this point, there will be no more Incrementals that are children of your first Full (they'll all be children of the 28 December Full), and the first Full from 30 November will get purged -- assuming you have Synthetic Fulls disabled.  And from there, the cycle will continue.  Each new Incremental created under the new 25 January Full will result in the oldest remaining Incremental from the 28 December Full being purged.

Edited 3 December 2020 12:00 AM by jphughan
TomP
TomP
Junior Member
Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)Junior Member (71 reputation)
Group: Forum Members
Posts: 61, Visits: 151
Many thanks !!
I am also in the process of installing Site-Manager and have Reflect my first 2 PCs installed locally, so will I be able to set up a single standard Site Manager process that can be used to replace the schedules each of these PCs and their individual HDs, with just the starting date changed?
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search