Schedule retention policy enforcement


Author
Message
Obl918
Obl918
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 9, Visits: 13
I've spent some time setting up Site Manager with five workstations and a server. So far, things are working well and seem to be what I need.

However, I do not see a way to do this:

I have Incremental Forever being used on all of my backups.  But I do not want synthetic full backups to be created six times a day every day (once the incremental retention limit is reached on each PC). What I want is to enforce the retention policy on a weekly basis. I'd like to set it up so that during the day on Sunday, after the backups have all be done overnight, synthetic full backups are updated and the oldest incrementals are deleted to comply with the retention policy.

Of course, I may not understand how to do it, or maybe it cannot be done.

And, it is not clear to me yet if a PC has to be online in order to update the retention for it in the repository. I should think not: Site Manager should be able to merge incrementals or create synthetic full backups without a PC being online, shouldn't it?  I ask this because Reflect Agent is not sophisticated enough to wake a PC to comply with the backup policy I've set for it. So, I have to resort to setting up my own task on each machine to have it wake up at night, hopefully at a time that allows the PC to be found by Site Manager and the backup started. I really wish this could be automated with Reflect Agent, as it is by the WS Essentials backup system that I am migrating away from... 


Edited 29 January 2018 7:29 PM by Obl918
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)
Group: Forum Members
Posts: 2.4K, Visits: 16K
The retention policy is evaluated with every execution of a job, and I don't believe there's a direct way to override that.  However, you could achieve what you want by creating two separate jobs, something like this:

- Modify your current job so that it runs on every day except when you want the Synthetic Full to be updated.  Uncheck all of its retention policy settings so that executions of this job never trigger a retention policy.
- Duplicate that job (right-click it and select Duplicate), and for this new copy, set it to run once per week only on the day you want the Synthetic Full to be updated.  Set its retention policy to 1 Full and 1 Incremental, and enable Synthetic Fulls.  That way, whenever this job runs, all previous Incrementals will be rolled up into the Full, with the exception of the Incremental that was just created by this job, but there isn't a way to have the current backup immediately rolled up into a Synthetic Full.

Edited 29 January 2018 8:33 PM by jphughan
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)
Group: Forum Members
Posts: 2.4K, Visits: 16K
One additional note: If you're backing up Windows 10 machines, be aware that upgrading to a new Windows 10 releases (e.g. Win10 1703 to Win10 1709) occasionally changes your partition layout, and when that happens, Reflect will not be able to create a new backup as a Diff or Inc on an existing set.  Instead, it will create a new Full, even if the schedule specified an Inc, and the log will note this deviation and why it occurred.  The partition layout change will also mean that the pre-upgrade backups will not be considered "matching" relative to the post-upgrade backups, which means that the retention policy in its default configuration will never delete those pre-upgrade backups.  You could of course delete them manually after you feel you'll no longer need them, or if you want Reflect to keep applying its retention policy to those backups, you can change your retention policy matching to "All backups in the destination folder" -- but if you do THAT, make sure that every workstation backs up to its own folder.

Also note that if you change your retention policy matching to "All backups", you may also wish to change your retention policy to 2 Fulls rather than 1.  The reason is that if you only retain 1 Full, then the new post-upgrade Full backup would cause all previous backups to be deleted, since they would all have been built off of your previous Synthetic Full.  That could be especially problematic if you later find that you wanted those old backups specifically to roll back that Windows 10 upgrade, for example.  Allowing 2 Fulls to be retained would avoid that outcome, although it would also cause those pre-upgrade backups to be retained for longer than you needed -- you'd essentially end up always having 2 Fulls for every workstation that upgrades Windows 10 releases, again unless you manually deleted the old backups an appropriate time after such an upgrade.

If you decide that staying with 1 Full and potentially losing your backup history immediately after a Windows 10 release upgrade is acceptable, then I would at least recommend that you disable "Run purge before backup", because if you have that enabled and only retain 1 Full, the existing backups would be deleted before the new post-upgrade backup even completes successfully.

More info about the Windows 10 upgrade issue here: https://forum.macrium.com/Topic3262.aspx

Edited 29 January 2018 8:53 PM by jphughan
Obl918
Obl918
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 9, Visits: 13
jphughan - 29 January 2018 8:32 PM
- Duplicate that job (right-click it and select Duplicate), and for this new copy, set it to run once per week only on the day you want the Synthetic Full to be updated.  Set its retention policy to 1 Full and 1 Incremental, and enable Synthetic Fulls.  That way, whenever this job runs, all previous Incrementals will be rolled up into the Full, with the exception of the Incremental that was just created by this job, but there isn't a way to have the current backup immediately rolled up into a Synthetic Full.

Thanks; this is exactly what I was planning to do once it was confirmed that there isn't a more direct approach. It will work fine. As a minor feature request, a pseudo-trigger that ONLY enforces retention policy would be handy for me, but it's a minor thing. Thanks again.
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)
Group: Forum Members
Posts: 2.4K, Visits: 16K
You're welcome!  I'm not sure how having a way to execute the retention policy without running a job would help, though.  The only way a "retention policy only" execution would do more than had already been done when the retention policy was evaluated during the previous backup would be if you had reduced your retention policy settings since that last backup.  But that change would've been a manual process, and if you're going to manually change your retention policy, you could also manually address any backups you no longer wished to retain at that point.  Fulls and Diffs could of course be manually deleted outright, and Incrementals can be manually consolidated using Macrium's standalone utility here in case you weren't aware of that application.

Obl918
Obl918
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 9, Visits: 13
WRT the retention policy -- I am migrating away from Windows Server Essentials Client Backup which has a once-weekly task that is run to clean up the backup database based on the retention policy. It runs without any of the client PCs having to be turned on at a time when the server is unlikely to be called upon to do anything else.

Whether or not this will matter to me depends on how long it takes for Reflect to create the synthetic backups. I don't have experience with that yet. Regardless, I can adapt.
jphughan
jphughan
Most Valuable Professional
Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)Most Valuable Professional (3.3K reputation)
Group: Forum Members
Posts: 2.4K, Visits: 16K
Ah ok, that was a useful piece of information.  I have a client that uses Reflect for their server but WSE Client Backup for their workstations, so I'm familiar with both, and there are some significant differences here.  The reason the WSE solution needs to be cleaned up on a regular basis is because it stores backups in a completely different way.  WSE maintains a single database of data clusters from all PCs being backed up, plus ancillary files to describe how to recreate a given partition keep track of all of these clusters. That has its share of drawbacks, and WSE's solution in general has several limitations that Reflect doesn’t, but one of the HUGE advantages of WSE’s design is that it allows single-instance storage when backing up multiple PCs, i.e. the database only contains a single copy of any cluster that exists on any PC being backed up.  So for example let's say you have 10 PCs all created from the same original Windows image and they all have about 20GB worth of data in common.  With WSE, that 20GB of data common to all of your PCs would only be stored once on the server, and the individual PC backups would simply say, "The image of this PC includes that cluster.” In fact since clusters may even be duplicated on the same PC (such as photos with lots of data in common), deduplication can even be done within a single backup. In addition to reducing storage requirements, this also dramatically accelerates backup times, because after the first PC has uploaded its data, subsequent PCs don't need to upload it again.  Anyhow, eventually (e.g. after updates are installed), it's possible that none of your PCs will have backups that include certain clusters that currently exist in the database, in which case it no longer makes sense for the database to retain that data -- and that's where the cleanup operation comes in.  Reflect doesn’t do single-instancing, at least not across backups from multiple PCs (not sure about within a single backup), and implementing it would require support for storing backups from multiple PCs in a single repository, which I imagine would be a significant undertaking. That design also arguably increases the risk and certainly the impact of data corruption.  Also note that with WSE, the system performing the cleanup is not the same as the system(s) that generated the backup(s).  Reflect to my knowledge doesn't have a way to run just a retention policy against an arbitrary set of backups that might have come from anywhere -- although now that you've described your use case, I can see where that would be handy.

WSE's strategy also obviates the need for consolidation.  In terms of the time Reflect takes to do that, it's roughly the time it takes to generate a file of the size that's being consolidated, i.e. if you're merging a 10GB Incremental into a Full (or another Inc), it will take roughly the time it would take to generate a 10GB backup.  The reason is that consolidation essentially copies that 10GB backup file into another file before deleting the source file; there isn't a way to just "move" one file's contents into another file in order to avoid the copy operation.  So if your Incremental sizes tend to be pretty consistent, as a general rule of thumb, assume that consolidation will roughly double your backup time.  Incrementals Forever isn't really intended to significantly reduce backup time, in fact it may mean that your PCs spend more time overall running backups compared to running a weekly Full and daily Incs.  The main advantages to Incrementals Forever are that a) you avoid ever having to do Full backups, which might take significantly longer than Incs, and b) it's the most storage-efficient way to retain a given backup history since there's only ever one Full and no Diffs, and therefore no redundant of data in your backup set.  Well, I should say it's the most storage-efficient strategy for "true image" solutions, i.e. excluding storage efficiencies that can be realized by WSE when backing up multiple similar PCs.  You should expect your storage requirements to increase under Reflect if you implement a similar retention policy.

Edited 3 February 2018 4:45 PM by jphughan
Obl918
Obl918
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 9, Visits: 13
Thanks for the comments.  I've been using Windows Server client backup since it was on Windows Home Server. Back then the only reason I even installed it on my server metal was because I wanted the deduplicated block-level backup system it provided. Windows Home Server 2011 never gave me problems with the backups -- I had two years of backups from half a dozen machines with no trouble. I had to upgrade it to Windows Server 2012 R2 due to some unrelated issues, using Essentials Server Role to get the same features. For whatever reason -- and believe me, I have spent many many hours and purchased addition hard drives, etc, trying to figure it out to no avail -- I cannot keep backups running without database errors creeping in for more than three weeks. After years of constantly having to run the backup repair tool and occasionally losing backup sets in the process, after changing hardware, after doing all kinds of things, after re-starting the backup chains every month or two -- I finally gave up on it. Spending money on more storage is a better use of my time than trying to fight for a deduplicated system that just doesn't want to keep working. ANYWAY... :-)

I prefer the real image system in Reflect anyway. And it takes backups much, much faster. Thanks for the details regarding time for merging backups. I suspect I will have no trouble -- I just set it up so it only does retention policy one day a week and all the other days are just incremental backups. Pretty sure it will be what I want
Edited 30 January 2018 1:43 AM by Obl918
Alex
Alex
Macrium Representative
Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)Macrium Representative (31 reputation)
Group: Moderators
Posts: 29, Visits: 241
Hi,

Just wanted to add that jphughan is correct in his advice and it applies equally to the Site Manager - retention rules are attached to Schedules, so you can create weekly schedule including the 6 days with the retention rules set to retain all incrementals, use the copy option in the schedule view to copy it then edit the copy to run on the seventh day and set the retention rules to create the synthetic full and retain no incrementals. Then the same backup definition can be scheduled against both schedules in the repository you are using.

I'll pass your request on to development for having schedules to perform consolidation only - now that we have the ability to browse the repository and delete unwanted backup sets manually, we are looking into providing manual consolidation options through that interface in the future - scheduled consolidation would be a very logical additional feature.

Regards,
Alex Stevenson,
Macrium Software
Obl918
Obl918
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 9, Visits: 13
Alex - 1 February 2018 10:30 AM
I'll pass your request on to development for having schedules to perform consolidation only - now that we have the ability to browse the repository and delete unwanted backup sets manually, we are looking into providing manual consolidation options through that interface in the future - scheduled consolidation would be a very logical additional feature.

Thanks.  I have implemented the suggested scheduling without problems.

While on this topic, I would also like to see a way to schedule verification of all the repository content, independent of backup tasks. I would like to verify all of the images in the repository once a week, independent of backup PCs being online or running backups, and be emailed a report letting me know that all the data is good.
Edited 1 February 2018 1:45 PM by Obl918
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search