Opportunistic backups (incl. resuming interrupted backups)

Macrium Evangelist
Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)Macrium Evangelist (16K reputation)
Group: Forum Members
Posts: 10K, Visits: 66K
This idea comes from my experience with the Client Computer Backup function included with Windows Server Essentials, and I think having it available in Reflect would be especially helpful to customers using Site Manager to back up laptops to network locations.  Using laptops can mean that connectivity to backup destinations may be unpredictable -- employees may not be in the office or on VPN on a predictable schedule, or they may simply lose network connectivity while taking their laptop from their desk to a conference room.  Currently, these issues would cause scheduled Reflect jobs to fail.  I therefore would recommend that Macrium consider including an option to perform backups "opportunistically", including being able to resume interrupted backups.  I'm envisioning something like this:

- The same scheduling options that exist today would continue, with an additional "Enable opportunistic mode" checkbox for the definition file, rather like the Synthetic Fulls checkbox.  If this is enabled, Reflect would create a permanent rather than temporary VSS snapshot, thereby allowing backups of a given data state to be resumed after interruptions, including reboots.  Reflect would also NOT immediately fail the job if the destination wasn't available when the backup began.  The job would instead get queued and complete as soon as possible, and additional scheduled occurrences of that same definition file would NOT be allowed to even queue until the existing job had completed.  So for example if the user schedule specified weekly Fulls and daily Incs with opportunistic mode enabled, then if the weekly Full did not complete until two days later due to interruptions, Reflect would not immediately start working on "backlogged" Incs; it would instead just begin a new Inc attempt at the next scheduled occurrence.

- A "partial backup maximum age" would be specified, such that if a backup that was begun on Sunday (for example) had still not completed by Wednesday due to interruptions or lack of connectivity to the destination, then Reflect would abandon that backup attempt, create a new VSS snapshot, and try again with the new snapshot to capture more current data. If backups still couldn't be completed, eventually Reflect would warn that you haven't had a backup in X days/weeks.  This "no complete backup warning" threshold would be user-configurable, so you could say, "Capture a weekly Full and a daily Inc opportunistically, but if a given backup hasn't completed after 2 days, restart it with a new snapshot, and warn me if I go 3 days without a completed backup."  In this scenario, if the first attempt of the weekly Full "expires", then Reflect would restart that Full attempt with a new snapshot until it completed one, without attempting any of the daily Incs in the interim.  My reasoning here is that if something is preventing Reflect from completing a Full even when it's resuming across interruptions, then there's little reason to believe an Inc would fare much better, and since the user's schedule clearly indicates a desire for regular Fulls, if a Full attempt "expired", it would not be acceptable for Reflect to just stop trying to create ANY Full and just carry on with an Inc instead.

- It would be possible to have the "partial backup maximum age" be less or greater than the "no complete backup warning" threshold.  For example, some users might be ok with Reflect going a few days without backups before throwing a warning but will always want Reflect trying to capture a fairly recent backup -- they would want a longer warning interval and shorter maximum age.  But other users might want to make sure they get a backup specifically of their Sunday night state, and therefore they would want a warning well before that backup "expires" so they can make sure that specific attempt completes -- they would want a shorter warning interval and a longer maximum age.

- Whenever a backup DOES complete, the Backup Date would be recorded as the time the successful attempt began, since that would match the time of the VSS snapshot used for the successful backup and therefore the state of the data contained in the backup.  For example: The first backup attempt begins Sunday morning but expires on Wednesday morning, not having been completed.  A new snapshot is automatically captured on Wednesday morning for another attempt, and that second attempt completes Thursday night.  The Backup Date here would be Wednesday morning, since that backup would contain your data as it was on Wednesday morning, not on Sunday morning when the first attempt began or Thursday night when the job finished.

- The Reflect and Site Manager UIs would be expanded to show any partial backups, including when they were begun and how far along they are.

One current limitation that would probably have to be removed for opportunistic mode to work is the inability to have jobs from different definition files running simultaneously. If I had an image job that backs up to a NAS and an F&F job that backs up to a local disk, I wouldn't want my opportunistic image job to have to complete, potentially over several days depending on NAS availability, before any of my F&F jobs could even begin.

I'd be curious to hear thoughts from other Reflect users about this idea.

Edited 12 March 2018 4:36 AM by jphughan

Merge Selected

Merge into selected topic...

Merge into merge target...

Merge into a specific topic ID...

Reading This Topic