"Easy mode" backups ala Windows File History

Macrium Evangelist
Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)Macrium Evangelist (10K reputation)
Group: Forum Members
Posts: 7K, Visits: 51K
Since Macrium has confirmed elsewhere that they are working on building their own scheduling engine rather than continuing to rely on Windows Task Scheduler, I thought this might be a good time to make a feature suggestion.  Incidentally, another suggestion that might become possible to implement with an internal scheduler is "opportunistic backups", which I envisioned here, but that was primarily to address cases of backing up business workstations that aren't always on the network at a predictable time.  What I'm about to describe below is intended to address cases where users do not want to learn about various backup types, retention policy "theory", and general best practices in order to configure their Reflect backups.  They just want to know they're getting backups.  I realize that Reflect offers templates today, but that merely populates values, so the user is still left wondering what it all even means, before even asking the question of whether those defaults would be appropriate for them.  I believe Macrium can offer a way for users to "just have backups", without taking anything away from users who want the level of control that Reflect offers today.

For this "easy mode", I'm thinking the yardstick should be Windows File History or macOS Time Machine.  Both of those options only require you to specify the destination drive, and you've now configured your backups.  There are other options that can be configured, but in typical use cases, the defaults are appropriate.  Even if Reflect can't QUITE get to that level of simplicity, I would envision "easy mode" working thusly:
  • Reflect would ask the user to specify a destination folder (not just a drive)
  • The default behavior would be to capture an image backup of the entire disk that contains the user's Windows partition.
  • The schedule would be backups every hour, and the retention policy would be something along the lines of, "Retain hourly backups for 8 hours, daily backups for a week, weekly backups for a month, and monthly backups going back as far as capacity allows, or up to a time or capacity limit that the user can optionally specify."  I believe both File History and Time Machine work this way.
  • Under the hood, the above would have to involve either a GFS strategy with Incremental Merge, or Incrementals Forever with Synthetic Fulls, but either way, given that there are 4 "tiers" of backups proposed here (hourly, daily, weekly, monthly), some creativity in Incremental consolidation might be required, i.e. rather than only ever consolidating the oldest Incrementals as occurs today, this implementation would require consolidating Incrementals in the middle of the chain.  For example, when a given hourly Incremental became more than 8 hours old, Reflect would consolidate it into the daily Incremental it would be keeping longer-term, but Reflect would ALSO have to keep even OLDER Incrementals in the chain that would be daily Incrementals of previous days.
  • Between GFS vs. Synthetic Fulls, GFS would have the advantage of giving the user multiple sets, which would be nice for resilience.  The downside is that the destination capacity requirements increase, potentially substantially with some people's systems.  And I would argue that with an "easy mode" solution, multiple sets might not be the top priority. Neither File History nor Time Machine does anything to maintain multiple sets to mitigate corruption issues, to my knowledge.  Synthetic Fulls is the most storage-efficient way to store a given set of data, and only having to capture and manage retention of Incrementals would likely simplify implementation of this solution -- but to make that feasible, Macrium would have to solve the Synthetic Full performance degradation problem that occurs today as the Full becomes increasingly fragmented from successive consolidations.  Otherwise, people who just wanted an easy backup solution would end up with a solution that gets significantly slower over time and wouldn't know to periodically create a new Full manually in order to work around that.
I realize that this would be a non-trivial addition to Reflect -- much like the opportunistic backups idea I wrote up earlier and mentioned at the beginning of this thread -- but experience on these forums has shown that many users do not grasp retention policy concepts well enough to correctly predict the resulting behavior of a given policy or to determine a strategy that would be best for them and align with best practices.  And that's just the people on this forum that are speaking up because they know enough to realize that they don't know -- so there will undoubtedly be more who don't even realize that what they're currently doing might not be ideal.  And then there are the people who might understand this all well enough, at least after some forum discussions, but are still left wondering why setting up a good backup strategy for them had to involve so much learning and thinking.

Edited 28 June 2020 10:58 PM by jphughan

Merge Selected

Merge into selected topic...

Merge into merge target...

Merge into a specific topic ID...

Reading This Topic