Advice on proposed backup strategy


Author
Message
nicopolous
nicopolous
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 9, Visits: 50
All,

I'm proposing to change my backup strategy and before I do I would like your expert advice on whether I'm taking too much risk or missing something obvious! I've had a read through a lot of the forum and it's been extremely helpful, so thanks for all your posts helping others.

My main requirement is to backup daily and retain 12 months of version history, so I'm proposing:

- Backup job A is daily incremental (with synthetic full applied), retain 365 backups
- Backup job B is monthly full, retain 3 backups

I would only ever want to restore from backup job A, but it is a very long chain. Backup job B therefore only exists to mitigate the risk job A poses (i.e. if the chain fails, I lose one month at most). These jobs go to separate destination folders.

I originally had a GFS scheme with retention for the fulls, diffs and incs to give me 12 months of versions (with a desired loss of granularity over time). HOWEVER, the problem was full backups were missed (this is not an always-on PC) and then when the retention rules were applied, I lost more version history than I wanted.

Am I missing a strategy that would give me what I want? Am I being completely reckless with the 365 incremental files I will eventually end up with?

Thanks very much all

Nick
Froggie
Froggie
Macrium Hero
Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 12K
nicopolous - 10 January 2021 1:29 PM
I originally had a GFS scheme with retention for the fulls, diffs and incs to give me 12 months of versions (with a desired loss of granularity over time). HOWEVER, the problem was full backups were missed (this is not an always-on PC) and then when the retention rules were applied, I lost more version history than I wanted.

@Nick, The problem you probably had with your GFS scheme was that you were probably using "# of" for your FULL retention rather than a time-based retention.  As such, MONTHLY was not available for use with FULL retention values.  This was a big miss on REFLECT's part.

So... using your general scheme, I went to a weekly/daily based based retention and set my FULL value to 8-weeks (not a month, but close) for an appx 2-mo retention for FULLs which were taken every 4-weeks (not monthly.  I then did weekly DIFFs and daily INCs.  The DIFFs had no retention value (they disappeared when my FULLs were culled), and I set my INC retention to 14-Days (not "# of").  This type of time-based setting always kept my timeline in tact and never culled older stuff when executing retention settings, either with INCs or FULLs.  Sure the cycle isn't monthly, but every 4-weeks is close and I never ran into the massive monthly scheduling probs that others have with daylight savings time issues, etc.

So for me, my settings for always having 2-FULLs, all DIFFs during the FULL retention period, and 14-days of INCs was...

Full: Keep 8 WEEKS
Diff:: <no retention>
Inc: Keep 14 DAYS

And this scheme used a scheduled FULL every 4-weeks, a scheduled DIFF weekly and a scheduled INC DAILY.  All the GFS schedules are set for the same day/time.  And with the INC retention set to DAILY, I have added both MANUAL INCs and privately scheduled (not through REFLECT) HOURLY INCs into the backup chain and all executes as expected, with INC culling occurring at the 14-day period.  This scheme has worked very well for me, and can easily be modified to obtain your extended needs... just not on a MONTHLY basis (using an every 4-week basis instead).


Edited 10 January 2021 2:47 PM by Froggie
nicopolous
nicopolous
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 9, Visits: 50
Hi Froggie,

Thanks for your reply, really appreciate it.

The problem I had wasn't really about # of backups vs. time-based retention.  It was that I often missed full backups (e.g. cancelled as was an inconvenient time and forgot to re-run).  Then I have a gap of, say, 3 months between full backups (instead of the desired one month gap).  When it came to deleting these fulls as part of retention, I lose 3 months of history instead of one month.  The strategy I outlined avoids this as my periodic full backups are independent of the daily incremental chain, so if I miss a full I don't lose any history. 

The other great benefit over a GFS scheme I forgot to mention is that I can make my full backups as often as I like and only need to keep the last few instead of keeping for as long as necessary to retain a year of history.  For example, I am planning to keep three fulls and backup monthly.  With a GFS scheme I would have to keep 12 of them to retain a year of history.  Weighing in at 1.5TB for each Full, I save a lot of space.

Hope that makes sense and if anyone reads this and thinks I'm missing an obvious way to achieve the same with a simpler or more elegant solution please let me know Smile


jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
UPDATE: Original post here written based on incorrect read of the proposed strategy. Revised reply below.
Edited 11 January 2021 8:25 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
Ok, my reply above was based on a misreading of the original post. Apologies for that. Now I see what’s being done. In that case I would still recommend revising Job A to include annual Fulls and monthly Diffs and then rely on Incremental Merge. That will cut down on the length of the Incremental chain significantly, likely without increasing required storage too much. A chain of 365 Incrementals seems a bit crazy to me. And I know from experience that the time required for consolidation into a Synthetic Full increases over time. In my own case I have a client with a disk rotation where no single disk is used more than once per week. Each Incremental is about 150 GB. Early on, an Incremental that takes an hour to create would take about an hour to be consolidated later. After several dozen though, that Incremental that took an hour to create can take 10 hours to consolidate. Macrium has traced this to increasing fragmentation within the Synthetic Full — which is NOT resolved by defragmenting the volume itself — and at the moment there isn’t a fix, though they’ve said they’re looking at options. So for now I just manually create a new Full to start over with a fresh, no-fragmentation file. Of course that isn’t technically “Incrementals Forever”.

But this idea would avoid Synthetic Fulls and therefore their fragmentation, while avoiding a situation where the Full NEVER gets updated.
Edited 11 January 2021 8:29 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
Froggie, I suspect the reason Reflect omitted months from the time-based options was because not all months have the same number of days and February doesn’t even have the same number of days each year. Days or weeks by comparison are consistent figures.
Froggie
Froggie
Macrium Hero
Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 12K
jphughan - 11 January 2021 8:25 PM
Froggie, I suspect the reason Reflect omitted months from the time-based options was because not all months have the same number of days and February doesn’t even have the same number of days each year. Days or weeks by comparison are consistent figures.

I believe you are correct, sir!
nicopolous
nicopolous
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 9, Visits: 50
jphughan - 11 January 2021 8:14 PM
UPDATE: Original post here written based on incorrect read of the proposed strategy. Revised reply below.

Thanks for the useful and insightful info!  Very good to know (although disappointing) to hear that there are issues with the synthetic full.  Given it's supposed to be "forever" incrementals, it's not really fit for purpose if the fragmentation issue eventually causes problems.  

Re. the long incremental backup chain, I completely understand the reasons it's advised against but I struggle to understand why.  Corruption from bad sectors (for example) isn't more likely in a full+incremental backup just because data is spread over multiple files vs. all in one large (full) archive.  I realise I'm probably oversimplifying!

Thanks again for your help, much appreciated.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
nicopolous - 12 January 2021 5:58 PM
Thanks for the useful and insightful info!  Very good to know (although disappointing) to hear that there are issues with the synthetic full.  Given it's supposed to be "forever" incrementals, it's not really fit for purpose if the fragmentation issue eventually causes problems.  

Re. the long incremental backup chain, I completely understand the reasons it's advised against but I struggle to understand why.  Corruption from bad sectors (for example) isn't more likely in a full+incremental backup just because data is spread over multiple files vs. all in one large (full) archive.  I realise I'm probably oversimplifying!

Thanks again for your help, much appreciated.

Happy to share a few thoughts.  I agree that it would be nice if Macrium was able to find a solution to the fragmentation issue.  They have talked about providing a defrag utility, but apparently it would take quite a while to run, and to my knowledge they haven't offered it.  I'm hopeful that will change so that it can truly be Incrementals Forever, but for now I'm working around the issue.

In terms of the chain, I concede that my viewpoint on it isn't entirely rational.  However, I will add a few points to consider.  First, the likelihood of data corruption being a problem generally DOES increase with a long chain compared to a short chain.  Let's say you have a Full backup from January 1 and daily Incrementals for each day thereafter.  In order to restore a backup created on December 31, you need your Full and a year's worth of Incrementals to be intact.  The total data footprint of those backups will be some amount.  Now consider a scenario where you have a Full from January 1, and thereafter you have monthly Diffs and daily Incs.  Now in order to restore a December 31 backup, you need that Full, the December 1 Diff, and a month's worth of Incrementals.  That will likely be less total data because that December Diff won't have to include data that might have been added and then deleted from January through November as the year-long Incremental chain would.  The difference in the total size of those groups of backups will of course vary based on each situation, but a smaller data set means fewer sectors that could potentially go corrupt.

But for a complete picture of risk you can't just look at the likelihood of a problem. You also have to look at the IMPACT of the problem, in this case a bad sector, if it actually did occur.  In your current plan, losing Incremental #1 would knock out up to a year's worth of subsequent backups.  With monthly Diffs and daily Incs, the impact of a problem with any particular backup would only be up to a month -- except of course for the Full, where in both proposed strategies a problem would knock out all backups of the set, i.e. up to a year's worth.  In your case, you're mitigating that with your secondary backup scheme, but that will only go back 3 months.

Edited 12 January 2021 6:16 PM by jphughan
Beardy
Beardy
Proficient Member
Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)
Group: Forum Members
Posts: 313, Visits: 1.4K
Seeing your desired outcome in terms of data state retention, I think I'd be wanting to shorten the chains for the reasons @jphughan mentioned, I'd probably assess the impact on storage of doing my fulls monthly, that way the worst a corrupt backup in a chain from say an unrecoverable read error would be a gap in that  chain of up to one month, less if such happened partway through a chain.  If that was going to use too much storage, then possibly full every 3 months, though I'd start being uncomfortable doing that, instinct tells me it's a long time with no full.

I see you already have a strategy to mitigate against failure of your main backup storage drive.

As an aside UREs do occur, & they're the reason industry has pretty much abandoned RAID 5 since a single one will fail a rebuild & render the entire array unrecoverable. For a single drive they only (usually) render a single file unrecoverable / corrupt unless they strike somewhere critical like the MFT.  For NAS & Enterprise class drives they even publish a claimed URE rate in the documentation, you can do sums to figure out your chances if you dig the full details up, & if you believe the rather optimistic looking numbers they publish.
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search