Delayed / scheduled consolidation?


Author
Message
NC-Tech
NC-Tech
New Member
New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)
Group: Forum Members
Posts: 10, Visits: 21
We are still using Macrium v6, including on our file server--a total of several TBs of information there...
My boss would like to start doing hourly Incrementals (x2 - to 2 backup servers) instead of just daily, but wait to do consolidation only overnight (every night).

Say I allowed 8 Incrementals before consolidation. I'd have one day of no work-hours consolidation, but would be sunk starting on the second day. (Hourly Consolidation would be time- and resource-intensive, and likely prevent the Incrementals from actually occurring hourly.)

One option that I thought of could be a nightly Differential, in theory, to break up the Incrementals--but our multi-TB Full backups are pretty rare, so the Differentials would continue to get bigger and take more time, and I don't know how long this option would be viable.

I found the "standalone backup set consolidation" utility at https://knowledgebase.macrium.com/display/KNOW/Standalone+backup+set+consolidation but am not seeing a way to script or automate that.

Not sure if the VBScript, PowerShell, or MS-DOS Batch tabs within Macrium would be able to somehow help here either (even supposing I did know how to write those!)

Is there a way to do this in v6? If not, is it somehow possible in v7?

Thank you for any ideas!
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)
Group: Forum Members
Posts: 4.5K, Visits: 33K
There's no way to have Reflect itself perform a "consolidation-only" job.  Using a script in conjunction with the standalone consolidation utility might be possible, but first the standalone consolidation utility would have to support being used via command-line rather than an interactive graphical interface, and it doesn't look like it currently does -- or at least the documentation doesn't mention it.  But supposing Macrium were to fix that, the next challenge would be writing a script that could properly identify the files you wanted to consolidate in order to feed to that utility.  Depending on your backup strategy, reliably identifying the desired backup set and avoiding "over-consolidating" in cases where for example the backup(s) may not have run as expected beforehand might be tricky.  That said, if Macrium adds command-line support to Consolidate.exe and you'd be willing to share some details about your backup strategy, I know some PowerShell, so I'd be willing to take a crack at helping you do this.  In any case, using Reflect's built-in support for creating scripts won't really help you here, because the assumption with those scripts is that you'll actually be running a backup amidst whatever else you decide to add before/after it, and it sounds like that wouldn't be true for your use case.  That means you'd be writing a script from scratch and scheduling it completely separately from whatever you do in Reflect.

All that said, "off-hours consolidation" is definitely an intriguing idea, so it might be nice if this could be more formally supported in Reflect itself in the future.  I can definitely relate, because I have a client where where the actual backup takes about 3 hours and then the consolidation takes 6-7 hours.  In their case though, they use a daily disk rotation, so the backups aren't available for later off-hours consolidation, and currently the whole backup+consolidation job can still complete overnight.

Edited 20 September 2018 7:31 PM by jphughan
NC-Tech
NC-Tech
New Member
New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)
Group: Forum Members
Posts: 10, Visits: 21
Thank you for your offer of help with possibly PowerShell'ing this, if Macrium supports it.

I've tried running commands like "consolidate.exe /?" (and -h, --help, and all combinations of the symbol and suffix).
As of Consolidate ver. 7.1.2917, if there is some sort of command-line support, it doesn't seem to be documented online or in the executable either.

I guess I'll either have to wait until someone from Macrium chimes in to illuminate us about undocumented options, or until they actually do add this feature in the first place!

Froggie
Froggie
Master
Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)
Group: Forum Members
Posts: 957, Visits: 8.3K
@NC-Tech, it's not very clear what kind and how much CONSOLIDATION you want to do and when, but if you're looking into running hourly Incrementals (say 8am - 10p) and you want to do a daily consolidation (into 1 or more Incrementals at the end of the day), this is actually pretty easy to do.  It cannot be done using a "Forever Incremental" type of scheme but is easy to do using a GFS type of backup scheme.

If this is where you're headed, explain your actual need ( and timing) and I'll be happy to spell out a scheme for you).

Edited 21 September 2018 12:00 AM by Froggie
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)
Group: Forum Members
Posts: 4.5K, Visits: 33K
The OP stated that Differentials were undesirable because they would continue to get larger over time and avoiding that by capturing a Full even periodically is apparently undesirable because it would be a multi-TB affair.

However, @NC-Tech I thought of a way to achieve what you want with much less effort and without having to wait for anything.  Here goes:
- Modify your current backup definition file to completely disable the Incremental retention policy.  This will become your "non-consolidating backup" definition file and the one that you will run for most of your backup needs, including these hourly backups.
- Right-click your modified definition file and click "Duplicate".  This new definition file will become your "consolidating backup" definition file.  For this one, set the Incremental retention policy to whatever you want to be left with after a consolidation occurs, but set this definition file's schedule to run only during the times when actually want to perform consolidations.  Or of course you could have no schedule at all for this one and run it only manually when you wanted to perform a backup that would perform consolidation.

The result will be that since both definition files are going to the same destination, they will both be appending backups to the same set; you will NOT end up with separate backup sets just because they're coming from separate definition files.  However, consolidation will never occur when you run jobs under the first definition file that has its Incremental retention policy disabled; Incremental backups will be allowed to accumulate without limit.  But when you run that second definition file, consolidation WILL occur, and THAT job will consolidate however many backups are necessary to comply with your settings.

Edited 21 September 2018 12:18 AM by jphughan
NC-Tech
NC-Tech
New Member
New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)
Group: Forum Members
Posts: 10, Visits: 21
Thank you both for your input and assistance!

@Froggie, for our smaller servers we currently do monthly Full backups and Incrementals every weekday (consolidating Incs. after 10 of them). But for our file server, a Full only happens 1-2 times per year, with a monthly Diff. and daily Incrementals, also consolidated after 10. The new piece of the puzzle is trying to move from daily Incs. to hourly for several servers.

@both So we do currently use Differentials some, but my particular concern about them was regarding trying to migrate to a nightly Differential as a strategy to deal with the consolidation issue, and how that frequency and timing of Diffs. would eventually become impractical for at least our biggest backup.

@jphughan, that approach had occurred to me, but I was under the mistaken impression that each definition file would result in a different backup set. Thanks for setting me straight. I'll make sure I've got it working alright on one of our smaller servers, then try to implement it on our biggie.


NC-Tech
NC-Tech
New Member
New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)New Member (40 reputation)
Group: Forum Members
Posts: 10, Visits: 21
I was in the process of implementing @jphughan's idea when I may have thought of a problem: whether it's feasible to consolidate the Hourlies while still being able to keep a multi-day history of Incrementals like we are currently doing.

By my understanding, which may be wrong somewhere again, the "consolidating" definition will see all Incs. as being equal, even those created by the non-consolidating definition. There's no way to automatically consolidate more recent Incrementals (today's Hourlies) while keeping 10 days' history intact; we can only consolidate the oldest Incs. of the set. So to have Hourly backups while still keeping 10 days' worth of Incs., we'd actually need to have the "consolidating" definition retain (9 x (# of Hourlies per day) + (1 consolidated oldest day)) Incrementals.

A backup every hour on the hour from 9AM-6PM would be 10 per day. We'd have to retain 91 Incs. to keep a 10-day history, and it would get up to ~101 Incs. each weeknight before the consolidation.

Yikes! Am I thinking this through correctly?
Thanks!

(Am realizing I don't know how to really do forum members' names in a post...)

Edited 21 September 2018 11:19 PM by NC-Tech
Froggie
Froggie
Master
Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)Master (1.7K reputation)
Group: Forum Members
Posts: 957, Visits: 8.3K
NC-Tech - 21 September 2018 11:17 PM
I was in the process of implementing @jphughan's idea when I may have thought of a problem: whether it's feasible to consolidate the Hourlies while still being able to keep a multi-day history of Incrementals like we are currently doing.

This is not feasible with Reflect... all INCs are treated equally when consolidation is performed.  This is exactly what <competitor A> does with their imaging package.  This is what I was hoping Macrium's "Continuous Incrementals" option, offered "soon after Feb. 2017," would do for me... but the option never showed up after all.
Edited 22 September 2018 12:04 AM by Froggie
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)Macrium Evangelist (6.6K reputation)
Group: Forum Members
Posts: 4.5K, Visits: 33K
@NC-Tech, I figured your consolidation interval would be at least as frequent as your desired "long-term retention" interval, i.e. if you wanted to keep one Incremental per day, you would consolidate your hourly Incrementals at least once per day.  If you instead want to perform hourly Incrementals, keep one Incremental per day on a longer-term basis, but only actually perform consolidations once per week, for example, then you'll probably need to consolidate manually.  For a second I was thinking that it might be possible to achieve what you want it in an automated fashion by taking advantage of another property of Reflect backups, namely that it is possible to create Incrementals in a separate location from all other parent backups as long as you have at least one Incremental already at the destination before creating more.  This is intended to enable backup strategies that involve, for example, creating a Full and a few Incrementals at Destination A and then archiving the Full and perhaps all but the newest Incremental off to Destination B while continuing to be able to create new Incrementals at Destination A.  However, as I thought about it, I couldn't think of a way to leverage this capability for your purposes in a way that could be guaranteed not to break a dependency chain and would be easy to sort out if you ever needed to restore something.  I'll keep mulling it over, but I suspect Froggie is right above. Sorry!

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search