Best backup strategy with limited disc space?


Author
Message
Anthony
Anthony
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 10
I have a 3TB backup drive. My C: drive backup uses about 300GB of that.  My D: drive backup uses 2TB. That only leaves about 700GB of free space to work with on the backup drive.

That's plenty to backup the C drive, then purge the previous backup. However, I can't backup my D drive unless I purge the previous backup before making a new backup. That basically leaves me without a backup during the backup process (I do have older backups on another drive stored offsite).

I'm currently using
@MaxSyncUp to sync my D: drive to my backup drive. This works fine given the limited storage available, but of course it does not have any ransomware protection like the Macrium Image Guardian.


I'm curious if there is a backup strategy I can use with Macrium Reflect (file or image backups) that would allow me to backup my D: drive nightly without purging the old data file when a full backup is needed (I usually do a full backup on Sundays and incremental's the rest of the week).

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
The backup method that's most likely to work here also happens be the most storage-efficient, and it's called "Incrementals Forever with Synthetic Fulls".  You only make Incrementals at every backup (hopefully you have enough capacity for at least an Incremental), and in theory you never have to capture a new Full because the Full is continuously updated.  Macrium has a nice animation of this concept here.

HOWEVER, the risk with this strategy is that if you only have a single backup destination, this strategy means you'll only ever have one backup set.  That's a fairly high-risk proposition because it means that if your Full ever becomes even partially corrupted or unreadable due to a file system or hardware issue on your backup drive, then all of your backups become useless.  That danger is magnified by the fact that you likely won't even know you have corruption or readability problems in your backups until you try to restore something, and if you're trying to perform a restore because your source disk already died, then you have no usable backups and no opportunity to create new usable backups.  This is why I typically only recommend using Incrementals Forever with Synthetic Fulls if you'll be using a disk rotation (so that you have multiple devices each with their own single backup set) or will at least be replicating your backup files to some other location.  In your case, since you currently never have more than one backup anyway and sometimes don't have any while a new backup is being created, I guess this would still be an improvement, but only having a single backup set is still dangerous.  Another unrelated issue is that the Synthetic Full can become fragmented over time, which will cause the "consolidation" operation to take longer.  In my case I have Incrementals that are each roughly 100 GB and after several dozen consolidations, a consolidation job that used to take about 2 hours might take 9 hours due to the fragmentation within the Full (this isn't something that gets solved by defragmenting the drive itself, fyi.)  I reported this to Macrium and they're looking at ways to mitigate it, but at the moment my workaround is to simply create a new Full occasionally so that I once again start with a new, fragmentation-free Full.  Since I use a disk rotation, I can "afford" to just delete all existing backups on each disk as they're reconnected in order to "reset" the set since I've got other backups elsewhere.  But if you've only got one destination, you'd want to make sure you have enough capacity to create a second Full without deleting your existing backup set if you ever needed to do this.

All that said, given how inexpensive storage is these days, my strong recommendation to you would be to purchase a larger hard drive that will allow you to always have at least 2 backup sets at any given time, so that if one set is ever corrupt you'll still backups in another set.  Even better would be to have two external hard drives, since of course if your backup drive fails then all of your backups are still gone regardless of how many sets you have.  Note that fulfilling that "2 sets at all times" requirement would mean you'll need enough storage to temporarily store at least 3 Full backups, since you don't want to delete the oldest Full (and its child backups) until after the new Full has been created.

Edited 24 October 2019 2:52 PM by jphughan
Anthony
Anthony
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 10
[quote]
jphughan - 24 October 2019 2:50 PM
The backup method that's most likely to work here also happens be the most storage-efficient, and it's called "Incrementals Forever with Synthetic Fulls".

my strong recommendation to you would be to purchase a larger hard drive that will allow you to always have at least 2 backup sets at any given time, so that if one set is ever corrupt you'll still backups in another set.  Even better would be to have two external hard drives

I saw the synthetic full option but was worried about corruption over time.

I do have two external hard drives (one local, one stored offsite). Until recently, I've had enough space to keep two backup sets on each backup drive. Unfortunately, I just exceeded that space last week.

Until I can buy a pair of larger drives, I've switched to syncing my largest drive. It works fine, and still leaves me room to grow on the drive, there's just no ransomware protection.

I'm thinking of buying a third external hard drive. That would let me swap two drives locally more often (maybe once a week or less), then swap out the third drive offsite once a month as usual (when I can remember to do it). That's probably better for ransomware protection anyway.

The biggest hassle with swapping drives is having to reset the drive letter each time I plug in a new drive. Windows 7 always assigns the drive to whatever first letter is available (not always the same), but I always assign my backup drives as drive X:

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
I've implemented Synthetic Fulls at multiple clients and haven't had any issues, although I did exchange some emails and PMs with Macrium developers asking all kinds of questions beforehand since I too was concerned about corruption.  My takeaway is that they implemented a clever design that mitigates the corruption risk and they tested failure scenarios at all possible steps of the process.  I for example have had multiple cases where someone accidentally disconnected the backup drive while the consolidation was still running.  Reflect just figured it out the next time it saw that disk, no problem.  The way the consolidation operation works is that none of the data in the Full that's part of the current state of that backup is directly overwritten by the new data being consolidated.  Instead, the new data from the consolidation gets APPENDED to the existing file, and then only AFTER that operation completes successfully is that new data considered "usable" and part of the new state of that backup.  At that point, the "stale" data from the now-previous state that's no longer needed is marked as scratch space, and those data blocks become eligible to be overwritten during subsequent consolidations.  The result is that if the consolidation fails partway through, perhaps due to someone disconnecting the disk when they shouldn't, all of the original state data for that Full is still in that file, so any data from the partial consolidation is ignored and the Full is still intact -- which means Reflect can reattempt the consolidation operation again later.

As to your drive letter issue, I use a rotation of 9 disks and don't have that issue because there are two options to solve it.  The first is to switch Reflect to use Volume ID-based targeting rather than drive letter targeting.  That way any destination disk is found by its unique volume identifier, which means Reflect will always find the right disk even if its drive letter changes.  The other is to use some external means to guarantee consistent drive letter assignment of any disk in your rotation.  For various reasons I went the latter route using a tool called USB Drive Letter Manager.  Despite the name, it works on non-USB disks and can do a lot more than manage drive letters, but I'm just using it for its "title role" purpose.  The way I implemented my setup is that the config file for USB Drive Letter Manager tells the application to check any attached USB disk for a INI file of a specific name at a specific path, and if it finds such a file on that disk, assign it the letter specified in that file.  I have a copy of that same INI file on all disks involved in my backup rotation, in all cases specifying drive letter Z.  So whenever one of those disks is connected, that application makes sure it gets assigned drive letter Z based on that INI file.  That application is free for personal use and very inexpensive for commercial use, and I'd be happy to share my application config file that achieves this if you want to use it as a reference for setting up your own rotation.  I even used this application to help someone who wanted a setup where Reflect would immediately start a backup job to a destination disk as soon as said destination disk was connected because he was supporting a user who would sometimes forget even to double-click a shortcut file on their desktop to run the job.

(If you didn't know, the limitation with manually reassigning a drive letter to something like X is that Windows only remembers that custom assignment if that device was the last device to use that drive letter.  So in a rotation scenario if you connect Disk 1 and assign it X, then disconnect it and connect Disk 2 instead, and assign that X as well, Windows won't assign X to Disk 1 the next time it sees it.)

Edited 24 October 2019 4:21 PM by jphughan
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
One more thought in addition to the reply above: If you're going to consider a disk rotation strategy rather than post-backup replication, Incrementals Forever with Synthetic Fulls is by far the easiest strategy to use, except I guess for 100% Full backups, but that's not always practical.  The reason is that with Incrementals Forever, you're only ever creating Incremental backups.  With other strategies like GFS, you have different types of backups occurring at different times, which means that if you add a disk rotation into the mix,  you need to make sure that the "right" disk is connected on a "Full day" or "Differential day" or whatever in order to make sure that each disk has the correct number of Fulls, Diffs, Incs, etc.  If you accidentally have the wrong disk connected on a "Full day", for example, you'd end up purging an older backup set on that disk sooner than that should have occurred, and when you next connect the "right" disk, if you don't manually perform a Full backup to compensate for the one that you missed, you'll end up with a longer Incremental chain built off the previous Full since that disk didn't get the newest Full that it should've.  That's not as much of a concern with Incrementals Forever.  I manage that entire rotation of 9 disks with a single Reflect definition file that has a single schedule entry: Incremental backup every weeknight.

Edited 24 October 2019 4:26 PM by jphughan
Anthony
Anthony
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 10
[quote]
jphughan - 24 October 2019 4:12 PM
I went the latter route using a tool called USB Drive Letter Manager.  Despite the name, it works on non-USB disks and can do a lot more than manage drive letters, but I'm just using it for its "title role" purpose.  The way I implemented my setup is that the config file for USB Drive Letter Manager tells the application to check any attached USB disk for a INI file of a specific name at a specific path, and if it finds such a file on that disk, assign it the letter specified in that file.  I have a copy of that same INI file on all disks involved in my backup rotation, in all cases specifying drive letter Z.  So whenever one of those disks is connected, that application makes sure it gets assigned drive letter Z based on that INI file.  That application is free for personal use and very inexpensive for commercial use, and I'd be happy to share my application config file that achieves this if you want to use it as a reference for setting up your own rotation.
Could you share your config file for USB Drive Letter Manager and how you use it?  Do you have a config file in the USBDLM folder, then another on each drive that will be plugged in?

I would love to be able to plug in any of my backup drives and have themt automatically assigned as the X: drive.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
Happy to!  My USBDLM.ini file stored in the USBDLM application folder is as follows:

; This configuration file contains only the settings (and their descriptions) needed
; for this specific implementation to function as needed. USB Drive Letter Manager
; (USBDLM) has several more settings that can be configured if desired. A full sample
; configuration file is included with the ZIP format download of the application.
; Go to http://www.uwe-sieber.de/usbdlm_e.html for the latest version and documentation.

[DriveLetters]
; Only act on USB drives classified as removable (flash drives) or fixed (spinning hard drives)
BusType=USB
DriveType=REMOVABLE, FIXED

; Only act on drives that have a file called DriveLetter.ini stored at the root of the volume
FileExists=%drive%\DriveLetter.ini

; The drive letter to be assigned is determined by the contents of the DriveLetter.ini file
; at the root of the volume.
Letters=%drive%\DriveLetter.ini

;Wait 5000 milliseconds for hard drive to mount in order to check for the INI file
FileTimeout=5000

[NetworkLetters]
; NetworkLetters are not allowed to be assigned to local drives.
; They overrule letters configured in external INI files.
; The letters below are used for mapped network drives at this organization.
Letters=F,G,H,J,K,P,V

[BalloonTips]
; For new drives, Windows shows rather uninformative balloon tips, like 'Ready to use', but
; without specifying which letter. USBDLM can remove the Windows balloon tip to show its own
; 0 - off, 1 - on, 2 - deactivate Windows balloons temporarily
SuppressWindowsBalloons=0


You can customize the path to the INI file if you want to place it somewhere else on your destination drives by changing the FileExists parameter (where to look for it) and the Letters parameter (which specifies to assign the letter based on the contents of that file).  You should also adjust or remove the Letters parameter in the NetworkLetters section if you don't have any mapped drive letters you want to prevent USBDLM from using.  As for the INI file located on the destination drives themselves, that's just this:

; This file is used by USB Drive Letter Manager which is configured to check the root of
; newly connected USB drives for a file named DriveLetter.ini (this file), and if present,
; to assign the newly connected volume the drive letter specified in the DriveLetter.ini file,
; configured below.
[DriveLetters]
Letter=Z


Anthony
Anthony
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 10
Awesome, thanks. I'll give it a go and see how it works out.

Anthony
Anthony
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 10
When using the synthetic full option, is there a minimum number of incremental files I should use for the retention policy?  If incrementals get merged with the full backup, why would I want more than one or two incremental files?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)Macrium Evangelist (22K reputation)
Group: Forum Members
Posts: 14K, Visits: 84K
You'll always need at least 1 Incremental because there's no way to tell Reflect to create an Incremental and then immediately merge it into the Synthetic Full.  You can do that yourself manually if you want to use the Consolidate.exe standalone application, but that application doesn't currently support command-line execution, so even scripting that wouldn't be an option for automation.  As to how many Incrementals to retain, the main factors that contribute to that decision are how far back in time you'll want to be able to go with your backups and how much storage you have available for storing said backups.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search