My system is very simple. I do one full backup of all of my hard drives, and then nothing but incrementals after that. My definition file is set to keep 50 backups, and to purge old backups when free space is below 50 GB.

This is perfect for my needs, except for one problem. If the external HDD runs out of space in the middle of the backup, then it will completely stop and require input from the user (me). For example, if the external HDD has 60 GB free space, and the next incremental requires 70 GB, then the whole task will stop at the 60 GB mark and ask for another place to store the remaining 10 GB. It will not continue until I do something.

The problem with this is that I can no longer leave town and trust that the program will finish the backup. It will hang until I return, possibly days later.

Because of this, I ask that an option be added to automatically merge incrementals to the full synthetic backup if the task runs out of space in the middle of the backup. This would solve my problem 99% of the time, because normally when I add a large amount of data (e.g., 70 GB) it means that I had to delete the same amount of data to make room for it.

In the event that there is still not enough space, even after the incrementals are merged, then there should also be options for how to proceed after that.

I am aware that, in the above example, I could simply change the free space threshold to say 75 GB and that would also solve the problem. However, it is impossible for me to know exactly how much space will be required for the next incremental. I greatly desire an automated solution to the problem.

Thank you for your consideration,
Matt

I cannot speak for Macrium but I cannot imagine them doing something like you request.  It would seem too difficult.

Reflect already allows you to specify a second destination to use if the first is unavailable or full so you should be able to solve your issue by providing a second drive to use when/if the first is full.

You said you set your system to retain 50 backups.  I believe that means 50 backup SETS. A backup set is a full backup and all its incremental (and differential) files.  So as you describe your setup you would never have more than 1 backup set.

If ANY of those incrementals is bad or gets corrupted, all the incremental files following it are useless.  It is risky to have a lot of incremental files.  typically one full backup is followed by anywhere from 6 to 30 incrementals - the more incrementals the greater the risk of not being able to do a restore.

I don't see why this would be hard. Macrium Reflect can already merge incrementals to the full backup. I'm just asking that it be done automatically when the backup destination is almost out of space.

I know that there are ways around this, but I'm looking for a completely automatic solution. If it needs me to provide a secondary source, then that means I have to monitor the backup from start to finish in case this prompt appears. I shouldn't have to do this. It's a waste of the user's time. 

My backup task was slightly more complicated than I originally wrote. I actually have 5 hard drives divided into 7 partitions. I have a full backup of each partition, and then retain 50 incrementals overall. Not 50 incrementals per partition, but 50 incrementals spread out over all 7 partitions. Oldest incrementals are merged with the full backup when free space falls below 50 GB.

I think maybe you defined a part of the difficulty yourself when you said:
Neither is it possible for Reflect to know that in advance of the actual backup operation.  So, when it runs out of free space as it sometimes may, it lets the user tell it what to do next.  Speaking only for myself, I sure wouldn't want that case-by-case decision making option to be replaced by some "fully automated" guess even if it were possible.


Regards, Richard V. ("Arvy")

I said it would be hard, because if Reflect is in the middle of a backup it would have to somehow save what it was doing, do the incremental merge, then go back and see if can finish the backup with the extra space it created.  That sounds messy to me. Remembering that Reflect can only do one backup at a time, this would seem to be doing two operations at the same time. 

Reflect must compile a list of files it has to write for the next incremental. It would be a simple matter of adding all the file sizes together. Reflect has access to the file system and can do arithmetic faster than a mere human.

Also, Time Machine on OS X does exactly this. It calculates the space required for the next incremental, deletes the oldest backup files if there is not enough free space, and has had this functionality since 2007. This is not an impossible task.


I'm not asking that the default behavior be changed. I just want options in the backup definition file so that I can automate the choices if I want. If you don't like it, then you aren't required to use it.


That's why Reflect should calculate the free space required in advance, so that it can merge the incremental(s) first. Once that's done, then it can perform the next incremental backup.

You seem to be talking about image backups, yet your reference to Time Machine suggests a file backup.  Which is it?  Image backups do NOT deal in files they deal in clusters.  There is no easy way to figure out which clusters have changed until the program has gone through them all to figure this out.  Since I do not know how Reflect works, I cannot comment on whether it could stop what it is doing, or just abort the process, delete some files, then try again.  Maybe Time Machine was written,. from the start, with this capability built in.  It is usually much easier to build in functionality then to add it at a later time.  In any case I am not sure how important this is as a feature. With a terabyte external HDD costing about $60,  I would think you could afford to put a clean drive on the system before going on vacation (if you can afford vacations, you can probably afford another drive).

If this were a simple task for Macrium or if it was useful to some minimal percentage of its clients then I would think they would definitely consider a solution.  However it seems that there are several options available to you to resolve the problem without Macrium's intervention.

Perhaps Macrium could add the functionality to the File & Folder Backup. That would work as a partial solution. (I only need to image 2 of my 7 partitions. The rest are data drives.)

I need to backup about 1.8 TB of data plus incrementals, not a trivial amount of data. External HDD is 2 TB, so the remaining 200 GB is a precious resource. Larger hard drives are not that cheap unfortunately.