Feature: Convert Differential backup file to its Incremental backup equivalent


Feature: Convert Differential backup file to its Incremental backup...
Author
Message
Alan1250
Alan1250
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 8, Visits: 19
I'm looking for a way to convert a series of Full and Differential backups into Incremental ones for archival purposes.  Is there already a way to do this?
Beardy
Beardy
Proficient Member
Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)
Group: Forum Members
Posts: 313, Visits: 1.4K
I'm not aware of any method, it wouldn't be easy to do either, & in some scenarios I could think up I wouldn't trust the outcome of any process I could think up to do it.
It would also render any child incrementals of the differentials useless since they reference that rather than the parent full.

You'd basically have to restore each and every machine state in turn & create a chain of incrementals by running a backup of each state in order to "convert".

You also have the issue that the longer a chain of inrementals becomes, the less reliable it is as an archive, since one unreadable block anywhere renders the entire chain afterwards unuseable, if that's in the parent full you've now lost everything.  Differentials are usually used to mitigate against that risk.

Take a year of daily backups starting January 1st as an extreme example to have a reliable archive & access the state at the end of the year you have to read the parent full & a chain of over 360 increments. Add weekly differentials and your longest chain at any point is full + diff + 6 increments. Damage anywhere except the parent full loses you at most 1 week of states, whereas in the chain of increments, damage to one in January loses you access to the rest of the entire year.

In view of the above, I'd see such a feature as undesirable.
Alan1250
Alan1250
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 8, Visits: 19
Beardy - 26 February 2021 6:55 PM
I'm not aware of any method, it wouldn't be easy to do either, & in some scenarios I could think up I wouldn't trust the outcome of any process I could think up to do it.
It would also render any child incrementals of the differentials useless since they reference that rather than the parent full.

You'd basically have to restore each and every machine state in turn & create a chain of incrementals by running a backup of each state in order to "convert".

You also have the issue that the longer a chain of inrementals becomes, the less reliable it is as an archive, since one unreadable block anywhere renders the entire chain afterwards unuseable, if that's in the parent full you've now lost everything.  Differentials are usually used to mitigate against that risk.

Take a year of daily backups starting January 1st as an extreme example to have a reliable archive & access the state at the end of the year you have to read the parent full & a chain of over 360 increments. Add weekly differentials and your longest chain at any point is full + diff + 6 increments. Damage anywhere except the parent full loses you at most 1 week of states, whereas in the chain of increments, damage to one in January loses you access to the rest of the entire year.

In view of the above, I'd see such a feature as undesirable.
Thank you for your response.

There are of course many types of risks to backup data.  Individual file data corruption being only one of them.  I'm also concerned about whole backup drive failures and ransomware attacks which can take out a whole backup set regardless of differential/incremental strategies.  I start each year with a full backup, and after that I do use a combination of differentials and incrementals on a monthly-daily basis for my nightly backup stage.  But after that they are automatically verified and copied to NAS units.  Monthly the whole set is copied to a NAS disk set that goes into a fireproof safe.  Nightly incrementals also get automatically copied to a Backblaze cloud server.

I'm only looking to use such a conversion on the historic prior year sets of backups.  These are also well protected against individual file corruption but consume a significant amount of space.  I would only be creating small number of incrementals for the months.  Daily incrementals have been discarded by that point.

To summarize, in my case I do not have to worry about incremental file corruption.  But if I could reduce the eleven differentials to incrementals, that would be a plus.





jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 64K
You can't convert a series of Differentials that have a common parent Full into a single Incremental chain.  The only sort of conversion that's allowed is that Reflect has a utility called Consolidate.exe that allows you to consolidate a chain of Incrementals into either a single Incremental or their parent Full.  The latter is only possible if the set has no Differentials though, since otherwise modifying the Full after its initial creation this way would break the Differentials and any child Incrementals they had.  So if you were to switch to using Incrementals, you would have a way of taking a month's worth of daily Incrementals and consolidating them all to a single remaining Incremental that matched the Day 30 Incremental, or carry the root Full forward to the state of the Day 30 Incremental.  Reflect supports these operations as part of its normal retention policy operations as well, but if you only want to consolidate after the fact, that would be a manual process, which is why Macrium provides this standalone application.  You'll find it in the Reflect program directory, but of course that would require rethinking your current strategy, which may or may not be acceptable for you.

Beardy
Beardy
Proficient Member
Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)
Group: Forum Members
Posts: 313, Visits: 1.4K
@Alan1250 If you really want to, this strategy might work:

  1. Restore the full backup to a spare disk.
  2. Make a full backup of the spare disk in a new folder where you're storing "Backups to be archived".
  3. Restore the first Diff to the same disk.
  4. Make an incremental in your new set.
  5. Restore the next Diff & create incrementals till you run out of Diffs.
  6. Archive your new set.
  7. Delete the original set & optionally clear out your spare drive.

Don't be too surprised if you don't save as much space as you hope after all that time and effort.

Personally I'd just budget for a big enough drive each year to dump the full & diffs on rather than jump through such hoops the extra space is probably cheaper than the value of your time.. put a nice clear label on it & store it.

The "proper" approach would be to archive the set on tape rather than hard drive probably on an "as you go along" basis, it's better for long term archival storage & cheaper per TB than hard drives, though it involves an investment in a drive.


Alan1250
Alan1250
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 8, Visits: 19
Beardy - 26 February 2021 9:08 PM
@Alan1250 If you really want to, this strategy might work:

  1. Restore the full backup to a spare disk.
  2. Make a full backup of the spare disk in a new folder where you're storing "Backups to be archived".
  3. Restore the first Diff to the same disk.
  4. Make an incremental in your new set.
  5. Restore the next Diff & create incrementals till you run out of Diffs.
  6. Archive your new set.
  7. Delete the original set & optionally clear out your spare drive.

Don't be too surprised if you don't save as much space as you hope after all that time and effort.

Personally I'd just budget for a big enough drive each year to dump the full & diffs on rather than jump through such hoops the extra space is probably cheaper than the value of your time.. put a nice clear label on it & store it.

The "proper" approach would be to archive the set on tape rather than hard drive probably on an "as you go along" basis, it's better for long term archival storage & cheaper per TB than hard drives, though it involves an investment in a drive.

Yes, thanks.  I'd already considered doing something like you suggested.  But I also had concluded it wouldn't be worth the effort.  I was looking for something a lot easier to be worth the disk savings.
Beardy
Beardy
Proficient Member
Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)Proficient Member (382 reputation)
Group: Forum Members
Posts: 313, Visits: 1.4K
Even if a tool existed, I expect it would pretty much have to do that dance, though maybe using a temporary image or VHD rather than a spare physical disk, & using tons of scratch space while processing.  I can't see the use case arising often enough for Macrium to invest the developer time in creating such a tool, or most users wanting to spend the time running it if it did exist.
Alan1250
Alan1250
New Member
New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)New Member (13 reputation)
Group: Forum Members
Posts: 8, Visits: 19
Beardy - 26 February 2021 11:38 PM
Even if a tool existed, I expect it would pretty much have to do that dance, though maybe using a temporary image or VHD rather than a spare physical disk, & using tons of scratch space while processing.  I can't see the use case arising often enough for Macrium to invest the developer time in creating such a tool, or most users wanting to spend the time running it if it did exist.

Well, in my opinion, (as a software engineer with over 50 years of experience), conceptually it could be accomplished just using the input files.  But the difficulty in developing it would depend largely on the available code base and current object-oriented design.   Only the Macrium software development team could determine that.  I doubt there is enough demand for such a tool though, since no one else seems to have asked about it.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 64K
I think another practical consideration that might contribute to reduced demand is that I suspect the majority of scenarios that involve Differential backups also involve Incrementals, i.e. relatively few use cases will use only Full and Differential backups.  In that case, this conversion from a series of Differentials into an Incremental chain would only be feasible if none of the Differentials involved in the conversion had child Incrementals of their own, since all of those would be invalidated as a result of this conversion.  That's certainly not to say this tool couldn't exist, since that restriction could certainly be enforced in a hypothetical conversion utility, but of course Macrium's development/engineering time is a finite resource and lots of users want a variety of things.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search