*** Save disk space converting (OLD or not...) differential backup files to incremental ***


Author
Message
Jordi
Jordi
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 12
Hello!
I suppose the reader knows the difference between differential and incremental post-full backup files:
- each differential in a determined backup set keeps track of the differences between the actual disk info and the full (first) backup, so you only have to restore (Macrium does it for us!) the full and the differential you choose to restore; of course a set can have a lot of differential files, so you can get back to any situation you have a differential of, and it's fast because you only use the two involved files, as described before. Each differential in a set contains redundant info that can be found in the previous files.
- each incremental in a determined backup set keeps track of the differences between the actual disk info and the previous backup, either the full one (first incremental) or the previous incremental one. This makes more difficult to get to a certain disk status if you have a few incremental files, as you have to restore the full and all these few files, and it can take more time.

My proposal is to have an utility that converts the (oldest or not! at your will) differential files in each selected set to incremental; the oldest ones having redundant info are (more...) eligible because they are old and not so often used to restore good versions of our disk; we keep it "just in case", but could be relieved from having all this extra, redundant info and save disk space. I personally dislike the option of compacting the full + differentials to a single file because it loses the possibility of recovering to an in-between time disk status, for instance when you install some software that keeps your PC from booting and it can only boot from PE environment; you have to restore it to the best previous working version!)

Thanks for reading this!
Jordi

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
You seem to be assuming that a backup set will consist of a Full and then just a series of Differentials.  Some people do use that strategy, but it's also possible to have something like monthly Fulls, weekly Diffs, and daily Incrementals.  In that case, each Differential will have its own chain of child Incrementals.  And for that matter, the Full could have its chain of Incrementals as direct children of the Full before the first Differential in the set.  In those cases, converting a Differential to an Incremental would not really be practical.  Reflect does not support "forked" sets, i.e. you could not have multiple completely independent chains of Incrementals all as direct children of a Full.  So suppose your backup strategy created a Full, then Incrementals #1-3, then Differential #2, then Incrementals #4-6, then Differential #2.  It wouldn't make a lot of sense to turn Differential #2 into an Incremental appended to Incremental #3.

Additionally, you seem to be basing this idea on the idea that data redundancy and disk consumption is a bad thing.  Typically when a user chooses to use Differentials, data redundancy is the GOAL because the user specifically WANTS to use a strategy that has fewer dependencies, i.e. a Differential and its parent Full rather than an entire chain of Incrementals.  Typically there is a tradeoff between resiliency/redundancy and disk consumption -- just like a user could choose not to make backups at all and save disk space, but normally they will WANT to have "redundant" copies of their data as backups for safety.  But if a Reflect user wanted to optimize for low storage consumption and was willing to give up redundancy to get there, then they could just use an all-Incrementals strategy in the first place.

Also, if you have a Differential or even a chain of multiple Incrementals, you don't have to restore the Full first and then manually restore the Differential or each Incremental one at a time afterward.  You can just choose to restore whichever backup you want to restore from, and Reflect will automatically pull the necessary data from previous backups in the set to perform that restore in a single operation.

Edited 4 July 2021 3:18 AM by jphughan
Jordi
Jordi
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 12
jphughan - 4 July 2021 3:16 AM
You seem to be assuming that a backup set will consist of a Full and then just a series of Differentials.  Some people do use that strategy, but it's also possible to have something like monthly Fulls, weekly Diffs, and daily Incrementals.  In that case, each Differential will have its own chain of child Incrementals.  And for that matter, the Full could have its chain of Incrementals as direct children of the Full before the first Differential in the set.  In those cases, converting a Differential to an Incremental would not really be practical.  Reflect does not support "forked" sets, i.e. you could not have multiple completely independent chains of Incrementals all as direct children of a Full.  So suppose your backup strategy created a Full, then Incrementals #1-3, then Differential #2, then Incrementals #4-6, then Differential #2.  It wouldn't make a lot of sense to turn Differential #2 into an Incremental appended to Incremental #3.

Additionally, you seem to be basing this idea on the idea that data redundancy and disk consumption is a bad thing.  Typically when a user chooses to use Differentials, data redundancy is the GOAL because the user specifically WANTS to use a strategy that has fewer dependencies, i.e. a Differential and its parent Full rather than an entire chain of Incrementals.  Typically there is a tradeoff between resiliency/redundancy and disk consumption -- just like a user could choose not to make backups at all and save disk space, but normally they will WANT to have "redundant" copies of their data as backups for safety.  But if a Reflect user wanted to optimize for low storage consumption and was willing to give up redundancy to get there, then they could just use an all-Incrementals strategy in the first place.

Also, if you have a Differential or even a chain of multiple Incrementals, you don't have to restore the Full first and then manually restore the Differential or each Incremental one at a time afterward.  You can just choose to restore whichever backup you want to restore from, and Reflect will automatically pull the necessary data from previous backups in the set to perform that restore in a single operation.



Jordi
Jordi
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 12

Hi jp,

Thanks for your very good contribution! But...
"Additionally, you seem to be basing this idea on the idea that data redundancy and disk consumption is a bad thing." Well, not at all; I think redundant OLD disk space is eligible to be get rid of, unless you're a hard disk manufacturer!
I've been in the mini (PDP digital) and PC area since 1976 so I think I know what I'm talking about! (and I know you are too, of course!)
The modern hard disks are not so reliable (or I have bad experience with those I bought lately, 2x8TB) and I want to keep my surviving disks with the most useful data I consider good to not be deleted, that's all. My home installation has been saved sometimes by an EVO and an "standard" NAND Flash units with 6+ months Windows installations! And that's good, for sure!
Of course I didn't mind of the other schemas you mentioned above, as I've always (professionally and at home) opted for simple ones.
Just to put this to almost an end, do you think only few people use my simple schema and keep lots and lots of old backups so the utility I was asking for will not be in other people wish-list?
Thanks! (any offence here is, of course, unintended; sorry if I did it!)

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)Macrium Evangelist (15K reputation)
Group: Forum Members
Posts: 10K, Visits: 65K
No offense taken at all. I just have difficulty envisioning a use case here. If somebody originally implemented a strategy that involved making a series of Diffs when they could have opted for a series of Incs, then typically it will have been because they wanted some benefits of Diffs over Incs, even at the cost of additional storage consumption. The use case here would be people who needed to free up space, AND weren’t willing to delete backups outright, AND were willing to give up the benefits of Diffs, AND had backup sets containing ONLY Diffs rather than any existing Incs. I don’t know how many cases would fit all of those criteria where users wished they could retroactively change their mind by converting their existing Diff group into an Inc chain.

Another potential wrinkle here is that only paid versions of Reflect create Incs, so this utility would have to check for licensing to maintain that feature differentiation. Given how long Diffs and Incs have been around, this to me sounds like something that would exist already if years of customer experience indicated there was a need for it. But you never know, and this is after all the Wish List section. Smile
Edited 4 July 2021 7:36 PM by jphughan
Drac144
Drac144
Guru
Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)
Group: Forum Members
Posts: 688, Visits: 2.7K
I have been using mini's (Including PDP-11) and PC's since 1972 - and before that I worked on larger computers.  And back then a hard disk (if there was one even available) was the size of a large refrigerator and had an MTBF that was measured in weeks and there was no error correction logic as there is today. Oh, and they cost thousands of dollars for a less than a mega-word of storage. But, yes, those drives were very low capacity (kilobytes would be a sufficient unit) - and most did not work in bytes - there were "words" which reflect the basic size of the computer's registers - typically in multiples of 6 bits.  But disk space was expensive so every word counted.  Now one can get multiple terabyte drives for under $100 and fast (SSD) terabyte drives for about $120.  So disk space is no longer precious.  

So I do not understand why you expect others (Macrium) to spend their valuable time and their effort to create a utility so you can recover a dollar's worth of disk space.  That does not seem productive or useful.  I never use differential backups - just full and incremental.  So I have no need for the utility you requested.  But if that utility is of use to even a few percent of Reflect users, then it certainly should be considered for development. 

Jordi
Jordi
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 12
Drac144 - 4 July 2021 8:48 PM
I have been using mini's (Including PDP-11) and PC's since 1972 - and before that I worked on larger computers.  And back then a hard disk (if there was one even available) was the size of a large refrigerator and had an MTBF that was measured in weeks and there was no error correction logic as there is today. Oh, and they cost thousands of dollars for a less than a mega-word of storage. But, yes, those drives were very low capacity (kilobytes would be a sufficient unit) - and most did not work in bytes - there were "words" which reflect the basic size of the computer's registers - typically in multiples of 6 bits.  But disk space was expensive so every word counted.  Now one can get multiple terabyte drives for under $100 and fast (SSD) terabyte drives for about $120.  So disk space is no longer precious.  

So I do not understand why you expect others (Macrium) to spend their valuable time and their effort to create a utility so you can recover a dollar's worth of disk space.  That does not seem productive or useful.  I never use differential backups - just full and incremental.  So I have no need for the utility you requested.  But if that utility is of use to even a few percent of Reflect users, then it certainly should be considered for development. 
"So I do not understand why you expect others (Macrium) to spend their valuable time and their effort to create a utility so you can recover a dollar's worth of disk space" Well, perhaps because I pay Macrium (as others do, of course) a license to have inc backups; instead, disk space costs me a lot of money I prefer to save if possible; as I've pointed out before, I have 2x8TB HDU, and one got blank without warning me; 350€, not cheap when my PC, the other disks etc have already a 4 near 5 figure budget, not counting photo cameras and electronic keyboards to mention a few; I'm unemployed since 2012 and every € counts!
I realize you are a bit older than me, and perhaps, probably, wealthier too. But don't forget that companies are supposed to give service to its customers, not the other way round... I want to save disk space, and I'm looking for an utility useful to me and perhaps to other people, that's it!
"But if that utility is of use to even a few percent of Reflect users, then it certainly should be considered for development" that's what I think too!

Thank you everybody!

Drac144
Drac144
Guru
Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)
Group: Forum Members
Posts: 688, Visits: 2.7K
Jordi,

I did assume you were on a tight budget or the decrease in disk space would not be an issue for you.  My point was that there are a lot of wish list requests and macrium has limited resources. So they must prioritize and make the improvements that benefit the most users.  I hope you would agree with that.

You have many options in the way you do your backups, how often you do backups, how long you keep backups, etc.  Possibly your backup strategy could be changed to reduce storage needs for backups, WITHOUT sacrificing realistic security for your data. Before asking Macium to provide you a utility that will reduce space needed for your backups - based upon YOUR backup philosophy - perhaps you should see if there are ways to make storage space reductions other than via the requested feature.
Jordi
Jordi
New Member
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Awaiting Activation
Posts: 5, Visits: 12
Drac144 - 5 July 2021 8:16 PM
Jordi,

I did assume you were on a tight budget or the decrease in disk space would not be an issue for you.  My point was that there are a lot of wish list requests and macrium has limited resources. So they must prioritize and make the improvements that benefit the most users.  I hope you would agree with that.

You have many options in the way you do your backups, how often you do backups, how long you keep backups, etc.  Possibly your backup strategy could be changed to reduce storage needs for backups, WITHOUT sacrificing realistic security for your data. Before asking Macium to provide you a utility that will reduce space needed for your backups - based upon YOUR backup philosophy - perhaps you should see if there are ways to make storage space reductions other than via the requested feature.

Hi Drac144,
Yes, you're right, I'm not a millionaire... But even if I were, I would be concerned (or an employee of mine! BigGrin) too, for a couple of reasons:
1- Every time one of my disks
(8 fixed + OS-SSD, and about 18 USB external, USB mountable at will, between 1 and 8 TB each, HDDs, SSDs...) dies or gets full, a lot of work has to be done; transferring between full and not-so-full disks, try recovering dead ones, trying to rescue old info based on file inventory, and so on... so if Macrium believe this is a good idea to implement, I'll be very happy and thankful!
2- This morning I've decided to use Inc schema... but changed back my mind; as someone has pointed out before, redundancy IS A GOOD THING! If your Inc backup #6 gets lost or corrupted, any next backup (#7, #8,...) become useless, or almost... Again, I will get the risk in older backups, not in the most recent ones...
Just for the record, I keep different backups of my photos since 2008 (and only since 2012 there are 5.3TB of unique data plus more than 17TB in spread copies... not an easy job maintain this; so its when my OS backups fill the backup disk that I'm concerned... I'll consider to remove some oldest sets after this summer... but I'll better keep the backups that saved my (PC's) life!!

Thanks everybody for posting here, it's been very enlighting! (and excuse my poor english, never studied it!)

Drac144
Drac144
Guru
Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)Guru (1K reputation)
Group: Forum Members
Posts: 688, Visits: 2.7K
Jordi,

WOW, that is a LOT of disks.  I have 2 internal disks in my computer (1 SSD and 1 spinner HDD).  I have 2 external drives.  It sounds like your photography hobby (or profession) is quite expensive.  Photography is one of my hobbies.  However I do not take a lot of pictures (But do have many that have been collected over the years).  I am quite selective, I initially delete about half the pictures I take as being not worth saving, not as good as some others, or too similar to another picture to justify keeping both.  I also delete pictures, over time, that in retrospect, do not seem worth keeping. So, in the long run, of each 100 pictures I take, I may only keep about 35.  I could probably review many of my older pictures and delete even more that I have not looked at or are, based on my current tastes and criteria, not all that great.  I am NOT suggesting that you delete some of your pictures.  I am just mentioning my own philosophy.

As you mention, there are issues with longer strings of incremental backups.  However what I do not understand is why you do not just make new full backups and then delete older backup sets (keeping a few older sets for redundancy).  If you have backups going back years for a set of pictures, there would not seem to be any value in the older backups if newer backups included those older pictures.  That is what I meant in a previous post, when I mentioned changes to your backup strategy. My point in these suggestions is to help you find ways to reduce the space needed for your backups (which is what you are trying to do with your original wish) without depending on Macrium to make a utility to change a differential to an incremental.


GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search