Can't access Full backup after manual consolidation - help please


Author
Message
TimFelix
TimFelix
New Member
New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)
Group: Forum Members
Posts: 3, Visits: 6
Hello there.

So.. I'm having troule recovering photos from a backup after using the consolidation executable. I may be missunderstanding the intended function of this tool and would really appreciate any insight anyone has. Please read on if this is of interest..

I had a Full backup fo all my photos (backup folder was M:\Memories, backup destination was on drive letter Y:\Memories_backup) the backup file was about 400GB
I tried to do an incremental backup on top of this but was told the backup folder did not exist. The drive letter of the backup source folder had somehow changed after making the Full backup.
So I changed the drive letter back to match the Full backup (Y:\)
I was able to run the incremental backup. This bacup file was about 50GB

I then wanted to consolidate the Full and Incremental backup into one file using the consolidate.exe utility
I chose the Full backup as the 'From' and the Incermental backu as the 'To'

I now have a single backup file of about 500GB.. so that seemed good.
But now when I try to restore of browse the backup, the first directory is 'Drive Y' and it only contains about 50GB of photos.
I'm not sure what this window (please see below) means.
I'm hoping as the .mrbak backup file is about 500GB the photos from the Full backup are in there somewhere.


Any ideas very gratefully received..
a little scared i've lost all my family photos.

-Tim


dbminter
dbminter
Macrium Hero
Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)Macrium Hero (2.3K reputation)
Group: Forum Members
Posts: 1.7K, Visits: 18K
I can only comment on what that image you posted means.


That's the interface for mounting a backup image set as a virtual drive.  It lists the properties of the image/drive that will be mounted.  When you check the check box on the far left side, you'll be prompted to assign the virtual drive a drive letter.  If the image backup is password protected, when you "check" that check box, you'll be prompted to enter the password.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 7.9K, Visits: 55K
I can provide insight as to how the purpose and function of the consolidation tool, but first I want to understand exactly what happened.  You say that when your Full was captured, you backed up source folder "M:\Memories" to a destination of "Y:\Memories_backup".  But then you said that on the initial failed Incremental attempt, the source folder had changed, so you changed it [the source] to match the full backup of "Y:\".  My confusion is that in the first statement, Y is your destination, and in the second statement, Y is your updated source to match your Full.  Which is correct?

Additionally, please go into the settings for the XML definition file you're using for this backup job and let me know what the Backup Set Matching option is currently set to.

Depending on your answers to clarify the first point above and the state of your Backup Set Matching setting, unfortunately it is POSSIBLE that you captured an Incremental that essentially overwrote your Full.  And if you ran a consolidation, then for reasons I can get into in an explanation of how Consolidate.exe works, you would have essentially replaced the contents of the original Full in that case as well.  The file size of the backup file itself is large because consolidation never shrinks files; it only expands them as needed.  However, since you've only performed ONE consolidation thus far, then due to the way consolidation works, technically all of the data blocks needed by the old Full that you just replaced are still in there.  Whether or not those are usable in some fashion is an entirely separate question, though, and even if they are, you would probably need help from Macrium Support to make those data blocks usable somehow again.

Is that truly the only backup you had?

Edited 21 September 2020 2:47 PM by jphughan
TimFelix
TimFelix
New Member
New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)
Group: Forum Members
Posts: 3, Visits: 6
jphughan - 21 September 2020 2:44 PM
I can provide insight as to how the purpose and function of the consolidation tool, but first I want to understand exactly what happened.  You say that when your Full was captured, you backed up source folder "M:\Memories" to a destination of "Y:\Memories_backup".  But then you said that on the initial failed Incremental attempt, the source folder had changed, so you changed it [the source] to match the full backup of "Y:\".  My confusion is that in the first statement, Y is your destination, and in the second statement, Y is your updated source to match your Full.  Which is correct?

Additionally, please go into the settings for the XML definition file you're using for this backup job and let me know what the Backup Set Matching option is currently set to.

Depending on your answers to clarify the first point above and the state of your Backup Set Matching setting, unfortunately it is POSSIBLE that you captured an Incremental that essentially overwrote your Full.  And if you ran a consolidation, then for reasons I can get into in an explanation of how Consolidate.exe works, you would have essentially replaced the contents of the original Full in that case as well.  The file size of the backup file itself is large because consolidation never shrinks files; it only expands them as needed.  However, since you've only performed ONE consolidation thus far, then due to the way consolidation works, technically all of the data blocks needed by the old Full that you just replaced are still in there.  Whether or not those are usable in some fashion is an entirely separate question, though, and even if they are, you would probably need help from Macrium Support to make those data blocks usable somehow again.

Is that truly the only backup you had?

Hi @jphughan (and thanks @dbminter)
Thank you for your answer.. you make it sound pretty simple when you put it like that!
To be honest I'm not totally sure what happenned.
I now suspect that what may have happenned was that my first fulll backup was taken when the folder being backed up was assigned to a different drive letter.
So when doing the incremental, I was told 'Y:\Memories' was not found, i used the windows disk management tool to change the drive letter to Y.
So then I was able to do the incremental backup.
So I then had the full backup and the incremental backup. both part of the same backup set (matching backup ID's)
I then tried to consolidate these two backups. (From the full, to the incremental)

I stupidly don't think I have a second backup of these photos.
I would be willing to renew my support licence if this can be recovered.
Maybe there's a way to separate the two backups again.

-Tim


Edited 22 September 2020 3:54 AM by TimFelix
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)Macrium Evangelist (11K reputation)
Group: Forum Members
Posts: 7.9K, Visits: 55K
If you want any hope at all of being able to recover the contents of the original Full, do NOT consolidate another backup into that file.

But if you're curious, I'll give you a bit of background on how consolidation works (and why it works that way), as well as my theory as to how you may have ended up here.

Picture a scenario where you create a Full backup on Sunday and create Incrementals on each of the subsequent days that week.  You now have a Full backup and 6 Incrementals from Monday through Saturday.  Let's say that later on, you decide you only really want to preserve the Saturday backup "state".  You no longer need the ability to restore to Sunday or any subsequent day; you just want to be able to recover to Saturday if needed later on.  Since you have a Full and a bunch of Incrementals, you can't simply delete the Sunday through Friday backups, since the nature of Incremental backups means that would render the Saturday backup useless.  But those intermediate backups are taking up space preserving unique "states" that you don't care about anymore.  The consolidation utility offers a solution to this.  If you choose to consolidate from the Sunday Full to the Saturday Incremental, the end result will be a single Full backup that contains the "state" of your data as it was at the time you made the Saturday backup.  This is called a "Synthetic Full", since you have a Full with contents equivalent to what you would have gotten if you had captured a Full backup on Saturday, even though you didn't actually do that.  But it's important to note that the individual "states" of all of those intermediate days are NOT available anymore. (Don't lose hope for your predicament based on what i just said; there's more to the story.)  The purpose of consolidation is NOT to preserve multiple backup states in a single file, i.e. you would not be able to use that consolidated backup to restore to your Tuesday state.  The purpose of consolidation is to retain only the data from those intermediate backups that is necessary to keep the Saturday backup usable.  So for example if you created a 50 GB file during that week and it ended up in your Friday backup, but you deleted it before your Saturday backup, then your consolidated Synthetic Full meant to retain only the Saturday state would not include that file anywhere.

Ok, so how is it that your data might still be there even though I said that the preceding backup states aren't retained?  That has to do with how the consolidation operation itself works in order to protect against data corruption if any unexpected events arise during consolidation.  Let's take a simpler example involving only a Sunday Full that captured 50 GB of source data, and a Monday Incremental that was captured after you had deleted 5 GB of source data and created 5 GB of completely different data.  (Assume uncompressed backup files for simplicity here.)  So you have a 50 GB Full and a 5 GB Incremental.  If you consolidate those backup files, you do NOT end up with a 50 GB Full even though the net change in source data between the Full and that first Incremental was 0 GB (5 GB deleted, 5 other GB created).  You will instead end up with a 55 GB Synthetic Full.  Why is that?  Because when Reflect performs a consolidation, any data blocks within the Full that are required for the backup that it stores are NOT overwritten.  Instead, the Full will grow as needed to "absorb" the new backup WITHOUT overwriting any of its existing data blocks.  The reason this is done is because if the consolidation operation fails partway through for whatever reason, the fact that none of the original data blocks were overwritten means that the original, pre-consolidation state of that backup is recoverable.  Reflect would just have to ignore the new data blocks from the failed consolidation.  If on the other hand the consolidation succeeds, then Reflect updates that backup to actually use those new data blocks, and then any data blocks that aren't needed to retain the NEW state of that backup get marked as scratch space.  THOSE data blocks will be eligible to be overwritten during the NEXT consolidation operation.  This means that the Full doesn't have to grow by the full amount of every backup being consolidated into it.  Instead, existing data blocks can be overwritten, but only if they aren't needed by the current state of the backup.

So, because you've only consolidated one backup into your Full, all of the data blocks from that original state should still be there.  But again, I have no idea whether you can actually "resurrect" that pre-consolidation state.  And if you can, I expect it will involve assistance from Macrium Support.

In terms of my theory as to how you ended up with an Incremental that seemingly blew away most of your photos, I'm looking at the drive letter confusion and that Backup Set Matching setting I asked you to check.  The default Backup Set Matching setting is Similar, which means that as long as an existing backup contains at least one source folder in common with the new backup you're making, then Reflect will allow you to create an Incremental or Differential from that existing backup.  But if you change that setting to "All", then Reflect will allow you to make a Differential or Incremental from an existing backup even if the existing backup and the one you're running now have NO source folders in common.  That plus the drive letter confusion makes leads me to suspect that your original Full backed up M:\Memories.  And then for your Incremental you may have inadvertently backed up only some OTHER folder that contained fewer source files, rather than the same source folder you originally backed up in your Full.  Either that or you may have assigned the original drive letter to the wrong drive and backed up a different source folder, even though the path was the same as the original Full.  Either way, the result will be an Incremental that essentially says, "The source data includes this one folder [which is different from your original Full] and nothing else, i.e. it does NOT include the folder that was originally backed up in your Full."  If you then choose to consolidate that Incremental into the root Full, you will end up with a single backup that matches the state of that Incremental, which contains only that second, incomplete folder and no record at all of the folder originally backed up in your Full.  But again, technically the data blocks would still be there, and potentially recoverable.

Hopefully you were able to follow all of that, and either way, good luck!! Smile

TimFelix
TimFelix
New Member
New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)New Member (5 reputation)
Group: Forum Members
Posts: 3, Visits: 6
jphughan - 22 September 2020 4:21 AM
If you want any hope at all of being able to recover the contents of the original Full, do NOT consolidate another backup into that file.

But if you're curious, I'll give you a bit of background on how consolidation works (and why it works that way), as well as my theory as to how you may have ended up here.

Picture a scenario where you create a Full backup on Sunday and create Incrementals on each of the subsequent days that week.  You now have a Full backup and 6 Incrementals from Monday through Saturday.  Let's say that later on, you decide you only really want to preserve the Saturday backup "state".  You no longer need the ability to restore to Sunday or any subsequent day; you just want to be able to recover to Saturday if needed later on.  Since you have a Full and a bunch of Incrementals, you can't simply delete the Sunday through Friday backups, since the nature of Incremental backups means that would render the Saturday backup useless.  But those intermediate backups are taking up space preserving unique "states" that you don't care about anymore.  The consolidation utility offers a solution to this.  If you choose to consolidate from the Sunday Full to the Saturday Incremental, the end result will be a single Full backup that contains the "state" of your data as it was at the time you made the Saturday backup.  This is called a "Synthetic Full", since you have a Full with contents equivalent to what you would have gotten if you had captured a Full backup on Saturday, even though you didn't actually do that.  But it's important to note that the individual "states" of all of those intermediate days are NOT available anymore. (Don't lose hope for your predicament based on what i just said; there's more to the story.)  The purpose of consolidation is NOT to preserve multiple backup states in a single file, i.e. you would not be able to use that consolidated backup to restore to your Tuesday state.  The purpose of consolidation is to retain only the data from those intermediate backups that is necessary to keep the Saturday backup usable.  So for example if you created a 50 GB file during that week and it ended up in your Friday backup, but you deleted it before your Saturday backup, then your consolidated Synthetic Full meant to retain only the Saturday state would not include that file anywhere.

Ok, so how is it that your data might still be there even though I said that the preceding backup states aren't retained?  That has to do with how the consolidation operation itself works in order to protect against data corruption if any unexpected events arise during consolidation.  Let's take a simpler example involving only a Sunday Full that captured 50 GB of source data, and a Monday Incremental that was captured after you had deleted 5 GB of source data and created 5 GB of completely different data.  (Assume uncompressed backup files for simplicity here.)  So you have a 50 GB Full and a 5 GB Incremental.  If you consolidate those backup files, you do NOT end up with a 50 GB Full even though the net change in source data between the Full and that first Incremental was 0 GB (5 GB deleted, 5 other GB created).  You will instead end up with a 55 GB Synthetic Full.  Why is that?  Because when Reflect performs a consolidation, any data blocks within the Full that are required for the backup that it stores are NOT overwritten.  Instead, the Full will grow as needed to "absorb" the new backup WITHOUT overwriting any of its existing data blocks.  The reason this is done is because if the consolidation operation fails partway through for whatever reason, the fact that none of the original data blocks were overwritten means that the original, pre-consolidation state of that backup is recoverable.  Reflect would just have to ignore the new data blocks from the failed consolidation.  If on the other hand the consolidation succeeds, then Reflect updates that backup to actually use those new data blocks, and then any data blocks that aren't needed to retain the NEW state of that backup get marked as scratch space.  THOSE data blocks will be eligible to be overwritten during the NEXT consolidation operation.  This means that the Full doesn't have to grow by the full amount of every backup being consolidated into it.  Instead, existing data blocks can be overwritten, but only if they aren't needed by the current state of the backup.

So, because you've only consolidated one backup into your Full, all of the data blocks from that original state should still be there.  But again, I have no idea whether you can actually "resurrect" that pre-consolidation state.  And if you can, I expect it will involve assistance from Macrium Support.

In terms of my theory as to how you ended up with an Incremental that seemingly blew away most of your photos, I'm looking at the drive letter confusion and that Backup Set Matching setting I asked you to check.  The default Backup Set Matching setting is Similar, which means that as long as an existing backup contains at least one source folder in common with the new backup you're making, then Reflect will allow you to create an Incremental or Differential from that existing backup.  But if you change that setting to "All", then Reflect will allow you to make a Differential or Incremental from an existing backup even if the existing backup and the one you're running now have NO source folders in common.  That plus the drive letter confusion makes leads me to suspect that your original Full backed up M:\Memories.  And then for your Incremental you may have inadvertently backed up only some OTHER folder that contained fewer source files, rather than the same source folder you originally backed up in your Full.  Either that or you may have assigned the original drive letter to the wrong drive and backed up a different source folder, even though the path was the same as the original Full.  Either way, the result will be an Incremental that essentially says, "The source data includes this one folder [which is different from your original Full] and nothing else, i.e. it does NOT include the folder that was originally backed up in your Full."  If you then choose to consolidate that Incremental into the root Full, you will end up with a single backup that matches the state of that Incremental, which contains only that second, incomplete folder and no record at all of the folder originally backed up in your Full.  But again, technically the data blocks would still be there, and potentially recoverable.

Hopefully you were able to follow all of that, and either way, good luck!! Smile

Hey @jphughan
Thank you so much! despite my concerns about the photos I somehow feel a lot better off for having read and understood your very thoughtful reply.
It sees a lot clearer now. Your explaination of how I may have got my incrmental miss-matched sounds very familiar and probably what happenned.
If I can't find a second backup I'll contact the Support guys and see if they can take a look.
Thank you again. Have a good one as they say here in Canada. Smile


GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search