Robustness of Reflect image files


Author
Message
Alan
Alan
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 8, Visits: 32
A few questions for the experts....

How robust are Reflect image files ?
More specifically, how many bit errors can they suffer after creation before an image file is likely to be rendered useless ?
Would not using compression at backup time reduce the risk ?
Are the answers somewhere in the Reflect user manual ? I didn't find them.

These questions relate to post-backup data integrity of the Reflect image files, assuming that they were viable when originally created.

Background:
1. I'm trying to decide whether I should be creating large image files (about 1TB) or break them down into smaller chunks at backup time. If a single-bit error affects only a single file then I might as well use large image files, but if the whole image file will be rendered useless then I need a larger number of partial backups.
2. I'm trying to decide how many redundant copies I need of my backups.
3. During the past few decades I have had many computer hardware failures (in addition to even more OS, software and user errors) involving every type of component in a computer. Most of them started as a barely perceptible deterioration that gradually worsened in a vague, intermittent and largely undiagnosable manner, until it was too late. It seems that I don't do sudden, total failures - they would be blissfully easy to find and overcome Smile 

Any help will be much appreciated.



jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)
Group: Forum Members
Posts: 4.9K, Visits: 36K
Subscribing to get the answer to this question.  I haven't read anything one way or another about whether Reflect adds any parity data to its files to grant some tolerance for bad data.  I know WinRAR can optionally do this when compressing data into a RAR file, for example, and you even can specify how much parity data you want -- the more you have, the more errors you can recover from, but the larger the file you get.  I don't see why you're thinking that compression would reduce the risk of bad data though, except in the sense that a compressed file inherently means there are fewer bits that can "go bad" in the first place, but I wouldn't really consider that a source of robustness.

I also don't see what you're envisioning with your split vs single file strategy.  Regardless of the answer to the parity question, enabling file splitting for a given backup job won't change the robustness scenario in either direction because even if you enable splitting and only one of your split files is corrupt, Reflect doesn't offer a way to perform partial data recovery from the remaining good pieces; all of the files that represent the backup need to be good.  Theoretically you could split your total backup requirements into multiple completely separate jobs, but that can become cumbersome, and if you're capturing images, you'll be limited in the "independence" you can achieve anyway since an image job has to target at least a single partition.  Splitting your source data into multiple independent jobs would be a more viable option in a File & Folder backup scenario.

I personally would recommend having at least 2 copies of your backups, and of course on separate physical media to protect yourself against destination media failure.  Even better than replicating your backup files to another location would be to have a destination rotation strategy where you create similar but unique backups to each location, because that sidesteps the risk of just making an identical copy of a corrupt backup.  So for example rather than running a backup on Monday night to Destination A and then copying that file to Destination B, keep the Monday night backup on Destination A, and then on Tuesday night create a completely separate backup directly to Destination B.  Alternatively, if you rotate your destinations on a weekly basis, you could take Destination A completely off-site during its "off" week to protect it against theft, natural disaster, etc.

Seekforever
Seekforever
Expert
Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)Expert (787 reputation)
Group: Awaiting Activation
Posts: 493, Visits: 7.6K
I'm not an expert regardless of what it says under my avatar. I have often posted that "one bad bit can render an image useless" and I have never had the statement challenged or corrected by anybody. Images are not written based on files but rather on disk clusters so a bad block in the image, often about 64K in size, that fails the checksum test could impact many files.  I don't recall seeing a discussion on Macrium adding some kind of error-correcting code to an image but some time ago did see this discussed on another popular imaging products forum. Adding such a code would significantly increase time and was not implemented.

You can Verify the image after it is written and this will confirm the image as created and stored is viable. However, this does not mean that it will be in future should something go wrong with the drive it is stored on. For example a later head crash could cause previously written areas to be damaged. This happened to me once  with a non-Macrium image and I had to go back 2 or 3 old images to get a good one. This is also a good reason not to go overboard on disk housekeeping - keep as many old images a practical on the drives.

Jphughan's last paragraph provides good advice on improving the integrity of you backup system. The primary rule is to diversify which means more than on device you write the backup on, avoid a common mode of failure such as copying a backup to a second device which may mean copying an error and storing the backups in different locations. Note that this error would likely have to be a Reflect data error; if the disk couldn't read the file properly it should give a CRC type error and abort the copy. To go to more extremes and avoid the common-mode failure of just using an external HD you would also write the backup on a different media such as DVDs or mag tape but that's a PITA.  If I copy a backup I always use a copy program that does a checksum compare or I immediately run Reflect's Verify to ensure the copy is good.

This issue is one of the reasons I prefer to backup my data files as individual files in their native format using another program rather than stuff them into a proprietary container file.​ Personal data files that are not available anywhere else are the most important items to be concerned about, you can always reload the OS and apps as the ultimate fall-back position if the images fail.

​​​
​​
Edited 3 September 2018 1:50 PM by Seekforever
john.p
john.p
Macrium Representative
Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)
Group: Administrators
Posts: 34, Visits: 177
In answer to the original post and in support of the other responses ...

> How robust are Reflect image files ?

Macrium backup files have very good error detection but contain no redundancy to enable error correction.

This is a pragmatic design decision both for Macrium backup files and most file systems (NTFS neither corrects or detects sector errors). The most effective place to detect and correct errors is as close as possible to the underlying storage medium. For example by storing a sector in a non-contiguous fashion, for examply by interleaving, on the disk surface, the effectiveness of error correction in the presence of a localised surface defect is maximised; this cannot be achieved off disk as the locality of data is abstracted away by the disk firmware.

We have carried out a design study on using RS coding to increase the robustness of Macrium files. However, it turns out that disk error correction is so effective that typically, once its error correction fails for a sector, it is close to general failure or a significant proportion of the disk becomes unreadable; in other words, when a disk starts to fail, there a very significant probability that the whole backup file or a large proportion will be lost. Further, as previously noted, the lack of storage locality outside the device severely reduces the effectiveness of any OS or application level error correction.

The most effective method to protect your backup files is replication on more than one device. Error detection is then vital to flag the requirement to swtich to the alternative backup.

> More specifically, how many bit errors can they suffer after creation before an image file is likely to be rendered useless ?

The vast majority of the file contains the data from the backed-up system. A bit error in that part of the file will only impact that part of the restored data. A bit error in the file meta-data can be more general however.

> Would not using compression at backup time reduce the risk?

The use of compression may actually reduce the risk as, by reducing the number of sectors used in the storage of the file, you are reducing the chance of any one of those sectors failing. The file is segmented such that a bit error only has an effect on a single unit of data, not the whole file. Further, though compression would increase the impact of a single bit error, due to the effect of disk firmware error correction, you never get a single bit error; either the errors are corrected or the whole sector (or sector group) is unreadable.

I hope this helps.

Kind regards,

Macrium Development.

Next Webinar


Alan
Alan
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 8, Visits: 32
Thank you both for your relies. I was replying to the first one a couple of days ago when my PC went AWOL on me again. Such things happen to me a lot, which is why I have lots of backups. However, retrieving stuff from a backup can sometimes be worse than trying to fix the problem. It's not always easy to make that call.

Anyway, it seems that the MR image files are not exactly bullet proof (though no worse than most container files, such as virtual drives). I had wondered if a non-compressed version might contain files in such a way that allows most of them to be extracted even from a corrupted container file, but it seems not. My intended solution is to use MR backups (drive images and folder & file backups) along with data copies that I make with Total Commander, and verify with ExactFile (using md5 checksums digests that I create just before making a new backup). There will be a mix of MR full and rolling ("forever") incremental backups that I make to a RAID 0 for the speed boost, plus ad-hoc copies that I'll make to other drives for historical backups. I prefer to get backups over and done with asap so that there is less chance of something going wrong mid-backup.

Cheers,
- Alan.

Alan
Alan
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 8, Visits: 32
Thank you too, John. You have put a different perspective on the hardware reliability vs non-hardware reliability.

You wrote
> The vast majority of the file contains the data from the backed-up system. A bit error in that part of the file will only impact that part of the restored data. A bit error in the file meta-data can be more general however.

Please confirm (or correct) my understanding of this:
a) A backup container file may suffer a damaged bit and there is no parity checking, etc., to detect or fix this error.
b) An unfixed single bit error in the backup container file is vastly more likely to affect just one of the backed up files within the container than the viability of the rest of the container file.
c) If such a damaged backup container file holds hundreds of thousands of backed up files, then chances are pretty good that all but one will be retrieved successfully by Reflect.
d) Losing lots of back-up files in a single backup container file is possible but relatively unlikely, so long as the storage device is not failing.
e) Using backup compression in Reflect compresses the backed-up files individually before containing them, rather than compresses the container file after containing all of the uncompressed backed-up files.

If my understanding is correct then I am relatively happy. I had been thinking that a single one-bit error would almost definitely trash the entire backup container file, especially if it had been compressed. So long as I verify any copies when I duplicate files, and look out for signs of failing hardware, the data should be ok .

- Alan


john.p
john.p
Macrium Representative
Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)Macrium Representative (97 reputation)
Group: Administrators
Posts: 34, Visits: 177
Hi,

> a) A backup container file may suffer a damaged bit and there is no parity checking, etc., to detect or fix this error.
Each block of data in the backup file is stored with a digest. Any number of bit errors will always be detected, but not fixed.

> b) An unfixed single bit error in the backup container file is vastly more likely to affect just one of the backed up files within the container than the viability of the rest of the container file.
True. Note that the smallest error on a modern disk is actually 8 sectors (or 4K).

> c) If such a damaged backup container file holds hundreds of thousands of backed up files, then chances are pretty good that all but one will be retrieved successfully by Reflect.
A single Macrium block typically contains 64K. This is the smallest quanta of data that can be lost. This is tiny compared with the size of a typical filesystem. Therefore, the proportion of data lost will correspondingly small. Please note that if that block contains filesystem meta-data or backup file meta-data, then the chances of partial retrieval are reduced.

> d) Losing lots of back-up files in a single backup container file is possible but relatively unlikely, so long as the storage device is not failing.
True. Non-failing disks see a large number of random bit errors due to the physics of high density storage. These will be corrected and it is very unlikely to see any non-corrected errors on non-failing disks. Note that there are other sources of backup corruption. Memory bit errors are another known cause; this is why server hardware often uses ECC memory modules.

> e) Using backup compression in Reflect compresses the backed-up files individually before containing them, rather than compresses the container file after containing all of the uncompressed backed-up files.
Mostly true. However, it is the blocks that are compressed individually. A block may contain multiple, or a part of a file, depending on the size of the file.

Again. I hope this helps.


Kind regards,

Macrium Development.

Next Webinar


Alan
Alan
New Member
New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)New Member (11 reputation)
Group: Forum Members
Posts: 8, Visits: 32
John, that helps me a lot.
Thank you very much - much appreciated.
- Alan

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)Macrium Evangelist (7.2K reputation)
Group: Forum Members
Posts: 4.9K, Visits: 36K
@john.p is there some way to tell Reflect to ignore/skip bad data discovered in a container file during a restore operation rather than halting the restore job with a failure message?  I've seen the option to ignore bad sectors when performing a backup, but I haven't seen anything like this for restores.  Or would data recovery from a damaged container file involve mounting it and manually extracting the desired content?

Edited 5 September 2018 4:21 PM by jphughan
Nick
Nick
Macrium Representative
Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)Macrium Representative (3.1K reputation)
Group: Administrators
Posts: 1.8K, Visits: 10K
jphughan - 5 September 2018 4:20 PM
@john.p is there some way to tell Reflect to ignore/skip bad data discovered in a container file during a restore operation rather than halting the restore job with a failure message?  I've seen the option to ignore bad sectors when performing a backup, but I haven't seen anything like this for restores.  Or would data recovery from a damaged container file involve mounting it and manually extracting the desired content?

We get very very few corrupt restores and we've had some where the problem was a noisy USB cable or hub. The verify before restore option will abort before restoring if verify fails.

We are going to add the option to skip bad blocks but add the moment it's necessary to mount the image. 

Kind Regards

Nick - Macrium Support

Next Webinar


GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search