By capair45 - 14 September 2023 10:48 AM
I have a scheduled, nightly, full File & Folder backup of an internal data disk. Nothing else is stored on this disk except data. The destination is an external USB HD. Twice in the last few weeks the backup has failed automatic verification. The earlier backup failed with an error code 23. Last night's backup failed with error code 121. After the 1st failure, I ran a variety of Disk tools to check the health of the disk. All tests passed (chkdsk, Western Digital disk diagnostics, and Hard Disk Sentinel surface test). I have included the log from last night's failed verification.
EDIT: This external drive also stores a variety of disk images that are created nightly. None of them fail verification; only this F&F backup.
Backup ID - EA3453BF5148A6FA
--------------------------------------------------------------------------------
Backup Summary
--------------------------------------------------------------------------------
Backup Definition File: C:\Users\David\Documents\Reflect\7 Western Digital F&F (Disk 1-W) Daily.xml
Auto Verify: Y
Maximum File Size: Automatic
Compression: Medium
Password: N
NTFS Permissions: Y
Folder Reparse Points: Follow user reparse points, Do not follow system reparse points
Backup set matching: Similar - Match on backups with at least one matching folder
Shutdown: Forced
Email On Success: Y
Recipients: xxxx
Email On Warning: Y
Recipients: xxxx
Email On Failure: Y
Recipients: xxxx
Current Time: 9/14/2023 4:31:24 AM
--------------------------------------------------------------------------------
Destination
Backup Type: Full
File Name: W:\Reflect F&F (D) Daily\EA3453BF5148A6FA-00-00.mrbak
--------------------------------------------------------------------------------
Backup Sources
Folder: D:\Documents\
Include File Filter: *.*
Exclude File Filter: *.temp; *.tmp; *.bak; *.~*
Include Sub Folders: Y
Exclude Folder Filter:
Exclude Hidden Files: N
Exclude System Files: N
--------------------------------------------------------------------------------
Creating Volume Snapshot - Please Wait
Volume Snapshots Created
D:\ \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3
Starting Backup - Thursday, September 14, 2023 4:31 AM
Write Method: Using Direct Disk I/O
Destination Drive: My Book (W - Free Space 1.11 TB
Image Guardian: Protected
Free space threshold: Delete oldest backup sets when free space is less than 5.00 GB
Determining files to copy
Total Number of Files: 38715
Total Number of Folders: 3561
Total Size: 80.64 GB
Copying files
Saving cluster information
Saving Index
New File: 78.93 GB EA3453BF5148A6FA-00-00.mrbak
Verifying 'EA3453BF5148A6FA-00-00.mrbak'
Failed
Gathering Windows Events - Please Wait
--------------------------------------------------------------------------------
Email Notifications
Recipients: xxxx
Email notification sent
--------------------------------------------------------------------------------
Backup aborted! - 'EA3453BF5148A6FA-00-00.mrbak' Verification failed at 16% (Error reading 262153 bytes from file. Error code = 121)
--------------------------------------------------------------------------------
Macrium Reflect Home v8.1.7638 - 76-xxxx
|
By capair45 - 14 September 2023 5:18 PM
Examining the Backup Log Summary, I noticed that both failed backups show "Error reading 262153 bytes from file" in the summaries. Not sure what that means but thought I'd throw it out there.
I might add that I also do a nightly image (mrimg.) of the same disk with automatic verification and I have never seen a failure of the image. Only with a backup (mrbak).
|
By JoeA - 15 September 2023 9:06 AM
The error code 121 may be referencing the semaphore timeout period has expired, this means that the backup target file system has stopped communicating with Windows and caused all open file handles to drop. This doesn't necessarily mean the issue lies on the disk but possibly the USB wire or the port itself. It could also be a one-off error that Windows likes to throw every now and then.
If you are using an external drive as the destination disk can you please try and connect this directly to the motherboard via the USB ports and see if this changes anything?
|
By capair45 - 15 September 2023 11:14 AM
Thanks for replying @JoeA!
Last night's verification was successful.
The external USB storage device is connected directly to a USB port on the computer. I'll replace the cable and keep an eye on it for awhile. If another failure happens, I'll connect the new cable to a different port.
In any event, I'll keep you in the loop.
|
By capair45 - 17 September 2023 7:46 PM
@JoeA I replaced the USB cable today so we'll see how the F&F backups go this week.
|
By capair45 - 18 September 2023 11:00 AM
Last night's scheduled F&F backup failed verification (log attached). I will attach the new USB cable to a different 3.0 port and see if that makes a difference. I find it odd that nightly images (mrimg) are not failing. It's only File & Folder backups that are failing verification.
Backup Summary
--------------------------------------------------------------------------------
Backup Definition File: C:\Users\David\Documents\Reflect\7 Western Digital F&F (Disk 1-W) Daily.xml
Auto Verify: Y
Maximum File Size: Automatic
Compression: Medium
Password: N
NTFS Permissions: Y
Folder Reparse Points: Follow user reparse points, Do not follow system reparse points
Backup set matching: Similar - Match on backups with at least one matching folder
Shutdown: Forced
Email On Success: Y
Recipients: XXXX
Email On Warning: Y
Recipients: XXXX
Email On Failure: Y
Recipients: XXXX
Current Time: 9/18/2023 4:31:12 AM
--------------------------------------------------------------------------------
Destination
Backup Type: Full
File Name: W:\Reflect F&F (D) Daily\BD19A1149B76C85D-00-00.mrbak
--------------------------------------------------------------------------------
Backup Sources
Folder: D:\Documents\
Include File Filter: *.*
Exclude File Filter: *.temp; *.tmp; *.bak; *.~*
Include Sub Folders: Y
Exclude Folder Filter:
Exclude Hidden Files: N
Exclude System Files: N
--------------------------------------------------------------------------------
Creating Volume Snapshot - Please Wait
Volume Snapshots Created
D:\ \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12
Starting Backup - Monday, September 18, 2023 4:31 AM
Write Method: Using Direct Disk I/O
Destination Drive: My Book (W - Free Space 996.04 GB
Image Guardian: Protected
Free space threshold: Delete oldest backup sets when free space is less than 5.00 GB
Determining files to copy
Total Number of Files: 38715
Total Number of Folders: 3561
Total Size: 80.66 GB
Copying files
Saving cluster information
Saving Index
New File: 78.95 GB BD19A1149B76C85D-00-00.mrbak
Verifying 'BD19A1149B76C85D-00-00.mrbak'
Failed
Gathering Windows Events - Please Wait
--------------------------------------------------------------------------------
Email Notifications
Recipients: XXXX
Email notification sent
--------------------------------------------------------------------------------
Backup aborted! - 'BD19A1149B76C85D-00-00.mrbak' Verification failed at 64% (Error reading 262153 bytes from file. Error code = 23)
--------------------------------------------------------------------------------
Macrium Reflect Home v8.1.7638 - 76-XXXX
|
By JoeA - 18 September 2023 11:19 AM
Hi @capair45,
Can you please open a support ticket so we can collect a support submission from you?
|
By capair45 - 18 September 2023 11:32 AM
#248062 Submitted
|
By capair45 - 18 September 2023 12:28 PM
MR Support staff is researching this. In the meantime, can anyone help me understand the significance of, "Error reading 262153 bytes from file"? This is shown in all 3 of the failures I've seen so far.
Additionally, a manual F&F backup with verification just completed successfully.
|
By dbminter - 18 September 2023 1:25 PM
I do notice that on both failures, you're getting different error codes. The first log's error code was 121 and the 2nd's is 23.
|
By capair45 - 18 September 2023 3:08 PM
Yes, that's correct as noted in my 1st post. Error code 23 can indicate HD issues although several utilities I've run find no issues. 121 seems to be a timeout issue. Odd that verification failures are only happening with F&F backups. Images written to the same storage device verify successfully.
|
By dbminter - 18 September 2023 3:23 PM
The only difference between a disk image and a F&F backup is, as far as I know, that the first backs up used sectors and the 2nd backs up actual files. So, it would seem the low level access has no issues, but the high level file access does. But, that wouldn't explain a Verification error. That would seem to be a target device issue.
I also find it a bit odd the error code changed.
A few things to check on:
1.) Did you try a manual Verify of this failed backup? Does it fail manual Verifies as well?
2.) Did you try performing a manual Verify from the Rescue Media and see if it fails there, too?
|
By capair45 - 18 September 2023 4:01 PM
Agreed on all counts! I'll post back to your questions shortly.
|
By pokeefe - 18 September 2023 4:02 PM
+xMR Support staff is researching this. In the meantime, can anyone help me understand the significance of, " Error reading 262153 bytes from file"? This is shown in all 3 of the failures I've seen so far. Whenever I see an unusual decimal number like that I convert it to hexadecimal: 0x40009. Looks to me like the misinterpretation of some flag bytes as a length. Feeble guess, and absolutely no help in dealing with the problem.
|
By dbminter - 18 September 2023 4:11 PM
My guess is manual Verifies in Windows would fail as well. But, I would be interested to see how the manual Verify behaves in Rescue Media with all the normal Windows processes, etc. removed from the equation.
|
By capair45 - 18 September 2023 4:27 PM
+xThe only difference between a disk image and a F&F backup is, as far as I know, that the first backs up used sectors and the 2nd backs up actual files. So, it would seem the low level access has no issues, but the high level file access does. But, that wouldn't explain a Verification error. That would seem to be a target device issue. I also find it a bit odd the error code changed. A few things to check on: 1.) Did you try a manual Verify of this failed backup? Does it fail manual Verifies as well? 2.) Did you try performing a manual Verify from the Rescue Media and see if it fails there, too?
Here's an oddity to add to the puzzle.
1 & 2. The failed backup of the 14th (121) and the 18th (23) do not show in the Existing Backups tab nor are they on the storage device so I cannot verify them manually within Reflect or with rescue media. The failure on 9/5 (23) has scrolled off. I only retain these backups for 7 days.
If a backup fails verification, does Reflect discard it? It would seem so but I've never encountered this before.
EDIT: I ran a manual full F&F backup of the scheduled backup that failed this morning (18th). The manual backup completed and verified successfully. This backup shows in Existing Backups tab as well as on the storage device.
|
By capair45 - 18 September 2023 4:29 PM
+x+xMR Support staff is researching this. In the meantime, can anyone help me understand the significance of, " Error reading 262153 bytes from file"? This is shown in all 3 of the failures I've seen so far. Whenever I see an unusual decimal number like that I convert it to hexadecimal: 0x40009. Looks to me like the misinterpretation of some flag bytes as a length. Feeble guess, and absolutely no help in dealing with the problem.
Hopefully, the Support staff can sort it out. I have plenty of images of this disk on several devices so no big worries.
|
By dbminter - 18 September 2023 4:34 PM
The only time I would think Reflect would delete a backup that fails Verification is if the backup set used file splitting. Each split part is Verified upon write completion. So, if a Verify fails at that stage, the backup aborts. Since it never completed a full backup, there would be no reason to retain a failed backup if the backup used file splitting.
You're not using file splitting though, right?
However, it is a good question to ask support. I've never had a Verify fail during backup before. I've had them fail manual Verifies after a backup completed, but never during one. And since I always use file splitting, my backups would be deleted upon Verification failure cancelling backup.
|
By capair45 - 18 September 2023 4:35 PM
I do not use file splitting.
|
By capair45 - 18 September 2023 4:36 PM
I'll be away from the computer for a few hours but will check back later.
|
By capair45 - 19 September 2023 11:12 AM
Another verification failure this morning with error code 23. No mrbak file was written to the storage (repository) device or shows under Existing Backup tab. I have submitted the requested information to the Support staff.
I am running a manual backup and verification of the disk now. I expect they will complete successfully.
|
By dbminter - 19 September 2023 1:15 PM
How large is this f&f data set you're backing up? I was thinking if it was easy enough, you might copy the XML to a dummy copy for editing and remove, say, half the data set. Then try again. See if you get the same issue. I would be working trying to isolate whatever factor might be triggering it. Like a particular file/folder that may be being backed up. Or a potential backup file size issue. Plus, if you say worked down the backup file size to something small enough for sending that still returns the error, you could send that data set to tech support, if you're inclined/don't mind them seeing the contents.
|
By capair45 - 19 September 2023 1:21 PM
The manual backup I just ran failed verification at 4% (error code 121).
I think I'll edit the definition file to exclude automatic verification and run it again. If the backup is successful, I'll attempt to verify it manually.
|
By capair45 - 19 September 2023 1:26 PM
+xHow large is this f&f data set you're backing up? I was thinking if it was easy enough, you might copy the XML to a dummy copy for editing and remove, say, half the data set. Then try again. See if you get the same issue. I would be working trying to isolate whatever factor might be triggering it. Like a particular file/folder that may be being backed up. Or a potential backup file size issue. Plus, if you say worked down the backup file size to something small enough for sending that still returns the error, you could send that data set to tech support, if you're inclined/don't mind them seeing the contents.
Data size is about 80GB
|
By dbminter - 19 September 2023 1:56 PM
Are the failure verification points roughly the same? Always at 4% or at others?
I've backed up well more than 80 GB before in F&F backups. In fact, my current monthly backup of changed files in temporary locations exceeds 300 GB right now for the year. But, that backup set uses file splitting so I can't say if it's a potential F&F file size issue.
|
By capair45 - 19 September 2023 3:58 PM
I edited the F&F definition file to exclude auto verify. I ran the backup manually and it completed successfully. Afterwards, I manually verified the mrbak. file. It verified correctly although it dis hesitate a bit at the 4% point). This file is shown in Existing Backups tab as well as on my storage (repository) device.
Just to clarify, I am having no issues with image backups (mrimg.) using the same USB port, cable, and storage device as the F&F backups. All images are verifying correctly. The only issue is with F&F verification.
|
By capair45 - 19 September 2023 4:16 PM
+xAre the failure verification points roughly the same? Always at 4% or at others? I've backed up well more than 80 GB before in F&F backups. In fact, my current monthly backup of changed files in temporary locations exceeds 300 GB right now for the year. But, that backup set uses file splitting so I can't say if it's a potential F&F file size issue.
2 failures at 4%. 1 failure 16%, 1 failure 64%
|
By dbminter - 19 September 2023 4:30 PM
Interesting progression rates there. 2 4%'s then 16%, which is 4 squared, and then 64% which is 4 x 16.
Your further testing seems to indicate the issue is during the auto verification phase. If you exclude auto verify, the backup completes successfully. If you manually Verify that completed backup, the manual Verify completes.
At this point, it's probably best left up to the support people to look into.
|
By capair45 - 19 September 2023 4:47 PM
+xInteresting progression rates there. 2 4%'s then 16%, which is 4 squared, and then 64% which is 4 x 16. Your further testing seems to indicate the issue is during the auto verification phase. If you exclude auto verify, the backup completes successfully. If you manually Verify that completed backup, the manual Verify completes. At this point, it's probably best left up to the support people to look into.
Agreed. I'll post back when I hear something
|
By capair45 - 19 September 2023 9:40 PM
I believe the storage disk is failing. I ran some other diagnostic tests and there were several read errors as well as semaphore timeouts. I've disconnected the disk and have attached another external USB HD. Backups, images, and verifications are completing successfully to this new destination.
Thanks to all who helped me with this.
|
By dbminter - 19 September 2023 10:06 PM
And you said you already tried a different USB cable, right? And a different USB port? If so, yeah, I'd be inclined to place the blame on the device itself. However, it doesn't explain why manual Verifies worked, why Verifies only failed on backups, and why image backup Verifies didn't fail. Still you can't argue with results. If connecting a different device doesn't have issues, you can pretty much blame the other device.
|
By capair45 - 26 September 2023 8:41 PM
The storage device was failing. Several disk utilities started to indicate that. Replacing the storage device has returned backups and verification to normal.
Macrium was very responsive and passed along the following possible solution. I had already replaced the storage device when I received their reply so was not able to give it a try. I'm including it for those who may want to see what they said.
"After looking at the logs and the system you are running, we believe this could be an issue with multithreading.
With the new backup engine, we also introduced a new verification engine which also utilizes multithreading. When the verification process takes place we believe the CPU can not handle the load and is causing the verification process to fail.
Can you please enable the legacy version of the software by following the instructions below and see if this allows you to verify the file after the backup has been taken?
Open the registry editor and create a new folder called verification under HKEY_LOCAL_MACHINE\SOFTWARE\Macrium\reflect and add the registry key below.
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Macrium\reflect\Verification
Key: Legacy
Type: DWORD
Value: 1
Once this has been done can you please perform another backup or wait for the next scheduled backup and report back to me? "
|