Verification Taking a Long Time


Author
Message
hurricane51
hurricane51
New Member
New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)
Group: Forum Members
Posts: 15, Visits: 18
Ver 7.1.2697.

Verification on my backups is taking around 1-1/2 times longer than the actual backup. Backup transfer rate averages 950 Mbps, whereas verify rate is 650 Mbps. CPU is 4 GHz i7 with 16GB RAM, and plenty of free drive space. I can't imagine why reading should take longer than writing the files.

Anybody have an idea?
Nick
Nick
Macrium Representative
Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)
Group: Administrators
Posts: 1.6K, Visits: 8.8K
hurricane51 - 2 December 2017 12:28 PM
Ver 7.1.2697.

Verification on my backups is taking around 1-1/2 times longer than the actual backup. Backup transfer rate averages 950 Mbps, whereas verify rate is 650 Mbps. CPU is 4 GHz i7 with 16GB RAM, and plenty of free drive space. I can't imagine why reading should take longer than writing the files.

Anybody have an idea?

Thanks for posting.

Verification time is at the mercy of the read speed of the backup drive. It's possible that when creating an image the read speed of the source drive is significantly faster than the write of the backup target. This means that both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete. In this case, verification could be slower than creating the image.  With that said, there are further optimisations that can be done to improve the efficiency of verification and we have a backlog case open to look at improving performance. 

Kind Regards

Nick - Macrium Support

hurricane51
hurricane51
New Member
New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)
Group: Forum Members
Posts: 15, Visits: 18
Nick - 3 December 2017 4:33 PM
hurricane51 - 2 December 2017 12:28 PM
Ver 7.1.2697.

Verification on my backups is taking around 1-1/2 times longer than the actual backup. Backup transfer rate averages 950 Mbps, whereas verify rate is 650 Mbps. CPU is 4 GHz i7 with 16GB RAM, and plenty of free drive space. I can't imagine why reading should take longer than writing the files.

Anybody have an idea?

Thanks for posting.

Verification time is at the mercy of the read speed of the backup drive. It's possible that when creating an image the read speed of the source drive is significantly faster than the write of the backup target. This means that both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete. In this case, verification could be slower than creating the image.  With that said, there are further optimisations that can be done to improve the efficiency of verification and we have a backlog case open to look at improving performance. 

The target drives ARE slower than the source drives. The target drives are Seagate 8TB Archive drives, designed (as you might tell) for storage and not daily use. The source drives are typical Seagate and WD desktop drives.

"...both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete."

I get that, but it would affect the inital backup write speed. Then you say:

" In this case, verification could be slower than creating the image."

Why? I don't see the connection between slow writes and even slower verification. How can the write speed of the traget drive have any effect on the verification process? The read times are typical of desktop drives, so I don't see where the bottleneck occurs. Also, I don't find this to be a problem with the various (seems like hundreds) of backup solutions I've used before. In fact, the opposite was usually true -- the verification was much faster than the original write. That's why it was such a surprise.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)
Group: Forum Members
Posts: 3.6K, Visits: 27K
hurricane51 - 4 December 2017 1:15 AM

The target drives ARE slower than the source drives. The target drives are Seagate 8TB Archive drives, designed (as you might tell) for storage and not daily use. The source drives are typical Seagate and WD desktop drives.

"...both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete."

I get that, but it would affect the inital backup write speed. Then you say:

" In this case, verification could be slower than creating the image."

Why? I don't see the connection between slow writes and even slower verification. How can the write speed of the traget drive have any effect on the verification process? The read times are typical of desktop drives, so I don't see where the bottleneck occurs. Also, I don't find this to be a problem with the various (seems like hundreds) of backup solutions I've used before. In fact, the opposite was usually true -- the verification was much faster than the original write. That's why it was such a surprise.

During image creation involving hardware where the source read speed is greater than the target write speed, the source's higher performance means that a chunk of data could be read from the source and hashed before the target has finished writing it out. When performing a verification afterward, Reflect has to wait for the target to actually read the data back before it can start hashing it.  Given enough of a performance disparity and/or enough CPU performance, the impact of the hashing on the time required to complete the overall operation could be zero in the former case, but it will always be non-zero in the latter.

Edited 4 December 2017 1:45 AM by jphughan
hurricane51
hurricane51
New Member
New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)
Group: Forum Members
Posts: 15, Visits: 18
jphughan - 4 December 2017 1:30 AM
hurricane51 - 4 December 2017 1:15 AM

The target drives ARE slower than the source drives. The target drives are Seagate 8TB Archive drives, designed (as you might tell) for storage and not daily use. The source drives are typical Seagate and WD desktop drives.

"...both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete."

I get that, but it would affect the inital backup write speed. Then you say:

" In this case, verification could be slower than creating the image."

Why? I don't see the connection between slow writes and even slower verification. How can the write speed of the traget drive have any effect on the verification process? The read times are typical of desktop drives, so I don't see where the bottleneck occurs. Also, I don't find this to be a problem with the various (seems like hundreds) of backup solutions I've used before. In fact, the opposite was usually true -- the verification was much faster than the original write. That's why it was such a surprise.

During image creation involving hardware where the source read speed is greater than the target write speed, the source's higher performance means that a chunk of data could be read from the source and hashed before the target has finished writing it out. When performing a verification afterward, Reflect has to wait for the target to actually read the data back before it can start hashing it.  Given enough of a performance disparity and/or enough CPU performance, the impact of the hashing on the time required to complete the overall operation could be zero in the former case, but it will always be non-zero in the latter.

Got it. And you said that you were investigating some ways to mitigate this?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)Macrium Evangelist (5.3K reputation)
Group: Forum Members
Posts: 3.6K, Visits: 27K
Well Macrium said they were looking at optimizations; I'm just another user. Smile  (Macrium reps have the Reflect logo as their user icon.)

Edited 4 December 2017 2:16 AM by jphughan
hurricane51
hurricane51
New Member
New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)New Member (19 reputation)
Group: Forum Members
Posts: 15, Visits: 18
jphughan - 4 December 2017 2:15 AM
Well Macrium said they were looking at optimizations; I'm just another user. Smile  (Macrium reps have the Reflect logo as their user icon.)

I thank you as one user to another.
Nick
Nick
Macrium Representative
Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)Macrium Representative (2.8K reputation)
Group: Administrators
Posts: 1.6K, Visits: 8.8K
hurricane51 - 4 December 2017 2:14 AM
jphughan - 4 December 2017 1:30 AM
hurricane51 - 4 December 2017 1:15 AM

The target drives ARE slower than the source drives. The target drives are Seagate 8TB Archive drives, designed (as you might tell) for storage and not daily use. The source drives are typical Seagate and WD desktop drives.

"...both the source read and MD5 digest calculation can be executed while waiting for the image file write to complete."

I get that, but it would affect the inital backup write speed. Then you say:

" In this case, verification could be slower than creating the image."

Why? I don't see the connection between slow writes and even slower verification. How can the write speed of the traget drive have any effect on the verification process? The read times are typical of desktop drives, so I don't see where the bottleneck occurs. Also, I don't find this to be a problem with the various (seems like hundreds) of backup solutions I've used before. In fact, the opposite was usually true -- the verification was much faster than the original write. That's why it was such a surprise.

During image creation involving hardware where the source read speed is greater than the target write speed, the source's higher performance means that a chunk of data could be read from the source and hashed before the target has finished writing it out. When performing a verification afterward, Reflect has to wait for the target to actually read the data back before it can start hashing it.  Given enough of a performance disparity and/or enough CPU performance, the impact of the hashing on the time required to complete the overall operation could be zero in the former case, but it will always be non-zero in the latter.

Got it. And you said that you were investigating some ways to mitigate this?

We are looking at optimising the decompression and hashing by using multiple cores to shave some time off the process, however, this still won't necessarily make verification faster than backup if the backup drive speed is the bottleneck in the backup process. 

Kind Regards

Nick - Macrium Support

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search