Cloning Multiple Dupes


Author
Message
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
I helped someone else set up this routine of cloning multiple source disks onto a single target.  The easiest method is to manually partition the target disk as needed to accommodate all of the intended source partitions.  Just create the necessary number of partitions of the required size and leave them empty.  Then open Reflect.  Select the first disk that will be one of the sources and click "Clone this disk".  Then in that wizard, drag each source partition you want to clone onto the desired empty destination partition. Do NOT use the "Copy selected partitions" function, since that will cause the source partitions to be copied to the destination in a way that the destination partitions will match the size and "offset" (position on disk) of the source partitions. That is not what you want here.  By comparison, the drag and drop method tells Reflect, "Clone this source partition into this existing destination partition, even if the destination partition is a different size and/or is located in a different portion of the target disk."  Then click Finish and choose to save this clone job as an XML file.  Finally, repeat this process for each of the other disks that need to be a source disk for this clone setup.  Obviously make sure you don't end up with different clone jobs targeting the same partition on the destination disk.

tgwilkinson
tgwilkinson
New Member
New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)
Group: Forum Members
Posts: 11, Visits: 62
Ah ha! Drag and drop! I see where it says those words at the top of the dialogue box now, but I had skipped over it because it's not obvious it would have a different effect than the Copy Selected Partitions method. Good catch! I have my definition files setup, but I ran into two additional problems when I ran the clone (I haven't had this many problems setting up a piece of software in years! so I appreciate your help).



The first is that I set up the first definition file to clone the partitions on one drive to two pre-formatted partitions on my backup destination drive (as shown in the first picture). The backup ran for two days, which seemed slow, but Macrium is still new to me and perhaps that's normal? After two days, when the clone completed, my destination drive had lost one of the partitions it had just backed up. The partition X: Video 1 was gone (I unfortunately forgot to take a screenshot with the missing partition), while the other partition had successfully cloned and was still in the same section of the disk.

So I set up the X: partition again and ran the same clone definition again. I expected it to go slightly faster this time because there was already a copy of the Music 1 data on the backup destination and I have rapid delta clone turned on, so the Music 1 portion of the clone operation should have flown by and most of the cloning time should have gone to the Video 1 partition clone—but the backup took another two days. At the end of cloning, I had another missing partition, but this time it was reversed: The U: Music 1 partition was missing and it was the X: Video 1 partition that had stuck around this time instead of disappearing (as shown in the second picture).

In contrast, the previous clone operation from the SAME definition file went S: T9 SSD, empty partition, U: Music 1, and then an empty partition in place of the X: Video 1.

So that's the first big mystery. The partitions are the same size as their source, which is what you recommended as necessary (which makes sense!), so I don't know why this is happening. Hopefully this problem isn't too hard to solve.

The bigger issue is that even testing out a clone job right now is PAINFUL because of how long it takes. I opened the Task Manager during the second attempt at cloning and discovered that there's a bottleneck somewhere, but I don't know how to troubleshoot it. The fastest transfer rate I'm seeing when using Macrium caps out at 29.6 MB/s, and sometimes dipped as low as 0.6 MB/s sustained transfer speed (see the third screenshot below), even though I'm transferring from a USB 3.0 drive to another USB 3.0 drive using the original WD USB 3.0 cables, and both are plugged directly into the motherboard's 3.0 USB ports and NOT into a hub (though I do have hubs attached for other peripherals). Here are four sample screenshots from over the course of the two days that show how low the transfer rate held at:

1. 

2.

3.

4.

That fourth screenshot especially hurts because I'm getting faster backup to the cloud with Code42's CrashPlan (36.4 MB/s) than I am from a local disk transfer (1.5 MB/s)!

The other thing I noticed is that in every screenshot except the fourth one Macrium's disk transfer speed and the System disk transfer speed are almost identical. I don't know if that's how Macrium is set up to work, or if the bottleneck is filtering down to other applications as well. In the last twelve hours of the second clone job the taskbar, Google Chrome, and the Task Manager became unresponsive. However, during that same time I was still able to click play on a game in Steam (I already had a Steam window open when everything else became unresponsive) and I was able to play the newest game in 1440p with ray tracing and everything without a hitch despite the I/O errors. The game is on the M.2 internal SSD, so that probably plays a role, but it's interesting none the less in trying to diagnose this.

The slow transfer rate also doesn't make sense because for most other applications the drive works fine. I ran a couple CrystalDiskMark tests and I'm getting the 110-150 MB/s data transfer rates I'm expecting.

Source:


Destination:


So the drives seem to be connected correctly, but something is wrong somewhere.

The only other time I have trouble like this with I/O stuff is using Adobe Lightroom to edit photos. The program will work relatively fine for the first half of a workday, but then a few hours into rating, labelling, and editing images (all of which have tiny read/write operations to a database file, as well as the initial read of the larger, original image file) all those changes add up somewhere in the system and the program will bog down—every operation then takes 1000x longer than it should, especially when I'm pulling metadata from multiple image files at once—until the program either becomes unresponsive or shifts into a "redraw the screen" state where the Lightroom window will toggle between a partial full screen mode and true full screen mode on infinite loop, even though I didn't trigger a change in full screen modes, with no way to stop it except a hard restart.

It seems like another example of a bottleneck in my system that eventually grinds everything to a halt and often makes the taskbar and File Explorer unresponsive similar to what I'm seeing happen with Macrium. I only bring it up because perhaps the problems are linked, perhaps not. But I though maybe one other example of a time I see similar system behavior might help in diagnosing the problem.

I'm at a real loss here for how to troubleshoot this, so any insight or help you can offer would be hugely appreciated. Let me know if sharing the specs of my system or some other piece of information would be helpful.

Thanks again,
Tom

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Hey Tom,

Well that's a rather mixed bag of findings!  I'll address what I can here.

First, looking at the first screenshot in your post above, it does appear that you've staged the clone perfectly correctly for what you're trying to do.  So you're saying that when you actually proceeded to run that clone you had staged, exactly as shown in that screenshot, the job cloned two partitions but only one of them actually existed on the destination when all was said and done?  And then the second time around you ran the same job and the OPPOSITE partition disappeared?  Unfortunately I can't account for what happened there at all.  That might be worth contacting Macrium Support about, because I can't imagine why it would even be POSSIBLE to stage a clone that would result in 2 partitions being cloned but one of them not actually existing at the end.  My GUESS as to what MAY be happening here is that somehow the partitions are being staged with some sector overlap, in which case whichever partition gets cloned second ends up surviving because it got laid down such that it partially encroached on the sector space of the adjacent partition, thereby rendering that other partition effectively gone.  And if so, this may be some bug that only gets triggered when dealing with large disks or something.  I don't have disks of the size you're working with to test with.  And again, that's just a guess.  But if you want to look into it, check the logs of the two clone jobs you ran to see the sequence of cloning for each one.  Given that a different partition survived on your two different attempts, do the logs indicate that the partitions were cloned in a different sequence on each of those two attempts?  If so, was the partition that ended up surviving after each attempt the first partition to be cloned, or the second?

As to your transfer rate, I've never cloned anything close to 4.76 TB before, but using 48 hours based on your note that it took 2 days, that would suggest an average transfer rate of just 27.54 MB/s.  That's roughly USB 2.0 speeds.  I can think of two possibilities right off the bat:
  • Third-party anti-virus can sometimes interfere with Reflect's operations, severely degrading its performance.  Are you running any such AV, other than Windows Defender?
  • HDDs that use SMR (shingled magnetic recording) can have very poor performance under workloads that involve sustained write activity, as would be the case for the target disk of a clone operation.  Does your target disk use SMR?

tgwilkinson
tgwilkinson
New Member
New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)
Group: Forum Members
Posts: 11, Visits: 62
Good news! I got half of my backups working.

I'm using Western Digital drives and only their 2-6 TB drives seem to use SMR, and I'm using 8 TB and 14 TB drives. I also checked the Macrium logs like you suggested and discovered that, while my USB 3.0 to USB 3.0 backups were going super slow, my internal Samsung SSD to USB 3.0 backup was working correctly at blazing fast speeds. So the kind of destination drive I'm using doesn't appear to be the source of the issue.

For what it's worth, I'm also not running any third-party anti-virus software. The closest I get to that is Code42's CrashPlan cloud backup service which runs constantly in the background logging which files have been changed so it can back those up to the cloud. But now that I've gotten a few of the backups working, I've tested running Macrium clones with CrashPlan running as well as paused and it doesn't seem to make an appreciable difference since the fastest transfer rates I'm seeing with CrashPlan cap out at 30 MB/s due to my cable internet speeds. I could see potentially pausing the CrashPlan backups for an hour or two when Macrium is set to run, but I'm not sure it'll make a difference in the long run.

So onto the solution! After reading an old post on this forum about Logitech wireless mouse dongles causing slow downs in Macrium backups, and another article related to my problems with slowness/crashing in Adobe Lightroom that suggested keeping my mouse and tablet drivers up to date, I decided to investigate the state of my mouse.

I made a few changes, and I'm not sure which of them solved the problem, so I'll list them all. First I tried unplugging the wireless dongle, like suggested in the linked Macrium forum thread above, but that did nothing for my transfer speeds. Next I tried to open Logitech's G Hub application (which manages the wireless mouse settings like DPI and button assignment) to check out my drivers and the program hung on the opening screen. I'd seen this behavior before, where the program hangs indefinitely, but didn't pay much attention to it in the past because I keep all my mouse settings loaded onto my mouse's internal memory. So I did two things next: first I repaired/reinstalled G Hub to get the most up-to-date mouse drivers installed and working on my system, and then, since I have my mouse in onboard memory mode and don't actually need G Hub managing my mouse settings in real time, I went into the G Hub settings and unchecked the box for "always start after logging in." And between those things, the next time I ran a USB 3.0 to USB 3.0 backup it worked perfectly. Transfer rates were 150-200 MB/s for the entire backup process, and the transfers completed in a few hours.

I also set up all new backup definitions and, rather than creating partitions in advance (using MiniTool Partition Wizard), I dragged and dropped the source drives onto unallocated space and let Macrium make the partitions so there were no overlapping sectors (because I think you're right, the sectors were overlapping, and the order in which the backups happened determined which drive stuck around). I also separated out the Music 1 and Video 1 clones into two separate clone definitions rather than doing them back to back in the same definition in case that helped in some way.

I share all that (a) in case it helps someone else in the future and (b) because I still have two drives that backup at painful 30 MB/s or less speeds (again sometimes as low as 0.1 MB/s!). But at least this problem has been narrowed down to be a more specific problem Wink

So here's what I know about the backups that don't work and maybe this rings some bells for you: My source drive is an OWC Thunderbay 8—it has eight drive bays (though I'm only currently using four) and it connects to my PC using a Thunderbolt 3 cable into a Gigabye Z390 Designare motherboard which supports Thunderbolt (one of the few PC motherboards that do). The Thunderbay currently has four drives in it, all 4TB HGST UltraStar 7K4000 (HUS724040ALE640). The drives operate at 7200 RPM, use 4096-byte sectors, and have a 64MB drive cache (they're a few years old!). The drives are joined together into a Dynamic Disk using RAID 0. I created the Dynamic Disk/RAID using standard Windows Disk Management because I'm only a year into a being a full-time PC user and wasn't sure what else to use (Is there a better piece of software for this? Should I have set up anything special in my BIOS I might have missed?). I can read and write from the drives fine in everyday use, and I'm getting 400 MB/s read/write speeds on them in CrystalDiskMark even now that the drives are 50% full, so as sources they should be fine, but when it comes to Macrium they're the only disks I can't backup at this point without running into super slow transfer speeds again.

So with that setup it feels like the problem is something related to either the RAID/Dynamic Disk or it's an issue with Thunderbolt since I've got other backups working fine now...and this is where my knowledge runs out. So any suggestions on things to try would be hugely appreciated!
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Glad to hear you've made some progress!  As I was reading through your post and saw your findings that internal to USB 3 worked as expected but USB 3 to USB 3 didn't, my next theory was going to be a bad or just poorly designed USB host controller chipset that just crumpled when heavy I/O was occurring on both reads and writes simultaneously.  But I'm glad to hear that you found a fix, although I never would have expected a Logitech receiver/driver to wreak that sort of havoc!  That rather makes me worry about what sorts of things Logitech is hooking into, since that side effect really shouldn't be possible.  The dongle itself though can be an issue from a HARDWARE standpoint because USB 3.0 traffic and 2.4 GHz radio frequencies do not play well together.  That's why some routers that have USB 3.0 ports have those ports located in places that are unintuitive or downright inconvenient, such as on the front edge or on the sides near the front, rather than on the back where everything else is -- because placing the USB 3.0 port in that location minimizes interference with the WiFi antennas.  My own ASUS RT-AC88U router has its USB 3.0 port on the front, and if I had anything plugged into it, having that cable sticking out would rankle even my minimal aesthetic sensibilities, but fortunately I don't use it.  Still, I remember some reviewers criticizing this router's USB 3.0 port placement because they didn't realize that it was a concession to function over form.

Great to hear that your partition issue is resolved too.

In terms of your Thunderbolt storage array, I'm not a fan of dynamic disks or RAID 0.  Dynamic disks can introduce a variety of hassles, especially if you ever need to restore them (link), which I realize you're not doing.  But I mostly don't like them because I tend to avoid software-based RAID setups in favor of hardware-based RAID setups -- although the needle is swinging back to software in this era of software-defined everything (storage, networking, firewalls, etc.)  Windows has more recently introduced Storage Spaces, which I believe is supposed to be a replacement for dynamic disks that have been around since at least the Windows XP days, but I admit I haven't looked into Storage Spaces a great deal.  You may wish to, however.  But I'm REALLY not a fan of RAID 0 in any form for the simple reason that the failure of any disk causes the entire virtual disk to be lost, which means your risk of data loss is multiplied with every disk that you add, i.e. your risk of losing all of your data in a 4-disk RAID 0 is 4x greater than your risk of losing all of your data when you only have a single drive.  Especially if you have drive bays available in your enclosure, I would really encourage you to consider adding disks so that you can switch to a RAID level that adds redundancy, such as RAID 6 or 10, while still maintaining the amount of usable storage you require.  In a 4-disk setup, both RAID 6 and 10 would give you the same amount of usable storage.  But RAID 6 offers more "guaranteed" redundancy in that ANY two disks can fail and the array will still be fine, whereas with RAID 10, certain combinations of two disks can fail while keeping the array intact, but you're only guaranteed to be able to survive a single disk failure.  But RAID 6 suffers on write speeds due to the required double parity calculations.  In a 6-disk setup, everything I just said still applies, but the additional consideration is that RAID 6 will give you more usable storage because with RAID 6, the usable storage is the total number of disks minus 2, whereas with RAID 10 the usable storage is 50% of all disks.

All that said, I can't immediately account for why you'd be able to get solid read speeds from the array in CrystalDiskMark but have poor performance when making a backup of that array in Reflect.  But what is the destination of that backup job?  And if thus far you've only benchmarked the storage array (the source) and the destination separately, then you haven't looked at the whole picture yet.  In order to get a point of comparison to the Reflect scenario, you'd need to test performance when copying a large file from the storage array to your Reflect backup destination in Windows.  What sorts of transfer rates do you see for that test?

tgwilkinson
tgwilkinson
New Member
New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)
Group: Forum Members
Posts: 11, Visits: 62
Ha! Funny. We're using the same router. The good news is my wireless dongles (for the mouse and Wacom tablet) are plugged into my monitor rather than my motherboard so they don't cause as much interference. And I have a 10' extension cable for the computer's WiFi antennas, so they're nowhere near the back of the case.

I copied and pasted a 10GB folder of medium sized image files (2k files total) and it copies at about 140-170 MB/s. I also tried a copy and paste of a 100GB folder of tiny cache files (the folder contained over 250k files) and it kept a 45-60 MB/s transfer speed. Both of those are much faster than Macrium could get maxed out as it was at < 30 MB/s. The transfer rate of the first copy and paste job was likely limited by the source drive being a single 3.0 USB drive. So the slow transfer rate seems to be limited to Macrium (and maybe other programs I'm unaware of yet).

I tried to follow your advice about Dynamic Disks to see if that fixed the problem. I backed up all the files on the raid enclosure and deleted the dynamic disk. But when I went to replace it with a Storage Space I got the following error:



I've updated to the most recent OS and Thunderbolt drivers. I'm using the Thunderbolt 3 cable that came with the enclosure. All of the disks are showing up in Disk Management and I can read/write to them. So I don't know what "check the drive connections" could mean in this context. I'm at a loss and stuck again.

As far as RAID goes, I've been running RAID 0 for six years without a serious issue. There's a reason I keep three backup copies! And it helps that I'm using enterprise hard drives rated for two million hours MTBF. Sure drives have failed, but they have a five-year warranty and get RMAed quickly, and I'm back up and running in no time. I wouldn't keep any fewer than my current three backup copies operating even if I were running a form of RAID with built in redundancy, so moving to another form of RAID feels like spending unnecessary money for a fourth backup copy I don't necessarily need. But I appreciate the explanation of RAID 6, which I was less familiar with. RAID 6 sounds appealing, but I'd need to read more about the performance hit to decide if it's worth it. At this point the biggest issue with changing anything with my RAID setup is that the drives are so old I could only get refurbished drives to match, and I wouldn't trust those. So I'd have to lay out over a thousand dollars put in 6-8 new drives to get RAID 6 or 10 working, which is out of the question for this year at least.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Ok, so you’ve validated the data path outside of Reflect and it’s performing within expectations. Macrium has said here that Reflect uses standard Windows APIs for reading and writing backup files, which are also commonly used by other applications. So my guess is that something else on your system doesn’t like Reflect itself. Normally that’s third-party anti-virus, and the fix is to whitelist Reflect’s various processes as well as excluding backup destination folders from scans. But if that’s not it, this might be a case for Macrium Support.

As for Storage Spaces, as I said I haven’t looked into it in detail, but if you haven’t already, I’d Google that hex error code. I suspect there may be some hardware requirement that isn’t being met here. Maybe Thunderbolt/PCIe storage isn’t allowed, for example?
tgwilkinson
tgwilkinson
New Member
New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)
Group: Forum Members
Posts: 11, Visits: 62
Where should I start with the whitelisting process? That's something I'm unfamiliar with.

And when you say to exclude backup destination folders from scans, which type of scans do you mean? And are those scans in Reflect or in Windows?

Searching the error code gave me back nothing useful, unfortunately, so I reached out to the enclosure manufacturer (OWC) to see if they could diagnose it.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Whitelisting and folder exclusions would be accomplished in the interface of any third-party anti-malware solution you may be running, and the exact steps would vary from one solution to the next. But if you’re only running Windows Defender, then you shouldn’t have to do that.

I kind of doubt OWC will be able to help with that issue, but can’t hurt to ask. You might be better off looking for Microsoft TechNet documentation about Storage Spaces, in particular its hardware and software requirements.
tgwilkinson
tgwilkinson
New Member
New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)New Member (12 reputation)
Group: Forum Members
Posts: 11, Visits: 62
All right. It's fixed. Wow was that stupid. I'll document what I did here in case it's useful to someone down the road:

I occasionally heard some dings indicating that USB devices were randomly disconnecting and reconnecting. So I installed Nirsoft's USBLogView to track what was happening. It turns out the USB ports/hub built into the back of my NEC MultiSync PA271W Monitor don't want to play nice with Windows. Even when I have nothing plugged into them the ports/hub still randomly unplugs and plugs itself back in. The monitor is seven and a half years old, so it is what it is.

I had two wireless dongles plugged into those ports: one for my Wacom Tablet and another for my Logitech G502 Lightspeed mouse. I unplugged both of those and, inspired by this forum post, I moved the Logitech wireless dongle off the USB 3 bus and, on the back of my PC, plugged it into the dedicated USB 2.0 mouse port. And le voilà Macrium is copying at full speed on both by USB and Thunderbolt 3 RAID enclosure now. I'm not going to test it any further because it's working, but either moving the wireless dongle off of the monitor port OR moving it to the USB 2.0 bus fixed the problem. And not only is Macrium getting better I/O speeds, but the entire computer feels more responsive. Huzzah!

So back to the original question of working with multiple clones. How do I go about setting up a script to rename drives (with dedicated letters thanks to USBDLM) after a clone operation has completed? I can't believe it took a month to get back to this Hehe
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search