Macrium Support Forum

New To Macrium (Home), need some clarification/advice

https://forum.macrium.com/Topic41727.aspx

By krashDH - 30 November 2020 9:43 PM

Hey all, 
So I've been using Macrium Free for a while and it's worked great.
Just as a bit of background on my Windows system FWIW. I have a 512 GB bootable that's my main drive (M.2 SSD) and is partitioned via windows default (I haven't edited it). I installed a 1TB M.2 SSD as my secondary drive. Both of these are on the motherboard.

I just recently bought another 1TB M.2 SSD with an external enclosure to use to store a lot of my old data from old computers and get rid of my dinosaur of an HDD that only had like 280 GB capacity or something . Dang thing sounded like a jet plane taking off every time I fire it up.
Anyway, with the free version, I have it set up to take a full disk image of my boot drive that includes my C: partition and other necessary items, once a month. I was doing differential images weekly. Nothing incremental as free didn't allow that, but in my case I don't think it's really necessary right now anyway, until I get into some of my more heavy projects.

With the purchase of this new drive, the idea was to keep my bases covered if either of my 2 drives installed in the laptop went belly up. So I thought that purchasing the Macrium Home edition would be a good idea, but now I'm not sure that I actually needed too.

Basically, I wanted to be able to copy the folder that houses my image backups (currently on my D: drive on the secondary 1TB SSD) onto the E: drive which is now my external SSD. As well, I'd like to copy other folders from my D: to my E: (Even though I'm talking in drive letters I guess think of it from internal secondary to external drive). I noticed that when I set up a temp schedule just for testing, just copying 4 folders (one being my large images folder that I make weekly) was taking a really long time. Like somewhere in the hour range.
I halted that operation and told that schedule only to copy the images folder to the external, knocked the time down to about 45 minutes. Now this is just for the initial "full" copy. I'm sure it will now go quicker in the future.

Here's another caveat. In the future, I may want to replace the current 1TB external SSD into the computer for the 512GB current SSD. So I want to keep everything structured to make that swap easy. I know it would require moving everything off the external drive somewhere else, cloning my bootable drive to the external, then installing it, and transfer all my stored data back to a different external drive. I also have a bootable USB so if I decide not to clone, or something dumps, I can restore my image to a new drive that way (as long as my secondary drive is still good, and external drive in this case, since they will both have copies of my bootable drive's partition images).

I'm trying to figure out the best structure to do this.
It would seem from an easy standpoint that I should just image the boot drive ---> secondary drive, then image the secondary drive---> external drive. Only downside is then I have to completely mount the secondary drive instead of picking through individual folders...this also may be an issue if I have other data on the external drive as it would want to wipe everything on the external. But this way would mean that I purchased the Home edition for no reason.

The other way, would be to do just a file backup for the image folder that I have on the secondary drive, onto the external drive, along with other folders. This seems really time consuming and larger? But it gives full control of folders. Which may be handy because I store a lot of coding projects on my secondary drive.

I'm really at a loss for the direction I should go with images vs folder backups, and what an appropriate schedule would be in each situation.
Any insight on this would be greatly appreciated.
Thanks in advance, sorry for the likely long post!
By jphughan - 30 November 2020 10:20 PM

Reflect doesn't copy folders.  The paid versions have a File & Folder backup function that can be used to back up folders, but in that case the backed up folders end up as MRBAK files and store point-in-time versions of those folders.  This is useful for people who want to perform and store periodic backups of data sets that are smaller than an entire partition.  But F&F backups do not result in copies of the source folders just sitting directly on the destination drive.  And if you were to use F&F backups for your scenario, at least some of them would cause you to store point-in-time image backups within point-in-time F&F backups, which doesn't make a ton of sense.  If you just want to copy folders over, just use Windows Explorer.  If you want to replicate the contents of folders on D to equivalent folders on E on a regular basis so that E gets updated as the data on D changes, then there are tools for that, but Reflect isn't one of them.  It doesn't do that sort of file-level data replication.

In terms of using the 1TB external SSD as a replacement for your current 512GB SSD, if the external SSD will ONLY ever store backup copies of your internal secondary 1TB SSD, as opposed to having data of its own that only exists on the external SSD, then your plan makes sense.  You'd clone the 512GB SSD to the external SSD, blowing away the current contents of the latter, and then you'd still have your data on your internal secondary 1TB SSD.  Whether you wanted to set up a new routine to have a backup copy of that would be up to you, but you'd of course need to find a storage device to store them.  Alternatively, if you ever did that, you could essentially invert the purposes of those SSDs.  Whereas right now the internal secondary 1TB SSD is your "primary" data set and you're backing up to the external SSD as a secondary, if you ever your "transplanted" your external SSD to be your new main SSD, then you could copy data from that internal secondary 1TB SSD onto it post-clone, then make that SSD your "primary".  Then you'd just configure image backups of that SSD to go to your internal secondary SSD.  Although backing up a 1TB SSD to another 1TB SSD will be a bit limiting if you ever start filling it up, because having equal capacities on your source and destination will limit the number of backups you can store.

I'm not really sure why you purchased the Home version based on your description, although now that you have, you may as well consider making daily Incrementals.  They're usually quick.  And if you ever have to restore a backup, the Rapid Delta Restore feature in the paid versions is pretty sweet.  There's also a Rapid Delta Clone feature that's useful if you want to clone a specific source to a specific destination on a regular basis to keep the latter current.  Technically you could do that, but I wouldn't necessarily recommend that in your case.  File replication is safer because with a clone, the destination gets destroyed each time the operation begins, and it doesn't become usable again unless it completes successfully.  This means that if you ever have a SOURCE disk failure in the middle of a clone operation, you'd be left with no usable source AND no usable destination.  By comparison, a disruption in the middle of a file copy doesn't ruin the entire destination partition.
By krashDH - 30 November 2020 10:55 PM

Thanks for the initial response!

Reflect doesn't copy folders. The paid versions have a File & Folder backup function that can be used to back up folders, but in that case the backed up folders end up as MRBAK files and store point-in-time versions of those folders.
Yup I totally get that part as I played with it a little bit and did a "mock" restore. It allows you to choose the folders and where you want to deploy them if I'm understanding correctly, even if it's on a different drive than where they originate.

If you just want to copy folders over, just use Windows Explorer. If you want to replicate the contents of folders on D to equivalent folders on E on a regular basis so that E gets updated as the data on D changes, then there are tools for that, but Reflect isn't one of them. It doesn't do that sort of file-level data replication.
Yes, this is partially what I want to do. I want it to be on a schedule. I have my regular image schedule that puts those files on my D. Yes, I'd like that folder at a minimum, to be updated on the E drive as my images change on my D drive. So I figured purchasing the Home edition that would give me access to backing up files and folders would work for that cause. I guess I'm wrong.

In terms of using the 1TB external SSD as a replacement for your current 512GB SSD, if the external SSD will ONLY ever store backup copies of your internal secondary 1TB SSD, as opposed to having data of its own that only exists on the external SSD, then your plan makes sense.
The external SSD currently is going to be used for everything. It's storing old data from other computers, as well as I want it to be a location to back up my windows images folder from my D: drive that Macrium creates (so I'm not relying on one location for my boot drive images), as well as backup any D: drive data or images (I'm not sure which yet I want to do, hence why asking on these forums the best plan of attack). But, that being said, one day, in the future, it could and probably a high chance that it will replace the 512 GB current main boot drive. Which would mean if I wanted to do that, I'd have to temp transfer everything to my secondary (D Smile drive (do a full image of E: at that point?) and then wipe my E: and get that ready to accept the clone from my current 512GB drive. I don't think I'd ever be swapping my primary and secondary drives around...my secondary currently is 1TB, my external is 1TB. So the current external would become the "new" boot drive.

I'm not really sure why you purchased the Home version based on your description, although now that you have, you may as well consider making daily Incrementals. They're usually quick. And if you ever have to restore a backup, the Rapid Delta Restore feature in the paid versions is pretty sweet.
I guess I purchased it in thinking that it could be an all inclusive package that would do what I needed or what I'm trying to explain, without messing with multiple "free" versions of programs. I was trying to keep everything clean and just with 1 program, and with the black Friday deal for Macrium Home it seemed to be a solid solution. Out of curiosity, what do the incrementals buy me except for a more refined points during the week. I think differentials are allowed daily correct, if so, what would incrementals be good for? 

Now that I have it, I'd like to figure out how to make it work for me. Free has been working great for imaging my boot drive and keeping me worry free if something goes haywire in my system, but I started thinking into upgrades, clones, folder backup, and identifying potential  system failure points that I could avoid. Maybe I'm doing it all wrong but I figured I could find some answers here.



By jphughan - 30 November 2020 11:37 PM

I wouldn't recommend making F&F backups of a source folder that contains MRIMG files.  It's just going to be a hassle because if you ever wanted to restore from one of those image backups, you'd have to first perform a restore of the MRBAK file just to extract the MRIMG file (or files, as the case may be), and THEN use the extracted MRIMG file to perform the image restore you actually wanted.

Regarding folder replication, paid Reflect does have an option to have it generate a PowerShell script to run your backup, and if you use that, the script creation wizard allows you to optionally enable a directory synchronization feature that will cause the backup destination folder to be replicated to some other location after each backup.  I use that myself. But that would only cover replicating the contents of the image backup folder on D over to E, not other data.  But maybe doing that plus making MRBAK backups of your OTHER data would be a winning combination for your use case.  It sort of depends on whether you want to retain multiple historical point-in-time snapshots of that other data or whether you just want the latest version replicated over, with no particular need to retain previous versions.  That's probably the most fundamental question to ask here.

In terms of your future swap plans, if the E drive will contain data from multiple other sources, then repurposing it as your boot drive would be more of a hassle.  But based on what you're describing, if E has a bunch of data that doesn't exist on D, AND you're already backing up most or all of D to E anyway, then from a logistical standpoint it would probably be easier to leave E as it is, repurpose D to become your boot drive (copying any data from D that did NOT already exist on E over there first), and then optionally install E internally if you don't want that SSD to be external anymore.  But I'm not sure since I don't know the precise details of what data you're putting where -- and I don't really need to.  You clearly understand the considerations involved here.  At the end of the day, you're going from 3 SSDs to 2 SSDs, so you'll have to either stop using one of your SSDs for its current use case entirely, or figure out some way to have one of the remaining SSDs do double duty.  Given the relatively low cost of 1TB SSDs even today, never mind later on whenever this might happen, it seems like it would almost be easier to just leave your 1TB SSDs as they are and simply replace your boot SSD with a new, larger SSD.  But I suspect this will be one of those cases where there are multiple paths to the same desired end result.

The benefit of Incrementals it that they're smaller and therefore faster to create compared to Differentials.  A Differential always has to contain all changes since the last Full backup.  An Incremental only has to contain changes since the last backup, whether that's a Full, Diff, or another Inc.  So if you're only making one Full backup per month and daily Differentials, the total size of your daily Diffs will be quite large because each Diff has to contain all changes made through the entire month up to the point of that Diff.  So your Day 1 changes will exist in all of the Diffs you create that month.  By comparison, if you run monthly Fulls and daily Incrementals, each Incremental only contains the changes made for that single day.
By krashDH - 1 December 2020 1:03 AM

I wouldn't recommend making F&F backups of a source folder that contains MRIMG files. It's just going to be a hassle because if you ever wanted to restore from one of those image backups, you'd have to first perform a restore of the MRBAK file just to extract the MRIMG file (or files, as the case may be), and THEN use the extracted MRIMG file to perform the image restore you actually wanted.
The only reason I was thinking of doing it this way is in case my D drive died on me. At least I'd have some way to access the images. Would be my second net. As long as the D drive is ok the images would be directly available to me.

In terms of your future swap plans, if the E drive will contain data from multiple other sources, then repurposing it as your boot drive would be more of a hassle.
Yes that's true since D is going to have less on it than E would. I was just thinking if I wanted to swap in my 1TB into the motherboard and remove the 512 GB. If the 512 is still good, I would likely clone my boot drive to my D drive. Then just R&R the 512GB with the 1TB (external) then transfer all of the data from my "old" external drive to my "new" 512 external. I'd have to remember to update any XML files and things like that though that Macrium uses.
But if my 512 died on me, I'd be booting to the USB and unpacking the image files to the D: drive, making it the new bootable, while waiting on a new 1TB SSD. My game plan is to always have/maintain 3. 2 will be internal, the 3rd would be the external drive. The only reason I would have 2 is if one died, and it would only be temp. But I need a way to ensure that I have all my data covered that is on all 3 drives.

The benefit of Incrementals it that they're smaller and therefore faster to create compared to Differentials.
So now that I have the paid version of Macrium and have access to incremental, what's a "best practice" of sorts?
Currently my fulls are once a month, and my diffs are once a week. I only keep 1 full and 2 diffs in the backup folder at any time.
Do I do 1 full, 1 diff, then incremental every few days after that? How much smaller does it making them? Also I'm assuming if you want to restore to an incremental, you need the diff and full images that it was derived from?
By jphughan - 1 December 2020 1:25 AM

Taking the image backups on your D drive and storing them on your E drive is a perfectly good idea in case D dies on you.  The point I'm trying to get across is that getting them onto E by performing a Reflect F&F backup is not a good idea. If you need to restore from an MRIMG file, you want that file to be available "directly", not stashed inside an MRBAK file that you'd have to open up first.  This is why I suggested using the Directory Synchronization function or some third-party file replication tool that would simply copy the MRIMG files over to E on a regular basis of some kind.  I realize you're trying to figure out how you can use Reflect for as much as possible here, but don't let that cause you to implement a solution that will become more of a chore overall.  But again, even the Directory Synchronization option would still be using Reflect for everything there.  You'd still have Reflect making image backups to your D drive, and then the Directory Synchronization function would sync the backups folder on D over to some other folder on E.

I'm going to drop the SSD swap discussion at this point.  It's a hypothetical, and it just doesn't seem worth continuing to write huge paragraphs back and forth to discuss.  Like I said earlier, you seem to grasp that there are various options each with their own pros and cons, and how you'd go about executing on them.

In terms of Incrementals, there's no single best practice.  A strategy of monthly Fulls, weekly Diffs, and daily Incs is somewhat common.  I personally run monthly Fulls and daily Incs without any Diffs.  Some people run monthly Fulls and daily Diffs with no Incs.  And the question of how many of each backup to retain is another question entirely.  For example, just because you make daily backups doesn't mean you have to retain them all, i.e. you don't have to delete backups in the order they were created.  The fact that you're only retaining 2 Diffs while creating 4-5 per month suggests that you already understand that.  But the question of how/whether to use Diffs and Incs depends on how long each type of backup takes, how large the backups are, and how much storage you have.  If neither time nor storage is a real concern, then maybe you can skip the Incrementals.  As you suspect, restoring from an Incremental does require any preceding Incrementals leading back to the parent Full (and the Diff in between if applicable) all be available, which technically increases risk somewhat.  By comparison, restoring a Diff only requires that Diff and its parent Full -- but as I said earlier, Diffs are larger than Incs because there's more "redundancy" in that strategy than with Incs.  But that redundancy is why that strategy is more resilient and less dependent on other backups.

If you're content only making weekly backups and only storing only a single monthly Full and 2 weekly Diffs, then there isn't much reason to consider switching to Incs.  There just aren't enough backups involved to make much of a difference.  But the size difference between using Incs and Diffs would depend entirely on how much your particular data expands or changes.

If you weren't already replicating your backups to another location though, I'd definitely be recommending that you retain more than one Full, perhaps by making a Full backup every 2 weeks rather than once per month.  The reason is that if you only have one Full and it turns out to be corrupt or partially unreadable, possibly due to an underlying hardware issue, then all of your backups become useless.  You at least have a second copy of your Full on another device, which is good, but even that wouldn't protect you against corruption that's endemic to the file itself as opposed to a problem with the device storing that file.  If your Full is corrupt, then a secondary copy is just a copy of corrupt data.
By krashDH - 1 December 2020 1:49 AM

The point I'm trying to get across is that getting them onto E by performing a Reflect F&F backup is not a good idea.
Sounds good, I'll look at other solutions for this

But again, even the Directory Synchronization option would still be using Reflect for everything there. You'd still have Reflect making image backups to your D drive, and then the Directory Synchronization function would sync the backups folder on D over to some other folder on E.
Ok I'll definitely be using directory synchro then. I'm sure Macrium has some good instructions on this, I'm not shy with powershell, cmd, or git if needed.

I'm going to drop the SSD swap discussion at this point. It's a hypothetical, and it just doesn't seem worth continuing to write huge paragraphs back and forth to discuss. Like I said earlier, you seem to grasp that there are various options each with their own pros and cons, and how you'd go about executing on them.
Sounds good, it's a bridge I can cross when I get there, just trying to set myself up for the most "painless" swap possible when the time comes!

I personally run monthly Fulls and daily Incs without any Diffs...As you suspect, restoring from an Incremental does require any preceding Incrementals leading back to the parent Full (and the Diff in between if applicable)...
How do you avoid having to have the required diff in the middle from the incremental--->full? I'm guessing there's something Macrium can do to avoid having to run a differential? Not saying I would, but it seems by your statement there is a shortcut. If there's documentation somewhere, feel free to link instead of explaining it all, I can definitely research.

If you weren't already replicating your backups to another location though, I'd definitely be recommending that you retain more than one Full, perhaps by making a Full backup every 2 weeks rather than once per month. The reason is that if you only have one Full and it turns out to be corrupt or partially unreadable, possibly due to an underlying hardware issue, then all of your backups become useless. You at least have a second copy of your Full on another device, which is good, but even that wouldn't protect you against corruption that's endemic to the file itself as opposed to a problem with the device storing that file. If your Full is corrupt, then a secondary copy is just a copy of corrupt data.
You are correct on this one. I'm only backing up my images to one location right now, the D drive. The thought process is there,  as I want second copy on my E drive, it's just apparent now my execution on "the best way to do that" was off.  I couldn't find a way to specify 2 locations for a backup. Would this be something that would be custom written with the Directory Sync? Or is it another "good idea" to do a "second" set of backups (different XML file) to the E drive (ones that aren't linked to the D drive ones, in case corruption...)?

I suppose if I do it the second way, writing the first "set" or XML file to the D, and the second "set" or XML to the E, is it even necessary to run a directory sync to backup that first set to the E drive? I suppose it can't hurt, but there is a lot of redundancy there. Also very valid point on the corruption chain. No amount of backups are going to help if the full backup is corrupt. I guess all of this is to just increase the probability of being able to restore if something happens.

I'll have to read more on this directory sync, but could I use this to basically backup D folders on the E drive without having to go get another software solution? Not that I'm opposed to the latter, just wondering what's possible. I've found the documentation to be somewhat helpful, but sometimes wading in the deep end I just end up going around in circles.

Thank you so much for all the info, this has definitely helped. I think I have a lot of tools now to ensure I'm covered for future issues!

By jphughan - 1 December 2020 2:01 AM

If you want to use Directory Synchronization, right-click your XML file, select "Create PowerShell script", and in the wizard popup, go down to Directory Synchronization and choose the replication path you want.  Then associate your backup schedules with the script (which you'll find under the PowerShell Scripts tab in Reflect) rather than the definition file.

If your Incrementals were built from a Diff, then you can't escape needing that Diff.  I only mentioned the Diff as a parenthetical because it's possible to create Incrementals as direct children of a Full, in which case obviously no Diff would be required in order to restore the Incrementals.  I myself run monthly Fulls with daily Incs, retaining 7 Incs and 2 Fulls.  So at any given time, I have daily backups going back a week, plus the Full from the beginning of the current month, and also the Full from the month before that.  No Diffs.

Reflect doesn't support backing up to multiple destinations in a single job.  It does have an "Alternative locations" feature where you can say, "If this destination isn't available, try these others," but that just causes Reflect to back up to the first available destination each time.  But you could certainly duplicate your XML file (right-click it and select Duplicate) and then change the destination on the duplicated version to be E.  Then set schedules for both of them.  In that case you would indeed end up with completely independent sets, which is better than a single replicated set.  And it's also better than just retaining 2 Full backups in a schedule, because in that case of course your older Full will be....older.  By comparison, if you run two backups each time, one to each destination, then if one set is bad, you'll have your other backup set available with backups from the same time period.  In that case you wouldn't use Directory Synchronization.  This setup is a bit more to manage than Directory Synchronization, so you'd just have to decide if that extra resilience is worth it to you.

The Directory Synchronization capability as implemented within Reflect is only intended to be used to replicate the backup destination folder to some other location after a backup completes.  Reflect won't help you use it to replicate other data outside of a Reflect job context.  However, since you're not shy with PowerShell, if you enable that option and examine the generated script, you'll see that Macrium is achieving that by simply calling Robocopy, which is a Microsoft utility that's been built into Windows for quite a while now.  And that absolutely CAN be used outside of Reflect.  You could easily set up some Robocopy commands with the desired source and destination paths and parameters to replicate data from D to E and then configure scheduled tasks to fire those commands, either individual scheduled tasks or a single task that called a batch file containing several such commands for whatever folders you wanted to replicate.
By krashDH - 1 December 2020 2:23 AM

If your Incrementals were built from a Diff, then you can't escape needing that Diff. I only mentioned the Diff as a parenthetical because it's possible to create Incrementals as direct children of a Full, in which case obviously no Diff would be required in order to restore the Incrementals. I myself run monthly Fulls with daily Incs, retaining 7 Incs and 2 Fulls. So at any given time, I have daily backups going back a week, plus the Full from the beginning of the current month, and also the Full from the month before that. No Diffs.
Does Macrium "automatically" build incremental from the diff if a full-->diff--->incremental schedule is made (grandfather, parent, child) or chosen from the pre-populated schedules? So if I were to make my backups like you have, would I just check the box for full and incremental, and set the schedules from each (making that the parent-child relationship)? Macrium then knows to cut out the middle man (diff) or is there another setting that has to be activated or changed?

It does have an "Alternative locations" feature where you can say, "If this destination isn't available, try these others," but that just causes Reflect to back up to the first available destination each time. But you could certainly duplicate your XML file (right-click it and select Duplicate) and then change the destination on the duplicated version to be E. Then set schedules for both of them. In that case you would indeed end up with completely independent sets, which is better than a single replicated set.
Yeah I did find that alternative locations option and read up on that but like you mentioned it clearly states that it will run through those until it finds the first "working" directory that has what it needs to create the image or backup. I'll probably go this route and duplicate the XML and just stagger the backup for 2 weeks. Increment full at beginning of month in one directory, then mid month in the second directory, with their corresponding children. It would be nice to have unrelated backups. If they all work, gives me more options to go backwards further when the famed "Windows Updates" push me something that proves to not work as well as the prior version.

The Directory Synchronization capability as implemented within Reflect is only intended to be used to replicate the backup destination folder to some other location after a backup completes. Reflect won't help you use it to replicate other data outside of a Reflect job context. However, since you're not shy with PowerShell, if you enable that option and examine the generated script, you'll see that Macrium is achieving that by simply calling Robocopy, which is a Microsoft utility that's been built into Windows for quite a while now. And that absolutely CAN be used outside of Reflect.
Good call on that, I could basically make my own local backups without needing any 3rd party software. Not that I oppose that at all but it seems like it could be done simply enough through powershell/cmd.

All really good stuff here, much appreciated. I originally went with Macrium after a lot of forum browsing and it seemed this product was at the top of the list for many users. I suppose I have a bit of a learning curve ahead of me but it seems to boil down pretty basic after it's all said and done. Although I probably didn't need the home edition, can't complain about more flexibility when it comes down to it. Plus, I don't mind supporting developers who give us the tools to make our lives easier. That's what the industry is about.
By jphughan - 1 December 2020 2:45 AM

Whenever Reflect creates an Incremental, it "appends" it to the most recent matching backup.  If there is no "matching" backup, then it will automatically create a Full even if you asked for an Incremental and note this and the reason for the deviation in the job log.  For image backups, an existing backup will be a "match" to the current backup if the existing backup contains EXACTLY the same partitions as those being backed up by the current job, including size and offset (position on disk).  So for example if you start a backup set and then decide to resize a partition, or add/remove a partition from your backup in your definition file, or rearrange your partitions on disk, you'll need to create a new Full -- which again Reflect will do for you automatically at the next backup if needed.  You wouldn't be able to add a new Diff or Inc to an existing backup created before that change.

The templates in the dropdown menu at the top of the schedule interface are only used for pre-populating certain schedule and retention policy configurations.  You can adapt those as needed after selecting one, or you can skip that entirely and build your own.  Those templates have absolutely no impact going forward beyond pre-configuring certain settings in that window.

If you don't want Diffs, then remove them from your schedule and don't create them manually either (you can manually run a backup by right-clicking the definition file and selecting Run Now).  But simply unchecking the Diff retention policy doesn't disable Diffs.  That disables the Diff retention policy, which would allow them to grow unconstrained, i.e. never getting purged (unless they were purged as a result of their parent Full being purged).  There are some use cases where users might not want to have retention policies defined on a particular "class" of backups and only have them get deleted as part of parent backup purges.  But if you want an Incremental to be created as a direct child of a Full, then the Full will need to be the newest matching backup when that Incremental backup runs, because again Reflect will always create an Inc from the newest matching backup, whatever type of backup that happens to be.

If it helps wrap your mind around how Reflect works, the XML definition file doesn't pre-define any relationships between future backups or maintain any sort of history of what backups were created when.  It's essentially just a settings file that says, "Whenever Reflect wants to run this backup, use these settings."  Each individual backup job is basically its own entity.  So for example if you run the definition file to create an Incremental, Reflect will look at what's at the destination and decide whether that's feasible based on whether a matching backup exists, and if so which backup is the latest matching backup.  It won't know or care if you deleted some of your previous backups "on the side".  And therefore it won't complain that some expected backups are missing (unless the missing backups are previous Incrementals in the chain that newer Incrementals depend on, e.g. you have Incrementals #1, #2, and #5).  This per-job independence can become quite powerful and freeing.  For example, I have a client that backs up to a rotation of 9 USB destination disks -- one for each of the 5 possible Mondays of the month and one each for the other 4 weekdays.  Only one disk is attached at any given time, and the disks are rotated daily.  That's not a problem for Reflect at all, in fact that entire setup is managed by one simple definition file containing one simple schedule of "Run an Incremental backup every day to Z:\ReflectBackups".  That's it.  And it works because when Reflect runs a backup, it doesn't freak out that the backup it wrote the previous night has disappeared and a completely different set exists at that destination now.  Instead, it just works with whatever happens to be at the destination every time you run a backup.

Happy to help, glad you've found it useful.  Reflect does have a bit of a learning curve for those who want to understand how it works and how to use it best for their purposes, but it's worth the time.  And this forum is a great resource.  It's the only somewhat large software company I know of where the active participants from the company are the actual engineers/developers of the application, not just customer support reps with minimal technical training.  And Macrium does respond quickly to bug reports and is fairly receptive to enhancement requests.  I've personally submitted several of both that I later saw get fixed/implemented.  I certainly agree about supporting good developers.  But in the meantime, another perk of paid Reflect is the option to use Image Guardian, which can be a nice anti-ransomware protection mechanism if you don't store your backups on a device that is kept physically offline most of the time.  Storage that's online is storage that's susceptible to malware/ransomware incidents -- the type you'd probably want to use a Reflect backup to recover from, but of course if your Reflect backups were rendered useless during the incident, you're stuck.  Image Guardian makes that somewhat less likely.
By krashDH - 1 December 2020 3:19 AM

jphughan - 1 December 2020 2:45 AM
Whenever Reflect creates an Incremental, it "appends" it to the most recent matching backup.  If there is no "matching" backup, then it will automatically create a Full even if you asked for an Incremental and note this and the reason for the deviation in the job log.  For image backups, an existing backup will be a "match" to the current backup if the existing backup contains EXACTLY the same partitions as those being backed up by the current job, including size and offset (position on disk).  So for example if you start a backup set and then decide to resize a partition, or add/remove a partition from your backup in your definition file, or rearrange your partitions on disk, you'll need to create a new Full -- which again Reflect will do for you automatically at the next backup if needed.  You wouldn't be able to add a new Diff or Inc to an existing backup created before that change.

The templates in the dropdown menu at the top of the schedule interface are only used for pre-populating certain schedule and retention policy configurations.  You can adapt those as needed after selecting one, or you can skip that entirely and build your own.  Those templates have absolutely no impact going forward beyond pre-configuring certain settings in that window.

If you don't want Diffs, then remove them from your schedule and don't create them manually either (you can manually run a backup by right-clicking the definition file and selecting Run Now).  But simply unchecking the Diff retention policy doesn't disable Diffs.  That disables the Diff retention policy, which would allow them to grow unconstrained, i.e. never getting purged (unless they were purged as a result of their parent Full being purged).  There are some use cases where users might not want to have retention policies defined on a particular "class" of backups and only have them get deleted as part of parent backup purges.  But if you want an Incremental to be created as a direct child of a Full, then the Full will need to be the newest matching backup when that Incremental backup runs, because again Reflect will always create an Inc from the newest matching backup, whatever type of backup that happens to be.

If it helps wrap your mind around how Reflect works, the XML definition file doesn't pre-define any relationships between future backups or maintain any sort of history of what backups were created when.  It's essentially just a settings file that says, "Whenever Reflect wants to run this backup, use these settings."  Each individual backup job is basically its own entity.  So for example if you run the definition file to create an Incremental, Reflect will look at what's at the destination and decide whether that's feasible based on whether a matching backup exists, and if so which backup is the latest matching backup.  It won't know or care if you deleted some of your previous backups "on the side".  And therefore it won't complain that some expected backups are missing (unless the missing backups are previous Incrementals in the chain that newer Incrementals depend on, e.g. you have Incrementals #1, #2, and #5).  This per-job independence can become quite powerful and freeing.  For example, I have a client that backs up to a rotation of 9 USB destination disks -- one for each of the 5 possible Mondays of the month and one each for the other 4 weekdays.  Only one disk is attached at any given time, and the disks are rotated daily.  That's not a problem for Reflect at all, in fact that entire setup is managed by one simple definition file containing one simple schedule of "Run an Incremental backup every day to Z:\ReflectBackups".  That's it.  And it works because when Reflect runs a backup, it doesn't freak out that the backup it wrote the previous night has disappeared and a completely different set exists at that destination now.  Instead, it just works with whatever happens to be at the destination every time you run a backup.

Happy to help, glad you've found it useful.  Reflect does have a bit of a learning curve for those who want to understand how it works and how to use it best for their purposes, but it's worth the time.  And this forum is a great resource.  It's the only somewhat large software company I know of where the active participants from the company are the actual engineers/developers of the application, not just customer support reps with minimal technical training.  And Macrium does respond quickly to bug reports and is fairly receptive to enhancement requests.  I've personally have submitted several of both that I later saw get fixed/implemented.  I certainly agree about supporting good developers.  But in the meantime, another perk of paid Reflect is the option to use Image Guardian, which can be a nice anti-ransomware protection mechanism if you don't store your backups on a device that is kept physically offline most of the time.  Storage that's online is storage that's susceptible to malware/ransomware incidents -- the type you'd probably want to use a Reflect backup to recover from, but of course if your Reflect backups were rendered useless during the incident, you're stuck.  Image Guardian makes that somewhat less likely.
Great info thanks.

I will consider how I want to do my next schedule for my E drive and if I want to revamp my schedule I have currently on my D drive.
I like the Full backup with the incremental approach idea. I think it can net what I need with a smaller footprint and still give plenty of flexibility for restore points. Also taking one potential failure point out of the equation is nice.

There aren't too many big changes that I really ever make on my system. Usually it's a couple of added or removed programs, altered files, location changes, etc. Not sure if it warrants a differential backup every week if I can cover the weeks in-between my full backups with an incremental (maybe 2 per week would work for me, and keep 4 so I could go back 2 weeks if an update was pushed out that proved it wasn't working smooth after it was deployed). Honestly from last month to this month, it's just been added programs/software, and file manipulation. I would think that if I was doing more heavy system modifications on a daily basis, a differential might be more ideal, but for me that's not so much the case.

I'll read up some more though on the details of each backup type. I can tell this community is very responsive just from the lurking I was doing prior to obtaining Macrium. I'm a part of a few vehicle forums, motorcycle forums, etc, and quite often do full tech write ups for those to help others. So it's much appreciated what communities like this provide.

FWIW, I'm currently a design engineer (and have been the last 8 years) but am taking classes and hoping to transition into programming/software dev/web dev. It's been an interest my entire life and finally jumping both feet first.
By jphughan - 1 December 2020 3:39 AM

Congrats on starting down a new path!  I've worked in various IT roles over the last 15 years or so, although I'm less interested in what I'm doing now, so I might be looking for a different role within that large umbrella before too long.  I've also been active on a few car forums (M3Post and Rennlist), although it's been a little while on each.  I mainly contribute here and on the Dell forums at the moment.  Good luck with your classes! Smile
By DanDanz - 1 December 2020 3:00 PM

@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   
By krashDH - 1 December 2020 3:36 PM

L. W. "Dan" Danz - 1 December 2020 3:00 PM
@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   

Thanks a lot for the support. It's pretty demanding right now putting in 10+ hours a day engineering then shifting to 4+ hours nightly for the classes. I think it will be worth it in the end. I've always been savvy with computers but no expert. Even just starting on this path has got me thinking about different things, and led me to here at this point. So it's all complimenting what I'm chasing. The area I'm living in is in a downward spiral. This is motivation that will help me re-invent my path. If I have time today I'm going to hopefully mess around with some backup schemes that I think will work with my situation.

Right now I'm thinking 1 monthly full per drive, staggered on 2 week intervals. Both of those will get bi-weekly incrementals, keeping 4 at one time in the storage folder (so 1 full + 4 increments). This will allow me to go back 2 weeks if necessary because of a unstable Windows Update. Thoughts?
By krashDH - 1 December 2020 7:31 PM

krashDH - 1 December 2020 3:36 PM
L. W. "Dan" Danz - 1 December 2020 3:00 PM
@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   

Thanks a lot for the support. It's pretty demanding right now putting in 10+ hours a day engineering then shifting to 4+ hours nightly for the classes. I think it will be worth it in the end. I've always been savvy with computers but no expert. Even just starting on this path has got me thinking about different things, and led me to here at this point. So it's all complimenting what I'm chasing. The area I'm living in is in a downward spiral. This is motivation that will help me re-invent my path. If I have time today I'm going to hopefully mess around with some backup schemes that I think will work with my situation.

Right now I'm thinking 1 monthly full per drive, staggered on 2 week intervals. Both of those will get bi-weekly incrementals, keeping 4 at one time in the storage folder (so 1 full + 4 increments). This will allow me to go back 2 weeks if necessary because of a unstable Windows Update. Thoughts?

After doing some more reading on this, it seems that if you are just doing full and incremental backups, this could cause an issue. From the Macrium information, if you want to restore to the latest incremental, then ALL of the incrementals preceding the latest one have to be present and functional. If one in the string is corrupt, then you're hosed, you need to go back to the full. So you are basically putting all your eggs in a basket and hoping that nothing gets corrupted.

I assume that goes for if you wanted to use the grandfather--father--son method. If you did a full-->diff-->multiple incrementals, everything in that line would have to be working to restore to the latest incr. If your incrementals got corrupted, you could still restore to your nearest diff update since you would still have a diff to use. This way, also, would give you more security I would think because you wouldn't have to go all the way back to the full to restore the image, whereas if you just took incrementals, if an early on one was corrupted, you're out of luck.

So I can see why now that these diff files are larger. They don't rely on each other for data...it only takes 1 diff image and 1 full image to restore a system. So 1 diff image a week, if you stored 2 of those, would give you the ability to restore to either of the last 2 weeks, since the image is being made off the full at that point in time.

Also makes sense incrementals are smaller since they only capture the changes from their previous image. Which would mean though, you'd need the entire chain to restore to the latest point. 
By jphughan - 1 December 2020 7:44 PM

The only correction I'd make to what you just posted is that if one of your Incrementals is corrupt, you don't necessarily have to go back to the Full.  You can go back to an Incremental that was created before the corrupt one.  But if the corrupt Incremental is the first one in the chain, then yes you'd have to resort to the Full.

But other than that, you've got the idea.  A group of Diffs that share a common Full will have some "redundant" copies of data between them, which is why they can be independent of each other and only dependent on the Full.  Incrementals don't have that redundancy, and therefore they rely on an unbroken chain leading back to the Full.  As in other data scenarios, redundancy increases resilience.

But the scenario you didn't explicitly address is the impact of a corrupt Full -- which would render all backups in the set useless.  And that's why I strongly suggest that users maintain at least 2 Fulls.
By krashDH - 1 December 2020 8:00 PM


The only correction I'd make to what you just posted is that if one of your Incrementals is corrupt, you don't necessarily have to go back to the Full. You can go back to an Incremental that was created before the corrupt one. But if the corrupt Incremental is the first one in the chain, then yes you'd have to resort to the Full.
 Correct, yes I didn't specifically state that in my post above but it's understood.

But the scenario you didn't explicitly address is the impact of a corrupt Full -- which would render all backups in the set useless. And that's why I strongly suggest that users maintain at least 2 Fulls.
This is also true I didn't state. If your full is all jammered up, then yeah you have nothing in that chain that will help you. In my case personally, I now have a second XML file that duplicates what the first is doing, except it alternates 2 week increments and puts that second set of images on my external drive.

The other thing I was wondering. Currently I have both XML files on my D drive. I know when I set up Macrium and the first XML file I changed the path of where I keep those XML files (same folder I keep the images in so they're easier to find). Now I can't seem to find where to change that filepath, where Macrium looks to run those.

I'm wondering, I should probably copy those XML files to my C drive in case something were to happen to my D drive, so it could still run backups. I know I could copy-paste duplicates, but is there a way to tell Macrium to go look there if they can't be "found" in the D drive, kind of like alternate specifications for the backup folder? This way if my D did dump, at least Macrium would go find the folders on my C and could at least execute backups to my external drive until I got a new D installed or figured out what happened. I guess I'm coming up short on finding where that setting is to specify the XML filepath.
By jphughan - 1 December 2020 8:20 PM

Reflect doesn't have a single XML file path; it doesn't require you to store all XML files in a specific folder.  When you step through the wizard each time, if you choose to save the job you just set up as an XML file and click the Browse button, the folder you see will just be the most last folder where you saved an XML file.

In general, the default location for XML files is worth keeping. I've seen people get hugely jammed up by storing their XML files on locations that weren't reachable when the backup was supposed to run.  They had configured email notifications for job failures -- but not successes -- and so they simply assumed that if they weren't getting any email, then their backups were proceeding as normal.  Well it turned out that the location of their XML file had become unavailable for some reason, which obviously prevented the backup from running -- but because email notification settings are in the XML file, which was inaccessible, Reflect didn't know to send an email notification about the failure.  They went quite a while before realizing their backups weren't running -- although at least they found out before they had a major incident.

You also absolutely do NOT want to store your XML files with your actual backups, because if you use encryption, then the password is stored in the XML file, in which case storing the XML file with the backups amounts to storing the key with the lock.  The password is itself encrypted in the XML file, but that's done with a static key, because Reflect installations on completely different PCs encrypt the same password the same way.  (Although I posted a Wish List thread proposing a new design for this that would improve security without breaking compatibility.)

There are some other risks to consider that may or may not be relevant to your use case but that are summarized in this best practices document from Macrium.

In terms of moving them, you can just copy them somewhere else and then import the new copies into Reflect using the "plus sign" icon in the Backup Definition Files tab.  However, there's no way to transfer schedules over.  You'll have to recreate those and then delete the previous XML file.  And yes I'd recommend copying and importing the new one first rather than removing the existing one from Reflect before copying/moving, because when you remove an XML file from Reflect, you have a choice of whether or not to delete the underlying XML file and scheduled tasks.  But in your case you'd want to keep the former but remove the latter (since those tasks would point to an incorrect XML file path).  There isn't a way to do that, so you'd either have to keep both and then manually clean up "orphaned" scheduled tasks or else delete both recreate your XML files from scratch.  It's easier to just copy, import new, set up new schedules, THEN delete the old XML files and their associated tasks.
By krashDH - 1 December 2020 8:38 PM

jphughan - 1 December 2020 8:20 PM
Reflect doesn't have a single XML file path; it doesn't require you to store all XML files in a specific folder.  When you step through the wizard each time, if you choose to save the job you just set up as an XML file and click the Browse button, the folder you see will just be the most last folder where you saved an XML file.

In general, the default location for XML files is worth keeping. I've seen people get hugely jammed up by storing their XML files on locations that weren't reachable when the backup was supposed to run.  They had configured email notifications for job failures -- but not successes -- and so they simply assumed that if they weren't getting any email, then their backups were proceeding as normal.  Well it turned out that the location of their XML file had become unavailable for some reason, which obviously prevented the backup from running -- but because email notification settings are in the XML file, which was inaccessible, Reflect didn't know to send an email notification about the failure.  They went quite a while before realizing their backups weren't running -- although at least they found out before they had a major incident.

You also absolutely do NOT want to store your XML files with your actual backups, because if you use encryption, then the password is stored in the XML file, in which case storing the XML file with the backups amounts to storing the key with the lock.  The password is itself encrypted in the XML file, but that's done with a static key, because Reflect installations on completely different PCs encrypt the same password the same way.  (Although I posted a Wish List thread proposing a new design for this that would improve security without breaking compatibility.)

There are some other risks to consider that may or may not be relevant to your use case but that are summarized in this best practices document from Macrium.

In terms of moving them, you can just copy them somewhere else and then import the new copies into Reflect using the "plus sign" icon in the Backup Definition Files tab.  However, there's no way to transfer schedules over.  You'll have to recreate those and then delete the previous XML file.  And yes I'd recommend copying and importing the new one first rather than removing the existing one from Reflect before copying/moving, because when you remove an XML file from Reflect, you have a choice of whether or not to delete the underlying XML file and scheduled tasks.  But in your case you'd want to keep the former but remove the latter (since those tasks would point to an incorrect XML file path).  There isn't a way to do that, so you'd either have to keep both and then manually clean up "orphaned" scheduled tasks or else delete both recreate your XML files from scratch.  It's easier to just copy, import new, set up new schedules, THEN delete the old XML files and their associated tasks.

OK I think now I'm officially lost. I've got a folder...windowsImageBackup, on my D drive. This is where my image files directly go. I have another folder within windowsImageBackup called ScheduleXML. Inside of that is where the XML files are. I originally made the D drive XML from scratch. Then copied that one and changed a few schedule parameters for my E drive backup. So both XML files are currently in that folder on my D drive, since I didn't step through the entire process to make the backup file for my E drive. So technically, they are in the XML and image files are in the same main directory, windowsImageBackup, but the XML files are in a separate folder within there.

Should I have them somewhere else? Should I store the XML file that I use for my external drive ON my external drive? Backup both of these folders somewhere on my C drive so if I need to import them if my D drive dies? I guess I missed the boat on your last post, I think that's above my pay grade
By jphughan - 1 December 2020 8:40 PM

The default location for XML files is a folder called "Reflect" inside your user profile's Documents folder.  Unless you have a specific reason for storing them elsewhere, I'd store them there.
By krashDH - 1 December 2020 8:41 PM

jphughan - 1 December 2020 8:40 PM
The default location for XML files is a folder called "Reflect" inside your user profile's Documents folder.  Unless you have a specific reason for storing them elsewhere, I'd store them there.

So I would need to copy-paste them back into that folder, then re-import them to macrium
By jphughan - 1 December 2020 8:45 PM

Correct.  Then set up your schedules on the newly imported copies, then delete the originals from Reflect, choosing to also delete the underlying XML file and scheduled tasks associated with the old copies
By krashDH - 1 December 2020 9:09 PM

jphughan - 1 December 2020 8:45 PM
Correct.  Then set up your schedules on the newly imported copies, then delete the originals from Reflect, choosing to also delete the underlying XML file and scheduled tasks associated with the old copies

Ok, moved to C drive, re-imported to Macrium, applied the schedule templates I had made earlier for each, deleted the original D XML files from Macrium backup def files, and deleted XML file folder that was on the D drive. Now the XML files reside in the Reflect folder on C drive. I think that covers it then
By jphughan - 1 December 2020 9:31 PM

As long as you don't have any orphaned scheduled tasks (you can check the Scheduled Backups tab for any items that might be under an old path for an XML file), then it sounds like you're all set. Smile