New To Macrium (Home), need some clarification/advice


Author
Message
krashDH
krashDH
New Member
New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)
Group: Forum Members
Posts: 12, Visits: 28
jphughan - 1 December 2020 2:45 AM
Whenever Reflect creates an Incremental, it "appends" it to the most recent matching backup.  If there is no "matching" backup, then it will automatically create a Full even if you asked for an Incremental and note this and the reason for the deviation in the job log.  For image backups, an existing backup will be a "match" to the current backup if the existing backup contains EXACTLY the same partitions as those being backed up by the current job, including size and offset (position on disk).  So for example if you start a backup set and then decide to resize a partition, or add/remove a partition from your backup in your definition file, or rearrange your partitions on disk, you'll need to create a new Full -- which again Reflect will do for you automatically at the next backup if needed.  You wouldn't be able to add a new Diff or Inc to an existing backup created before that change.

The templates in the dropdown menu at the top of the schedule interface are only used for pre-populating certain schedule and retention policy configurations.  You can adapt those as needed after selecting one, or you can skip that entirely and build your own.  Those templates have absolutely no impact going forward beyond pre-configuring certain settings in that window.

If you don't want Diffs, then remove them from your schedule and don't create them manually either (you can manually run a backup by right-clicking the definition file and selecting Run Now).  But simply unchecking the Diff retention policy doesn't disable Diffs.  That disables the Diff retention policy, which would allow them to grow unconstrained, i.e. never getting purged (unless they were purged as a result of their parent Full being purged).  There are some use cases where users might not want to have retention policies defined on a particular "class" of backups and only have them get deleted as part of parent backup purges.  But if you want an Incremental to be created as a direct child of a Full, then the Full will need to be the newest matching backup when that Incremental backup runs, because again Reflect will always create an Inc from the newest matching backup, whatever type of backup that happens to be.

If it helps wrap your mind around how Reflect works, the XML definition file doesn't pre-define any relationships between future backups or maintain any sort of history of what backups were created when.  It's essentially just a settings file that says, "Whenever Reflect wants to run this backup, use these settings."  Each individual backup job is basically its own entity.  So for example if you run the definition file to create an Incremental, Reflect will look at what's at the destination and decide whether that's feasible based on whether a matching backup exists, and if so which backup is the latest matching backup.  It won't know or care if you deleted some of your previous backups "on the side".  And therefore it won't complain that some expected backups are missing (unless the missing backups are previous Incrementals in the chain that newer Incrementals depend on, e.g. you have Incrementals #1, #2, and #5).  This per-job independence can become quite powerful and freeing.  For example, I have a client that backs up to a rotation of 9 USB destination disks -- one for each of the 5 possible Mondays of the month and one each for the other 4 weekdays.  Only one disk is attached at any given time, and the disks are rotated daily.  That's not a problem for Reflect at all, in fact that entire setup is managed by one simple definition file containing one simple schedule of "Run an Incremental backup every day to Z:\ReflectBackups".  That's it.  And it works because when Reflect runs a backup, it doesn't freak out that the backup it wrote the previous night has disappeared and a completely different set exists at that destination now.  Instead, it just works with whatever happens to be at the destination every time you run a backup.

Happy to help, glad you've found it useful.  Reflect does have a bit of a learning curve for those who want to understand how it works and how to use it best for their purposes, but it's worth the time.  And this forum is a great resource.  It's the only somewhat large software company I know of where the active participants from the company are the actual engineers/developers of the application, not just customer support reps with minimal technical training.  And Macrium does respond quickly to bug reports and is fairly receptive to enhancement requests.  I've personally have submitted several of both that I later saw get fixed/implemented.  I certainly agree about supporting good developers.  But in the meantime, another perk of paid Reflect is the option to use Image Guardian, which can be a nice anti-ransomware protection mechanism if you don't store your backups on a device that is kept physically offline most of the time.  Storage that's online is storage that's susceptible to malware/ransomware incidents -- the type you'd probably want to use a Reflect backup to recover from, but of course if your Reflect backups were rendered useless during the incident, you're stuck.  Image Guardian makes that somewhat less likely.
Great info thanks.

I will consider how I want to do my next schedule for my E drive and if I want to revamp my schedule I have currently on my D drive.
I like the Full backup with the incremental approach idea. I think it can net what I need with a smaller footprint and still give plenty of flexibility for restore points. Also taking one potential failure point out of the equation is nice.

There aren't too many big changes that I really ever make on my system. Usually it's a couple of added or removed programs, altered files, location changes, etc. Not sure if it warrants a differential backup every week if I can cover the weeks in-between my full backups with an incremental (maybe 2 per week would work for me, and keep 4 so I could go back 2 weeks if an update was pushed out that proved it wasn't working smooth after it was deployed). Honestly from last month to this month, it's just been added programs/software, and file manipulation. I would think that if I was doing more heavy system modifications on a daily basis, a differential might be more ideal, but for me that's not so much the case.

I'll read up some more though on the details of each backup type. I can tell this community is very responsive just from the lurking I was doing prior to obtaining Macrium. I'm a part of a few vehicle forums, motorcycle forums, etc, and quite often do full tech write ups for those to help others. So it's much appreciated what communities like this provide.

FWIW, I'm currently a design engineer (and have been the last 8 years) but am taking classes and hoping to transition into programming/software dev/web dev. It's been an interest my entire life and finally jumping both feet first.

jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Congrats on starting down a new path!  I've worked in various IT roles over the last 15 years or so, although I'm less interested in what I'm doing now, so I might be looking for a different role within that large umbrella before too long.  I've also been active on a few car forums (M3Post and Rennlist), although it's been a little while on each.  I mainly contribute here and on the Dell forums at the moment.  Good luck with your classes! Smile

L. W. "Dan" Danz
L. W. "Dan" Danz
Advanced Member
Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)Advanced Member (401 reputation)
Group: Forum Members
Posts: 199, Visits: 3.3K
@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   

  L. W. "Dan" Danz, Overland Park KS  

krashDH
krashDH
New Member
New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)
Group: Forum Members
Posts: 12, Visits: 28
L. W. "Dan" Danz - 1 December 2020 3:00 PM
@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   

Thanks a lot for the support. It's pretty demanding right now putting in 10+ hours a day engineering then shifting to 4+ hours nightly for the classes. I think it will be worth it in the end. I've always been savvy with computers but no expert. Even just starting on this path has got me thinking about different things, and led me to here at this point. So it's all complimenting what I'm chasing. The area I'm living in is in a downward spiral. This is motivation that will help me re-invent my path. If I have time today I'm going to hopefully mess around with some backup schemes that I think will work with my situation.

Right now I'm thinking 1 monthly full per drive, staggered on 2 week intervals. Both of those will get bi-weekly incrementals, keeping 4 at one time in the storage folder (so 1 full + 4 increments). This will allow me to go back 2 weeks if necessary because of a unstable Windows Update. Thoughts?
krashDH
krashDH
New Member
New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)
Group: Forum Members
Posts: 12, Visits: 28
krashDH - 1 December 2020 3:36 PM
L. W. "Dan" Danz - 1 December 2020 3:00 PM
@krashDH:  I too would encourage you to follow a new path.  In 1968, I went from TV News to be a tech writer for Collins Radio Co.   While in between writing successive military manuals about  Military Microwave and Troposcatter communications, I noticed an announcement on the bulletin board:  The company was offering a two-week course aimed at teaching the many electrical and mechanical engineers how to program in FORTRAN so they could use this huge slide rule called a Univac 1108 the company had just installed.   Curiosity got the best of me.   I took the course, and drove the instructor nuts with questions, but I was hooked. Several fortuitous situations eventually had me doing systems programming for that same Univac computer, and then for Sperry Univac support 1100-series systems, and then for Stratus Technology -- troubleshooting Stratus fault tolerant computer systems worldwide.   I finally retired in 2011 after a very fulfilling and lucrative career in computing that included almost two years living in Australia and traveling to support sites throughout the Far East.   So, I say: GO FOR IT.   

Thanks a lot for the support. It's pretty demanding right now putting in 10+ hours a day engineering then shifting to 4+ hours nightly for the classes. I think it will be worth it in the end. I've always been savvy with computers but no expert. Even just starting on this path has got me thinking about different things, and led me to here at this point. So it's all complimenting what I'm chasing. The area I'm living in is in a downward spiral. This is motivation that will help me re-invent my path. If I have time today I'm going to hopefully mess around with some backup schemes that I think will work with my situation.

Right now I'm thinking 1 monthly full per drive, staggered on 2 week intervals. Both of those will get bi-weekly incrementals, keeping 4 at one time in the storage folder (so 1 full + 4 increments). This will allow me to go back 2 weeks if necessary because of a unstable Windows Update. Thoughts?

After doing some more reading on this, it seems that if you are just doing full and incremental backups, this could cause an issue. From the Macrium information, if you want to restore to the latest incremental, then ALL of the incrementals preceding the latest one have to be present and functional. If one in the string is corrupt, then you're hosed, you need to go back to the full. So you are basically putting all your eggs in a basket and hoping that nothing gets corrupted.

I assume that goes for if you wanted to use the grandfather--father--son method. If you did a full-->diff-->multiple incrementals, everything in that line would have to be working to restore to the latest incr. If your incrementals got corrupted, you could still restore to your nearest diff update since you would still have a diff to use. This way, also, would give you more security I would think because you wouldn't have to go all the way back to the full to restore the image, whereas if you just took incrementals, if an early on one was corrupted, you're out of luck.

So I can see why now that these diff files are larger. They don't rely on each other for data...it only takes 1 diff image and 1 full image to restore a system. So 1 diff image a week, if you stored 2 of those, would give you the ability to restore to either of the last 2 weeks, since the image is being made off the full at that point in time.

Also makes sense incrementals are smaller since they only capture the changes from their previous image. Which would mean though, you'd need the entire chain to restore to the latest point. 
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
The only correction I'd make to what you just posted is that if one of your Incrementals is corrupt, you don't necessarily have to go back to the Full.  You can go back to an Incremental that was created before the corrupt one.  But if the corrupt Incremental is the first one in the chain, then yes you'd have to resort to the Full.

But other than that, you've got the idea.  A group of Diffs that share a common Full will have some "redundant" copies of data between them, which is why they can be independent of each other and only dependent on the Full.  Incrementals don't have that redundancy, and therefore they rely on an unbroken chain leading back to the Full.  As in other data scenarios, redundancy increases resilience.

But the scenario you didn't explicitly address is the impact of a corrupt Full -- which would render all backups in the set useless.  And that's why I strongly suggest that users maintain at least 2 Fulls.

krashDH
krashDH
New Member
New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)
Group: Forum Members
Posts: 12, Visits: 28

The only correction I'd make to what you just posted is that if one of your Incrementals is corrupt, you don't necessarily have to go back to the Full. You can go back to an Incremental that was created before the corrupt one. But if the corrupt Incremental is the first one in the chain, then yes you'd have to resort to the Full.
 Correct, yes I didn't specifically state that in my post above but it's understood.

But the scenario you didn't explicitly address is the impact of a corrupt Full -- which would render all backups in the set useless. And that's why I strongly suggest that users maintain at least 2 Fulls.
This is also true I didn't state. If your full is all jammered up, then yeah you have nothing in that chain that will help you. In my case personally, I now have a second XML file that duplicates what the first is doing, except it alternates 2 week increments and puts that second set of images on my external drive.

The other thing I was wondering. Currently I have both XML files on my D drive. I know when I set up Macrium and the first XML file I changed the path of where I keep those XML files (same folder I keep the images in so they're easier to find). Now I can't seem to find where to change that filepath, where Macrium looks to run those.

I'm wondering, I should probably copy those XML files to my C drive in case something were to happen to my D drive, so it could still run backups. I know I could copy-paste duplicates, but is there a way to tell Macrium to go look there if they can't be "found" in the D drive, kind of like alternate specifications for the backup folder? This way if my D did dump, at least Macrium would go find the folders on my C and could at least execute backups to my external drive until I got a new D installed or figured out what happened. I guess I'm coming up short on finding where that setting is to specify the XML filepath.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
Reflect doesn't have a single XML file path; it doesn't require you to store all XML files in a specific folder.  When you step through the wizard each time, if you choose to save the job you just set up as an XML file and click the Browse button, the folder you see will just be the most last folder where you saved an XML file.

In general, the default location for XML files is worth keeping. I've seen people get hugely jammed up by storing their XML files on locations that weren't reachable when the backup was supposed to run.  They had configured email notifications for job failures -- but not successes -- and so they simply assumed that if they weren't getting any email, then their backups were proceeding as normal.  Well it turned out that the location of their XML file had become unavailable for some reason, which obviously prevented the backup from running -- but because email notification settings are in the XML file, which was inaccessible, Reflect didn't know to send an email notification about the failure.  They went quite a while before realizing their backups weren't running -- although at least they found out before they had a major incident.

You also absolutely do NOT want to store your XML files with your actual backups, because if you use encryption, then the password is stored in the XML file, in which case storing the XML file with the backups amounts to storing the key with the lock.  The password is itself encrypted in the XML file, but that's done with a static key, because Reflect installations on completely different PCs encrypt the same password the same way.  (Although I posted a Wish List thread proposing a new design for this that would improve security without breaking compatibility.)

There are some other risks to consider that may or may not be relevant to your use case but that are summarized in this best practices document from Macrium.

In terms of moving them, you can just copy them somewhere else and then import the new copies into Reflect using the "plus sign" icon in the Backup Definition Files tab.  However, there's no way to transfer schedules over.  You'll have to recreate those and then delete the previous XML file.  And yes I'd recommend copying and importing the new one first rather than removing the existing one from Reflect before copying/moving, because when you remove an XML file from Reflect, you have a choice of whether or not to delete the underlying XML file and scheduled tasks.  But in your case you'd want to keep the former but remove the latter (since those tasks would point to an incorrect XML file path).  There isn't a way to do that, so you'd either have to keep both and then manually clean up "orphaned" scheduled tasks or else delete both recreate your XML files from scratch.  It's easier to just copy, import new, set up new schedules, THEN delete the old XML files and their associated tasks.

Edited 1 December 2020 8:21 PM by jphughan
krashDH
krashDH
New Member
New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)New Member (18 reputation)
Group: Forum Members
Posts: 12, Visits: 28
jphughan - 1 December 2020 8:20 PM
Reflect doesn't have a single XML file path; it doesn't require you to store all XML files in a specific folder.  When you step through the wizard each time, if you choose to save the job you just set up as an XML file and click the Browse button, the folder you see will just be the most last folder where you saved an XML file.

In general, the default location for XML files is worth keeping. I've seen people get hugely jammed up by storing their XML files on locations that weren't reachable when the backup was supposed to run.  They had configured email notifications for job failures -- but not successes -- and so they simply assumed that if they weren't getting any email, then their backups were proceeding as normal.  Well it turned out that the location of their XML file had become unavailable for some reason, which obviously prevented the backup from running -- but because email notification settings are in the XML file, which was inaccessible, Reflect didn't know to send an email notification about the failure.  They went quite a while before realizing their backups weren't running -- although at least they found out before they had a major incident.

You also absolutely do NOT want to store your XML files with your actual backups, because if you use encryption, then the password is stored in the XML file, in which case storing the XML file with the backups amounts to storing the key with the lock.  The password is itself encrypted in the XML file, but that's done with a static key, because Reflect installations on completely different PCs encrypt the same password the same way.  (Although I posted a Wish List thread proposing a new design for this that would improve security without breaking compatibility.)

There are some other risks to consider that may or may not be relevant to your use case but that are summarized in this best practices document from Macrium.

In terms of moving them, you can just copy them somewhere else and then import the new copies into Reflect using the "plus sign" icon in the Backup Definition Files tab.  However, there's no way to transfer schedules over.  You'll have to recreate those and then delete the previous XML file.  And yes I'd recommend copying and importing the new one first rather than removing the existing one from Reflect before copying/moving, because when you remove an XML file from Reflect, you have a choice of whether or not to delete the underlying XML file and scheduled tasks.  But in your case you'd want to keep the former but remove the latter (since those tasks would point to an incorrect XML file path).  There isn't a way to do that, so you'd either have to keep both and then manually clean up "orphaned" scheduled tasks or else delete both recreate your XML files from scratch.  It's easier to just copy, import new, set up new schedules, THEN delete the old XML files and their associated tasks.

OK I think now I'm officially lost. I've got a folder...windowsImageBackup, on my D drive. This is where my image files directly go. I have another folder within windowsImageBackup called ScheduleXML. Inside of that is where the XML files are. I originally made the D drive XML from scratch. Then copied that one and changed a few schedule parameters for my E drive backup. So both XML files are currently in that folder on my D drive, since I didn't step through the entire process to make the backup file for my E drive. So technically, they are in the XML and image files are in the same main directory, windowsImageBackup, but the XML files are in a separate folder within there.

Should I have them somewhere else? Should I store the XML file that I use for my external drive ON my external drive? Backup both of these folders somewhere on my C drive so if I need to import them if my D drive dies? I guess I missed the boat on your last post, I think that's above my pay grade
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)Macrium Evangelist (13K reputation)
Group: Forum Members
Posts: 8.8K, Visits: 59K
The default location for XML files is a folder called "Reflect" inside your user profile's Documents folder.  Unless you have a specific reason for storing them elsewhere, I'd store them there.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search