how can I get Reflect to automatically do differentials after a full backup (from the command line)


how can I get Reflect to automatically do differentials after a full...
Author
Message
DocDJ
DocDJ
New Member
New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)
Group: Forum Members
Posts: 27, Visits: 50
jphughan - 2 April 2022 3:01 PM
That's mostly correct, and I'll give you an example of why it works that way.  My own retention policy specifies to retain 2 Fulls and 2 weeks' worth of Incrementals.  But that does NOT mean that I want a Full every two weeks.  My schedule is in fact to create a new Full only every month at the beginning of the month.  So in my use case, at the end of a given month, I'll have my last two weeks' worth of Incrementals (which I create daily) and then a time gap of no remaining backups until the parent Full, because I started purging some of my daily Incrementals within the month.  So in my case, it would be entirely incorrect for Reflect to assume based on my retention policy that it should create a new Full after accumulating 2 weeks of Incrementals.  And similarly in your case, merely having a Diff retention policy specifying a certain period of time or a certain quantity of Diffs doesn't necessarily mean that when that limit is reached, it's time to create a new Full.  It could just mean you want to start purging older Diffs.

Or here's the opposite example.  Let's say I want to retain 4 weeks of daily Incrementals and therefore specify that as my retention policy, but I want weekly Fulls as a risk mitigation strategy.  Under the way you thought it worked, Reflect would see a 4-week retention policy and assume that I only wanted a Full every 4 weeks, which isn't correct.

Lastly, I'll also point out that the retention policies are optional.  You can uncheck any or all of the checkboxes beside the Full/Diff/Inc rows.  Some people do that because they prefer manual purging.  And in that case, you would certainly have to tell Reflect what sort of backup you want at any given time.

And that is why as I said above, retention policies govern the purging of existing backups, not the creation of new ones.  Reflect will not make any assumptions on the type of backup you want based on your retention policy and your existing backups, because those data points won't be sufficient to make correct assumptions for everyone's use case.

The reason I said "mostly correct" earlier is that you don't need a separate definition file to run Differential backups.  Just call the same definition file but specify "Diff" as the parameter.  If you'd be able to set up a solution wherein your batch file called one of two completely different batch files, I'm not sure why you wouldn't be able to set it up such that it tweaked a parameter instead.

Thanks VERY MUCH for this great explanation. Since I now have the full scripting capability (and the examples linked above), I think I have 2 ways to go:
1. I won't NEED an external batch file to call Macrium. I think I can create a pair of Definitions, one for my semi-monthly Full and one for semi-weekly Diffs and if I set the dates properly, I can let Macrium do the purging based on my retention rules and not accidentally create a "donut hole" with only Diffs and no Full. Do I assume correctly that, ifI do that, Macrium will always use the latest Full as the BASE for the next Diffs? This method would keep all the pieces together. 

2. I could, as an alternative, create 2 batch files (scheduled by Windows scheduler) to run the alternating Full and Diff backups, and use THOSE batch files to run my "reminder-window" programs. I really want to get this to work like Acronis, so I can drop Acronis (it has been giving me fits related to not completing runs. I have a bug report open on it). PS. I am trying to get Macrium running my way ever since MaximumPC recommended it.

Am I missing anything? Do you have any hints as to which is more flexible or easier to debug?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
Reflect will never leave you with only Diffs and no Fulls.  If Reflect purges a Full, it will automatically purge its child Diffs, and it can't create a new Diff if there's no valid parent Full.  The only way you'd end up with "orphaned" Diffs would be if you or some other application created that outcome.

Yes, when you request a Differential, Reflect will always append it to the newest "matching" Full.  (If you have a Full created by some completely separate backup job that contains completely separate source data, obviously Reflect won't build a Diff there.)  Similarly, Incrementals are always appended to the newest matching backup, regardless of type, because Incrementals can be appended to a Full or Diff or another Inc.

I still don't see any need for multiple definition files here, so I don't understand why you seem invested in that idea.  It's also not clear to me whether you want to have scheduled backups.  If you do, then starting with a Reflect-generated batch file and having Full and Diff schedules associated with that single batch file would be the way to go, because that way your "unified" batch file will run every time a Reflect backup of ANY kind is scheduled, but Reflect will still take care of making sure the correct backup type is run from that file because it will handle providing the appropriate parameter to achieve that.  If you do NOT wish to use scheduled backups, then I still don't see the need for separate definition files.  You can use a single definition file to create multiple types of backups.  You just need to add the appropriate parameter when you call the definition file.

As Beardy said earlier, it would probably help if you could explain in a bit more detail exactly what you're trying to achieve in terms of cadence, scheduled vs. manual backups, etc.  You say you want to get this working like Acronis, but that doesn't tell any of us here what you've got there.

DocDJ
DocDJ
New Member
New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)
Group: Forum Members
Posts: 27, Visits: 50
jphughan - 2 April 2022 3:58 PM
Reflect will never leave you with only Diffs and no Fulls.  If Reflect purges a Full, it will automatically purge its child Diffs, and it can't create a new Diff if there's no valid parent Full.  The only way you'd end up with "orphaned" Diffs would be if you or some other application created that outcome.

Yes, when you request a Differential, Reflect will always append it to the newest "matching" Full.  (If you have a Full created by some completely separate backup job that contains completely separate source data, obviously Reflect won't build a Diff there.)  Similarly, Incrementals are always appended to the newest matching backup, regardless of type, because Incrementals can be appended to a Full or Diff or another Inc.

I still don't see any need for multiple definition files here, so I don't understand why you seem invested in that idea.  It's also not clear to me whether you want to have scheduled backups.  If you do, then starting with a Reflect-generated batch file and having Full and Diff schedules associated with that single batch file would be the way to go, because that way your "unified" batch file will run every time a Reflect backup of ANY kind is scheduled, but Reflect will still take care of making sure the correct backup type is run from that file because it will handle providing the appropriate parameter to achieve that.  If you do NOT wish to use scheduled backups, then I still don't see the need for separate definition files.  You can use a single definition file to create multiple types of backups.  You just need to add the appropriate parameter when you call the definition file.

As Beardy said earlier, it would probably help if you could explain in a bit more detail exactly what you're trying to achieve in terms of cadence, scheduled vs. manual backups, etc.  You say you want to get this working like Acronis, but that doesn't tell any of us here what you've got there.

In Acronis, I tell it my schedule (daily, weekly, monthly). I tell it I want Full+Diff vs Full+Inc and how many of each to save before purging.I enter the names of the before and after batch files. My batch files open/close the "reminder-windows". All the backups for this set of data (files or partitions) are controlled by this ONE schedule (Acronis runs a Full, then runs Diffs until the purge rule is met, then automatically switches to a new Full). Acronis looks at the dest folder and determines whether to run a Diff or a new Full. After the backup, it purges both Full+Diffs based on my rules. I do not TELL it whether to run a Diff or a Full (there IS an option to explicitly run one or the other..
So I hope you see where my mind keeps going (even though I hate to be the one who says, "But I've always done it that way"). I want to be able to have Macrium do the same things, without ME providing input (after it's all set up, of course). And centralization helps, if I want to clone my operations for another backup.
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
Ok, that batch file design sounds totally different from what you've been describing.  Now you said you have before and after batch files.  Up until now it's sounded like you had a single batch file that you wanted to use to call Reflect in addition to performing other tasks.  Wanting Reflect to call before and after batch files is different from wanting to a batch file that itself calls Reflect, among possible other actions.  There are valid use cases for each technique, but they are distinct.

But if you do want backups to run on a schedule and you want to leverage before and after batch files, then create a single definition file.  Specify your desired retention policy, but NOT your schedule here.  Then right-click that definition file and choose "Generate a PowerShell script file".  In the Run Programs section of the wizard, enable the before and after items and specify your batch files, as well as any parameters that may be appropriate for those batch files themselves -- nothing to do with Reflect parameters.  Click OK.  (Note: The reason I suggested PowerShell is because Reflect's wizard for creating a batch file does not offer the Run Programs option.  I'm not sure why. )

Now go to your Scripts tab, where you will find that newly generated PowerShell script.  Right-click that and select Schedule.  Create your Full and Diff schedules as desired.

At this point, Reflect will have scheduled tasks that will call that single PowerShell script, automatically including the parameter for the appropriate backup to run at each scheduled occurrence.  And regardless of which backup type is running, that PowerShell script will call your before and/or after batch file(s).

Edited 2 April 2022 5:11 PM by jphughan
DocDJ
DocDJ
New Member
New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)
Group: Forum Members
Posts: 27, Visits: 50
jphughan - 2 April 2022 5:05 PM
Ok, that batch file design sounds totally different from what you've been describing.  Now you said you have before and after batch files.  Up until now it's sounded like you had a single batch file that you wanted to use to call Reflect in addition to performing other tasks.  Wanting Reflect to call before and after batch files is different from wanting to a batch file that itself calls Reflect, among possible other actions.  There are valid use cases for each technique, but they are distinct.

But if you do want backups to run on a schedule and you want to leverage before and after batch files, then create a single definition file.  Specify your desired retention policy, but NOT your schedule here.  Then right-click that definition file and choose "Generate a PowerShell script file".  In the Run Programs section of the wizard, enable the before and after items and specify your batch files, as well as any parameters that may be appropriate for those batch files themselves -- nothing to do with Reflect parameters.  Click OK.  (Note: The reason I suggested PowerShell is because Reflect's wizard for creating a batch file does not offer the Run Programs option.  I'm not sure why. )

Now go to your Scripts tab, where you will find that newly generated PowerShell script.  Right-click that and select Schedule.  Create your Full and Diff schedules as desired.

At this point, Reflect will have scheduled tasks that will call that single PowerShell script, automatically including the parameter for the appropriate backup to run at each scheduled occurrence.  And regardless of which backup type is running, that PowerShell script will call your before and/or after batch file(s).

That helps a LOT! It sounds like exactly what I need. I think the confusion was that I was discussing the way I was trying to do it with the NON-paid version (which doesn't allow scripts) vs the PAID version (which is more like Acronis). Sorry about that. I'll set it up and try to remember to post how it goes. May take a couple of days... Thanks again for your patience and GREAT details on the "how to do it". 
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
Happy to help!  The Free version does allow being called by a script, but it won't give you any help in generating the scripts that will do that or scheduling their execution.  You'd have to do all that yourself.  The paid version gives you that wizard to help you build a script that will run before and/or after programs and do some other things, as well as managing schedules for those scripts within Reflect. Smile

Edited 2 April 2022 5:54 PM by jphughan
DocDJ
DocDJ
New Member
New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)
Group: Forum Members
Posts: 27, Visits: 50
jphughan - 2 April 2022 5:53 PM
Happy to help!  The Free version does allow being called by a script, but it won't give you any help in generating the scripts that will do that or scheduling their execution.  You'd have to do all that yourself.  The paid version gives you that wizard to help you build a script that will run before and/or after programs and do some other things, as well as managing schedules for those scripts within Reflect. Smile

I followed the instructions and when the backup tried to run, it tried to access the target drive BEFORE it ran my scripts. of course, the drive was powered off, so the backup stopped. The primary reason for my "before" script is to warn me to turn on the switch for the target and to pop up a window saying the backup is in progress. Am I wrong about how it actually operates?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
Exactly how did you try to run the backup? There are several ways to do that in Reflect, and they don’t all work the same way. Did you also verify that executing the script directly from PowerShell Console results in the script behaving as expected?
DocDJ
DocDJ
New Member
New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)New Member (28 reputation)
Group: Forum Members
Posts: 27, Visits: 50
jphughan - 3 April 2022 1:58 PM
Exactly how did you try to run the backup? There are several ways to do that in Reflect, and they don’t all work the same way. Did you also verify that executing the script directly from PowerShell Console results in the script behaving as expected?

i let Macrium run the backup at the time specified in the schedule (which was set for a few minutes after I created the script and applied my batch files' names to the before and after slots). I tested the script which ran OK (and DID run my batch files), it also popped a Macrium notice that the backup would be run in "n" seconds (this also happened when it ran on the schedule.) Is there a way to BLOCK this Macrium notice?
jphughan
jphughan
Macrium Evangelist
Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)Macrium Evangelist (18K reputation)
Group: Forum Members
Posts: 12K, Visits: 72K
Ok, I didn't realize (or maybe forgot from earlier in this thread) that your batch file involves doing things that you expect to be visible to the user.  Reflect by default runs scheduled tasks under the SYSTEM account, which doesn't allow that because it's a separate user account running in a separate context.  If you go to Defaults and Settings > Schedule, you can change that to run under your own user account -- although be aware that if you do this, you'll have to remember to update that saved password if you ever change the Windows account password, otherwise backups will break.  And actually I'm not 100% sure even that will allow this to work as desired for you, because I think scheduled tasks run in a separate context even when the user account matches, so it might not be able to surface content in your actual user session.  But give it a try.

Regarding the countdown, that's under Defaults and Settings > Advanced > Advanced Backup Settings.  Just uncheck the notification option.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search