Macrium Support Forum

how can I get Reflect to automatically do differentials after a full backup (from the command line)

https://forum.macrium.com/Topic60510.aspx

By DocDJ - 1 April 2022 3:24 PM

I just bought Macrium Home 8. I have created a backup schedule with Full + Diff backups. I want to invoke Macrium from a batch file that will automatically decide whether it is time to do a new Full backup or do another Diff in the current set of Diffs.. The command line parameters only allow a value of Full, Diff or Incr with a default of Full if no type is specified.
By jphughan - 1 April 2022 4:43 PM

I don't understand your note that command line parameters "only" allow specifying a Full, Diff, or Incr. Those are all of the possible types of backups.  So what are you trying to achieve that you can't achieve?  And in your use case, what is the "decision criteria" for whether a Full or Diff should be created?  As long as you can express that in a batch file, you should be able to dynamically set a variable that will then be used as the backup type parameter in the line of your batch file that calls Reflect.

EDIT: If you want to use both a "traditional" schedule AND a batch file, then the right way to do that would be to create scheduled executions of the batch file, not the XML definition file.  To do that, right-click your definition file and choose to have Reflect generate a BAT file for you.  You can use the available options in the wizard or not, but after you click OK, that new batch file will then appear in Reflect under the Scripts tab.  Right-click THAT and select Schedule, and create your desired schedule there.  You can also then customize the batch file further as needed.
By DocDJ - 1 April 2022 6:50 PM

jphughan - 1 April 2022 4:43 PM
I don't understand your note that command line parameters "only" allow specifying a Full, Diff, or Incr. Those are all of the possible types of backups.  So what are you trying to achieve that you can't achieve?  And in your use case, what is the "decision criteria" for whether a Full or Diff should be created?  As long as you can express that in a batch file, you should be able to dynamically set a variable that will then be used as the backup type parameter in the line of your batch file that calls Reflect.

EDIT: If you want to use both a "traditional" schedule AND a batch file, then the right way to do that would be to create scheduled executions of the batch file, not the XML definition file.  To do that, right-click your definition file and choose to have Reflect generate a BAT file for you.  You can use the available options in the wizard or not, but after you click OK, that new batch file will then appear in Reflect under the Scripts tab.  Right-click THAT and select Schedule, and create your desired schedule there.  You can also then customize the batch file further as needed.

The purpose of the batch file is NOT to control Macreium, just start it running and let Macrium apply the rules defined in the GUI. The purpose of the batch file is to open a window BEFORE the backup starts and close the window after the backup ends. The window is only to remind me that a backup is still running and to not reboot, shutdown or power off the backup drive. The windows are used by MANY of my programs to be sure I turn the drive on/off at the proper time. The windows are created by a pair of Win32 programs that use a parameter to identify the program using the drive and to identify which window to close (thus allowing multiple programs to run simultaneously on different drives.)
I originally had Macrium free and used the bat file to run Macrium and pop the windows, but now that I have Macrium Home paid-for, I can use Macrium to run the batch files to pop the windows. Either way, the question of which type of run to perform is strictly dependent on what Macrium does and my bat files are independent of the program they work for. And I can use my batch files for anything that needs the switchable drives, by just creating a new parameter value.
By Drac144 - 1 April 2022 6:57 PM

Doc DJ,

Correct.  That is what JP is trying to help you do. By creating a scheduled backup within Reflect, a batch file is created which is what is run by the scheduler.  You can then EDIT THAT BATCH FILE and add you commands to open a window before Reflect is run (Reflect is initiated by a command in that batch file) and then close the window after Reflect completes. 
By jphughan - 1 April 2022 7:00 PM

Ok. If you schedule executions of the batch file within Reflect, then you can create separate Full and Diff schedules, in which case Reflect will pass the appropriate parameter to the batch file at each execution based on which schedule was triggered. And you can still have other customizations within the batch file to do things before and/or after Reflect starts.

If you don’t want to use Reflect to schedule executions of your batch file, then you’ll have to use some other mechanism to specify the appropriate parameter at each execution.

But there’s no way to say, “Call Reflect and have it determine what backup is supposed to run.” You have to TELL Reflect what you want it to do every time you call it. Even if you use Reflect to create schedules, what it does under the hood is essentially create a scheduled task that runs a command line that includes the appropriate backup type parameter based on the type of schedule you created. It doesn’t just “know” what to do if you call it without any parameter.
By Beardy - 1 April 2022 7:34 PM

I'm certain the outcome of what @DocDJ wants to happen is doable, I'm less certain exactly what it is though...
Getting Reflect to open a command window when the schedule runs a batch is pretty much as simple as having the schedule run it in their own account rather than SYSTEM.. the batch itself can contain logic to determine all sorts of things, opening or closing programs, determining what type of backup to run.. etc. etc. I actually do something similar, soo the batch can determine full diff or inc based off the day of the week & which (Monday) it is in the month.. & have the cmd window show me both progress & exit status of robocopy afterwards.

@DocDJ if we knew in detail rather than vague terms exactly what you wanted to happen, someone could probably suggest how to do it, & probably more easily using powershell than plain batch.. I may be one of the few here who actually prefer the older & more primitive scripting, which can be a royal PITA to get right if you're not familiar with it, more so than the other languages offered.
By jphughan - 1 April 2022 7:53 PM

My interpretation is that the OP wants to configure a schedule within Reflect that essentially says, “If some external script happens to call you at this particular tine of this particular day of the week, then run a Diff.” But that isn’t how scheduling works. Schedules in Reflect aren’t just a reference for what ti do IF Reflect gets called. They cause Reflect to MAKE something happen at the configured times, which would be entirely independent of a batch file you create — unless again you start with a Reflect-generated template, associate your schedules with that, and then customize it as needed for your purposes.
By Beardy - 1 April 2022 8:15 PM

I was suspecting possibly cart & horse reversed also.. a script might tell Reflect what to do.. but typically not vice-versa.
By DocDJ - 2 April 2022 11:56 AM

Beardy - 1 April 2022 8:15 PM
I was suspecting possibly cart & horse reversed also.. a script might tell Reflect what to do.. but typically not vice-versa.
@jphughan @Drac144 @Beardy
Let me see if I have this straight. If I create a single Macrium "definition" file with 2 "retention rules" (1 for Full, 1 for Diff), then am I correct in thinking Macrium does not have a method for determining whether to run a Full or Diff by the number of each in the target folder and matching that to my rules? Even though the GUI specifically says, "rules will be applied to all matching backup sets in the destination folder." (That's the way Acronis works, so my mind keeps coming back to that.) Both the GUI and the batch command require ME to specify whether to do a Full or Diff at the time I run Macrium. Of course, as suggested above, since I now have the paid version, I can create 2 separate definitions with the batch files being called by Macrium, but I'm still not clear on how/when/if Macrium decides that I am at the last Diff in my rule count and automatically switch to Full and clean up the Full's & Diff's. 
By jphughan - 2 April 2022 3:01 PM

That's mostly correct, and I'll give you examples of why it doesn't work the way you figured.  My own retention policy for my OS backup job specifies to retain 2 Fulls (in case one is corrupt) and 2 weeks' worth of Incrementals.  But that does NOT mean that I want a Full every two weeks.  My schedule in fact creates a new Full only on the first day of each month.  So in my use case, at the END of a given month, I'll have my last two weeks' worth of Incrementals (which I create daily) and then a time gap of no remaining backups until the parent Full, because my setup involves purging some of my daily Incrementals within the current month.  So in my case, it would be entirely incorrect for Reflect to assume based on my retention policy that it should create a new Full after accumulating 2 weeks of Incrementals.

Or here's the opposite example.  Let's say I want to retain 4 weeks of daily Incrementals and therefore specify that as my retention policy, but I want weekly Fulls as a risk mitigation strategy.  Under the way you thought it worked, Reflect would see a 4-week retention policy and assume that I only wanted a Full every 4 weeks, which isn't correct.

So back to your case, merely having a Diff retention policy specifying a certain period of time or a certain quantity of Diffs doesn't necessarily mean that when that limit is reached, it's time to create a new Full. It could just mean you want to start purging older Diffs.  Or for that matter it doesn't necessarily mean that Reflect should have waited that LONG to create a new Full.  Maybe you want to split your total Diffs across multiple Fulls.  (That's another thing to be aware of.  The Diff and Incremental retention policies are TOTAL figures, not per-Full figures.  And that design also enables use cases that wouldn't otherwise be possible, such as retaining 6 weeks of Incrementals across two monthly Fulls rather than requiring each Full to have the same time period or quantity of Incrementals.)

Lastly, I'll also point out that the retention policies are optional.  You can uncheck any or all of the checkboxes beside the Full/Diff/Inc rows.  Some people do that because they prefer manual purging.  And in that case, you would certainly have to tell Reflect what sort of backup you want at any given time.

And that is why as I said earlier, retention policies govern the purging of existing backups, not the creation of new ones.  Reflect will not make any assumptions on the type of backup you want based on your retention policy and your existing backups, because those data points won't be sufficient to make correct assumptions for everyone's use case.

The reason I said "mostly correct" earlier is that you don't need a separate definition file to run Differential backups.  Just call the same definition file but specify "Diff" as the parameter.  If you'd be able to set up a solution wherein your batch file called one of two completely different batch files, I'm not sure why you wouldn't be able to set it up such that it tweaked a parameter instead.
By DocDJ - 2 April 2022 3:39 PM

jphughan - 2 April 2022 3:01 PM
That's mostly correct, and I'll give you an example of why it works that way.  My own retention policy specifies to retain 2 Fulls and 2 weeks' worth of Incrementals.  But that does NOT mean that I want a Full every two weeks.  My schedule is in fact to create a new Full only every month at the beginning of the month.  So in my use case, at the end of a given month, I'll have my last two weeks' worth of Incrementals (which I create daily) and then a time gap of no remaining backups until the parent Full, because I started purging some of my daily Incrementals within the month.  So in my case, it would be entirely incorrect for Reflect to assume based on my retention policy that it should create a new Full after accumulating 2 weeks of Incrementals.  And similarly in your case, merely having a Diff retention policy specifying a certain period of time or a certain quantity of Diffs doesn't necessarily mean that when that limit is reached, it's time to create a new Full.  It could just mean you want to start purging older Diffs.

Or here's the opposite example.  Let's say I want to retain 4 weeks of daily Incrementals and therefore specify that as my retention policy, but I want weekly Fulls as a risk mitigation strategy.  Under the way you thought it worked, Reflect would see a 4-week retention policy and assume that I only wanted a Full every 4 weeks, which isn't correct.

Lastly, I'll also point out that the retention policies are optional.  You can uncheck any or all of the checkboxes beside the Full/Diff/Inc rows.  Some people do that because they prefer manual purging.  And in that case, you would certainly have to tell Reflect what sort of backup you want at any given time.

And that is why as I said above, retention policies govern the purging of existing backups, not the creation of new ones.  Reflect will not make any assumptions on the type of backup you want based on your retention policy and your existing backups, because those data points won't be sufficient to make correct assumptions for everyone's use case.

The reason I said "mostly correct" earlier is that you don't need a separate definition file to run Differential backups.  Just call the same definition file but specify "Diff" as the parameter.  If you'd be able to set up a solution wherein your batch file called one of two completely different batch files, I'm not sure why you wouldn't be able to set it up such that it tweaked a parameter instead.

Thanks VERY MUCH for this great explanation. Since I now have the full scripting capability (and the examples linked above), I think I have 2 ways to go:
1. I won't NEED an external batch file to call Macrium. I think I can create a pair of Definitions, one for my semi-monthly Full and one for semi-weekly Diffs and if I set the dates properly, I can let Macrium do the purging based on my retention rules and not accidentally create a "donut hole" with only Diffs and no Full. Do I assume correctly that, ifI do that, Macrium will always use the latest Full as the BASE for the next Diffs? This method would keep all the pieces together. 

2. I could, as an alternative, create 2 batch files (scheduled by Windows scheduler) to run the alternating Full and Diff backups, and use THOSE batch files to run my "reminder-window" programs. I really want to get this to work like Acronis, so I can drop Acronis (it has been giving me fits related to not completing runs. I have a bug report open on it). PS. I am trying to get Macrium running my way ever since MaximumPC recommended it.

Am I missing anything? Do you have any hints as to which is more flexible or easier to debug?
By jphughan - 2 April 2022 3:58 PM

Reflect will never leave you with only Diffs and no Fulls.  If Reflect purges a Full, it will automatically purge its child Diffs, and it can't create a new Diff if there's no valid parent Full.  The only way you'd end up with "orphaned" Diffs would be if you or some other application created that outcome.

Yes, when you request a Differential, Reflect will always append it to the newest "matching" Full.  (If you have a Full created by some completely separate backup job that contains completely separate source data, obviously Reflect won't build a Diff there.)  Similarly, Incrementals are always appended to the newest matching backup, regardless of type, because Incrementals can be appended to a Full or Diff or another Inc.

I still don't see any need for multiple definition files here, so I don't understand why you seem invested in that idea.  It's also not clear to me whether you want to have scheduled backups.  If you do, then starting with a Reflect-generated batch file and having Full and Diff schedules associated with that single batch file would be the way to go, because that way your "unified" batch file will run every time a Reflect backup of ANY kind is scheduled, but Reflect will still take care of making sure the correct backup type is run from that file because it will handle providing the appropriate parameter to achieve that.  If you do NOT wish to use scheduled backups, then I still don't see the need for separate definition files.  You can use a single definition file to create multiple types of backups.  You just need to add the appropriate parameter when you call the definition file.

As Beardy said earlier, it would probably help if you could explain in a bit more detail exactly what you're trying to achieve in terms of cadence, scheduled vs. manual backups, etc.  You say you want to get this working like Acronis, but that doesn't tell any of us here what you've got there.
By DocDJ - 2 April 2022 4:54 PM

jphughan - 2 April 2022 3:58 PM
Reflect will never leave you with only Diffs and no Fulls.  If Reflect purges a Full, it will automatically purge its child Diffs, and it can't create a new Diff if there's no valid parent Full.  The only way you'd end up with "orphaned" Diffs would be if you or some other application created that outcome.

Yes, when you request a Differential, Reflect will always append it to the newest "matching" Full.  (If you have a Full created by some completely separate backup job that contains completely separate source data, obviously Reflect won't build a Diff there.)  Similarly, Incrementals are always appended to the newest matching backup, regardless of type, because Incrementals can be appended to a Full or Diff or another Inc.

I still don't see any need for multiple definition files here, so I don't understand why you seem invested in that idea.  It's also not clear to me whether you want to have scheduled backups.  If you do, then starting with a Reflect-generated batch file and having Full and Diff schedules associated with that single batch file would be the way to go, because that way your "unified" batch file will run every time a Reflect backup of ANY kind is scheduled, but Reflect will still take care of making sure the correct backup type is run from that file because it will handle providing the appropriate parameter to achieve that.  If you do NOT wish to use scheduled backups, then I still don't see the need for separate definition files.  You can use a single definition file to create multiple types of backups.  You just need to add the appropriate parameter when you call the definition file.

As Beardy said earlier, it would probably help if you could explain in a bit more detail exactly what you're trying to achieve in terms of cadence, scheduled vs. manual backups, etc.  You say you want to get this working like Acronis, but that doesn't tell any of us here what you've got there.

In Acronis, I tell it my schedule (daily, weekly, monthly). I tell it I want Full+Diff vs Full+Inc and how many of each to save before purging.I enter the names of the before and after batch files. My batch files open/close the "reminder-windows". All the backups for this set of data (files or partitions) are controlled by this ONE schedule (Acronis runs a Full, then runs Diffs until the purge rule is met, then automatically switches to a new Full). Acronis looks at the dest folder and determines whether to run a Diff or a new Full. After the backup, it purges both Full+Diffs based on my rules. I do not TELL it whether to run a Diff or a Full (there IS an option to explicitly run one or the other..
So I hope you see where my mind keeps going (even though I hate to be the one who says, "But I've always done it that way"). I want to be able to have Macrium do the same things, without ME providing input (after it's all set up, of course). And centralization helps, if I want to clone my operations for another backup.
By jphughan - 2 April 2022 5:05 PM

Ok, that batch file design sounds totally different from what you've been describing.  Now you said you have before and after batch files.  Up until now it's sounded like you had a single batch file that you wanted to use to call Reflect in addition to performing other tasks.  Wanting Reflect to call before and after batch files is different from wanting to a batch file that itself calls Reflect, among possible other actions.  There are valid use cases for each technique, but they are distinct.

But if you do want backups to run on a schedule and you want to leverage before and after batch files, then create a single definition file.  Specify your desired retention policy, but NOT your schedule here.  Then right-click that definition file and choose "Generate a PowerShell script file".  In the Run Programs section of the wizard, enable the before and after items and specify your batch files, as well as any parameters that may be appropriate for those batch files themselves -- nothing to do with Reflect parameters.  Click OK.  (Note: The reason I suggested PowerShell is because Reflect's wizard for creating a batch file does not offer the Run Programs option.  I'm not sure why. )

Now go to your Scripts tab, where you will find that newly generated PowerShell script.  Right-click that and select Schedule.  Create your Full and Diff schedules as desired.

At this point, Reflect will have scheduled tasks that will call that single PowerShell script, automatically including the parameter for the appropriate backup to run at each scheduled occurrence.  And regardless of which backup type is running, that PowerShell script will call your before and/or after batch file(s).
By DocDJ - 2 April 2022 5:51 PM

jphughan - 2 April 2022 5:05 PM
Ok, that batch file design sounds totally different from what you've been describing.  Now you said you have before and after batch files.  Up until now it's sounded like you had a single batch file that you wanted to use to call Reflect in addition to performing other tasks.  Wanting Reflect to call before and after batch files is different from wanting to a batch file that itself calls Reflect, among possible other actions.  There are valid use cases for each technique, but they are distinct.

But if you do want backups to run on a schedule and you want to leverage before and after batch files, then create a single definition file.  Specify your desired retention policy, but NOT your schedule here.  Then right-click that definition file and choose "Generate a PowerShell script file".  In the Run Programs section of the wizard, enable the before and after items and specify your batch files, as well as any parameters that may be appropriate for those batch files themselves -- nothing to do with Reflect parameters.  Click OK.  (Note: The reason I suggested PowerShell is because Reflect's wizard for creating a batch file does not offer the Run Programs option.  I'm not sure why. )

Now go to your Scripts tab, where you will find that newly generated PowerShell script.  Right-click that and select Schedule.  Create your Full and Diff schedules as desired.

At this point, Reflect will have scheduled tasks that will call that single PowerShell script, automatically including the parameter for the appropriate backup to run at each scheduled occurrence.  And regardless of which backup type is running, that PowerShell script will call your before and/or after batch file(s).

That helps a LOT! It sounds like exactly what I need. I think the confusion was that I was discussing the way I was trying to do it with the NON-paid version (which doesn't allow scripts) vs the PAID version (which is more like Acronis). Sorry about that. I'll set it up and try to remember to post how it goes. May take a couple of days... Thanks again for your patience and GREAT details on the "how to do it". 
By jphughan - 2 April 2022 5:53 PM

Happy to help!  The Free version does allow being called by a script, but it won't give you any help in generating the scripts that will do that or scheduling their execution.  You'd have to do all that yourself.  The paid version gives you that wizard to help you build a script that will run before and/or after programs and do some other things, as well as managing schedules for those scripts within Reflect. Smile
By DocDJ - 3 April 2022 1:19 PM

jphughan - 2 April 2022 5:53 PM
Happy to help!  The Free version does allow being called by a script, but it won't give you any help in generating the scripts that will do that or scheduling their execution.  You'd have to do all that yourself.  The paid version gives you that wizard to help you build a script that will run before and/or after programs and do some other things, as well as managing schedules for those scripts within Reflect. Smile

I followed the instructions and when the backup tried to run, it tried to access the target drive BEFORE it ran my scripts. of course, the drive was powered off, so the backup stopped. The primary reason for my "before" script is to warn me to turn on the switch for the target and to pop up a window saying the backup is in progress. Am I wrong about how it actually operates?
By jphughan - 3 April 2022 1:58 PM

Exactly how did you try to run the backup? There are several ways to do that in Reflect, and they don’t all work the same way. Did you also verify that executing the script directly from PowerShell Console results in the script behaving as expected?
By DocDJ - 3 April 2022 3:18 PM

jphughan - 3 April 2022 1:58 PM
Exactly how did you try to run the backup? There are several ways to do that in Reflect, and they don’t all work the same way. Did you also verify that executing the script directly from PowerShell Console results in the script behaving as expected?

i let Macrium run the backup at the time specified in the schedule (which was set for a few minutes after I created the script and applied my batch files' names to the before and after slots). I tested the script which ran OK (and DID run my batch files), it also popped a Macrium notice that the backup would be run in "n" seconds (this also happened when it ran on the schedule.) Is there a way to BLOCK this Macrium notice?
By jphughan - 3 April 2022 3:28 PM

Ok, I didn't realize (or maybe forgot from earlier in this thread) that your batch file involves doing things that you expect to be visible to the user.  Reflect by default runs scheduled tasks under the SYSTEM account, which doesn't allow that because it's a separate user account running in a separate context.  If you go to Defaults and Settings > Schedule, you can change that to run under your own user account -- although be aware that if you do this, you'll have to remember to update that saved password if you ever change the Windows account password, otherwise backups will break.  And actually I'm not 100% sure even that will allow this to work as desired for you, because I think scheduled tasks run in a separate context even when the user account matches, so it might not be able to surface content in your actual user session.  But give it a try.

Regarding the countdown, that's under Defaults and Settings > Advanced > Advanced Backup Settings.  Just uncheck the notification option.
By DocDJ - 3 April 2022 3:51 PM

jphughan - 3 April 2022 3:28 PM
Ok, I didn't realize (or maybe forgot from earlier in this thread) that your batch file involves doing things that you expect to be visible to the user.  Reflect by default runs scheduled tasks under the SYSTEM account, which doesn't allow that because it's a separate user account running in a separate context.  If you go to Defaults and Settings > Schedule, you can change that to run under your own user account -- although be aware that if you do this, you'll have to remember to update that saved password if you ever change the Windows account password, otherwise backups will break.  And actually I'm not 100% sure even that will allow this to work as desired for you, because I think scheduled tasks run in a separate context even when the user account matches, so it might not be able to surface content in your actual user session.  But give it a try.

Regarding the countdown, that's under Defaults and Settings > Advanced > Advanced Backup Settings.  Just uncheck the notification option.

it didn't run the script.So the backup failed. But the suppression of M's backup warning worked.
By jphughan - 3 April 2022 3:57 PM

That's not a lot of information to go on, especially if it's not even precisely accurate.  Saying "it didn't run the script" may not even be accurate. The script not running at all is different, and would have different possible causes, compared to the script running and erroring out, or running and getting stalled somewhere.  The more information you can provide about problem behavior, the more likely others will be able to help.

You can manually invoke scheduled tasks under the Scheduled Backups tab.  That's also where you will see a exit code for the last execution of the task.  What does that say?  And for diagnostic purposes it might also help to customize the batch file to do something to indicate how far it's getting, such as writing out to a log file that you can review later.  But if the Before batch file called by your script involves displaying an interactive prompt and waiting for input before proceeding, that might be an issue, because you might have an interactive prompt in a user context that your user session can't access.  In that case, the script will be running but not able to complete.
By DocDJ - 3 April 2022 4:25 PM

jphughan - 3 April 2022 3:57 PM
That's not a lot of information to go on, especially if it's not even precisely accurate.  Saying "it didn't run the script" may not even be accurate. The script not running at all is different, and would have different possible causes, compared to the script running and erroring out, or running and getting stalled somewhere.  The more information you can provide about problem behavior, the more likely others will be able to help.

You can manually invoke scheduled tasks under the Scheduled Backups tab.  That's also where you will see a exit code for the last execution of the task.  What does that say?  And for diagnostic purposes it might also help to customize the batch file to do something to indicate how far it's getting, such as writing out to a log file that you can review later.  But if the Before batch file called by your script involves displaying an interactive prompt and waiting for input before proceeding, that might be an issue, because you might have an interactive prompt in a user context that your user session can't access.  In that case, the script will be running but not able to complete.

"didn't run" implies you are correct that I have an interactive prompt in a user context that my user session can't access, because that is the first thing it does after it tests for the existence of the drive (so it prompts me to turn on the drive, THEN pops up my "I'm running" window). I guess I may need to research if it is possible to run a program from my "user context" ino my REAL context and how to do it.
By jphughan - 3 April 2022 4:49 PM

You can manually create scheduled tasks in Windows Task Scheduler that work that way. Just have the scheduled task call PowerShell with the “File” parameter to specify the script path and the desired backup type. And then set the task to run as your user account and only while you’re logged in. Those scheduled tasks won’t show up in Reflect, but you can manage them from Windows Task Scheduler.

The catch to running backups in your user session is that a) they won’t run if you’re not logged in, and b) they will be cancelled if you log off mid-backup. Background scheduled tasks don’t have those limitations.
By DocDJ - 3 April 2022 4:54 PM

jphughan - 3 April 2022 4:49 PM
You can manually create scheduled tasks in Windows Task Scheduler that work that way. Just have the scheduled task call PowerShell with the “File” parameter to specify the script path and the desired backup type. And then set the task to run as your user account and only while you’re logged in. Those scheduled tasks won’t show up in Reflect, but you can manage them from Windows Task Scheduler.The catch to running backups in your user session is that a) they won’t run if you’re not logged in, and b) they will be cancelled if you log off mid-backup. Background scheduled tasks don’t have those limitations.

I'll give that a try. However I LIKE the fact that it has to run while I'm logged on (otherwise, how can the drive switch be turned on?)
PS. I have enquired with some manufacturers about USB switches that can be involved via KASA smart programming, without voice., which would solve that problem. Nobody makes anything other than 110V AC switches.
By jphughan - 3 April 2022 5:36 PM

Then it sounds like that might be a good solution for you.  I was just explaining why Reflect-generated scheduled tasks don't work like that.  If they did, it wouldn't be possible to have scheduled backups of most servers, where user logons are rare.  And having to stay logged in until a backup completes could be annoying in that scenario as well.
By JK - 3 April 2022 8:26 PM

@DocDJ
The primary reason for my "before" script is to warn me to turn on the switch for the target and to pop up a window saying the backup is in progress.


FYI, Steve Smith (MVP in the Acronis Forum) developed a Powershell script to connect and disconnect an external USB drive, without the need for manual intervention.  It is available here:

https://forum.acronis.com/comment/557182#comment-557182
By jphughan - 3 April 2022 8:38 PM

^ If the drive remains physically connected, then malware can potentially do whatever that PowerShell script can.
By JK - 3 April 2022 8:44 PM

JP, good point, and this is also discussed somewhat in the linked thread (see here).  Nonetheless, could be a reasonable strategy considering that what is possible is not always probable.
By DocDJ - 3 April 2022 9:22 PM

JK - 3 April 2022 8:44 PM
JP, good point, and this is also discussed somewhat in the linked thread (see here).  Nonetheless, could be a reasonable strategy considering that what is possible is not always probable.

having had a ransomware attack on a system I supported (the user clicked on a poison link) and all the attacks on FB users, I am more than a little paranoid. I have been digitizing 70 years' worth of my own photos and family heirloom shots to conserve space. If I get attacked, they are gone, with nothing for my descendants to look at (assuming they might want to) once the paper is chucked out. That's why I'm looking for a Kasa enabled USB switch, whether voice operated or programmable, for unattended operations. At the moment, I schedule my backups for a time when I am (generally) at my PC, so I can manually power them on, but I need to be reminded to NOT power them off while the backups are running. So this batch operation is really important to me. My PC is always on from 6AM to midnight. But in idle mode (NEVER sleep mode), it uses about as much power as a light bulb. Acronis allows this and my batch files work well, but Acronis has other problems that Macrium seems not to have, not to mention annual "subscription" fees.
By Beardy - 4 April 2022 1:21 AM

Well I've not yet transferred to the Macrium scheduler, but the Windows one can certainly run batch files that open visibly to the logged in user if they're set to run in their account, I ran my backups that way for years (and am considering going back to it, an upgrade switched the schedule to system & I've yet to bother reverting the change) just because I could monitor robocopy's progress post backup that way.
By jphughan - 4 April 2022 2:50 AM

That's true, but even when Reflect is using Windows Task Scheduler, the scheduled tasks that Reflect creates within WTS don't work that way.  And to my knowledge it's not possible to have Reflect create them in the way that would make them interactive within the logged-on user's session.
By Beardy - 4 April 2022 5:13 AM

Indeed, but the path & name of the batch is known, & Windows has this "create task" wizard within the task scheduler, which is quite friendly to use, created tasks aren't that difficult to edit eithe.

By DocDJ - 4 April 2022 10:45 AM

jphughan - 3 April 2022 4:49 PM
You can manually create scheduled tasks in Windows Task Scheduler that work that way. Just have the scheduled task call PowerShell with the “File” parameter to specify the script path and the desired backup type. And then set the task to run as your user account and only while you’re logged in. Those scheduled tasks won’t show up in Reflect, but you can manage them from Windows Task Scheduler.The catch to running backups in your user session is that a) they won’t run if you’re not logged in, and b) they will be cancelled if you log off mid-backup. Background scheduled tasks don’t have those limitations.
Doesn't this technique "specify the script path and the desired backup type" go back to making my batch file responsible for deciding when to do Diff vs Full backups?
By Beardy - 4 April 2022 11:29 AM

Well something needs to specify it, can be the schedule, can be something in your script (batch powershell or other). Can even be a parameter passed to your script.
Reflect doesn't determine what backup type to do, with the exception that if there's none you get a full.

Retention policies won't help you, they're about storage space management, and what history you retain, not what type of backup is done on any particular run.

I'm not sure we can be a great deal more help without knowing exactly what you want to happen, some things would be easy to simply schedule..
e.g. Full every Monday, diff every other day
Full first Monday of month, Diff every Tuesday, inc other days..
Pretty much any regular pattern is doable, peculiar ones are tedious (or can be) to set up & might need the Windows scheduler to do.

Some things will take some scripting to actually figure out what you have, so say "Every time I've accumulated 9 diffs I want a new full" regardless of producing backups on an ad-hoc basis with no regularity would take measures like parsing a directory listing & counting them to set a parameter based off the findings.. you'd have to do that the hard way.  But that's pretty much your only option if you do sporadic irregular backups & want a set pattern of so many diffs after each full.

Frankly (my opinion) you're far better deciding a regular schedule you can just set..
By DocDJ - 4 April 2022 12:24 PM

Beardy - 4 April 2022 11:29 AM
Well something needs to specify it, can be the schedule, can be something in your script (batch powershell or other). Can even be a parameter passed to your script.
Reflect doesn't determine what backup type to do, with the exception that if there's none you get a full.

Retention policies won't help you, they're about storage space management, and what history you retain, not what type of backup is done on any particular run.

I'm not sure we can be a great deal more help without knowing exactly what you want to happen, some things would be easy to simply schedule..
e.g. Full every Monday, diff every other day
Full first Monday of month, Diff every Tuesday, inc other days..
Pretty much any regular pattern is doable, peculiar ones are tedious (or can be) to set up & might need the Windows scheduler to do.

Some things will take some scripting to actually figure out what you have, so say "Every time I've accumulated 9 diffs I want a new full" regardless of producing backups on an ad-hoc basis with no regularity would take measures like parsing a directory listing & counting them to set a parameter based off the findings.. you'd have to do that the hard way.  But that's pretty much your only option if you do sporadic irregular backups & want a set pattern of so many diffs after each full.

Frankly (my opinion) you're far better deciding a regular schedule you can just set..

@Beardy I understand the scheduling operation and I think I may be stuck with creating Windows-scheduled programs which call Macrium via command-line and tell it what kind of backup to do. HOWEVER, I don't think that batch files will solve my problem of getting my pre-and post window dialogs to show up at the right times. Yes, I can handle the pre-backup popup in the batch file that STARTS Macrium, but in order to create the POST- backup window, that batch file must remain on my screen until the backup is finished and Macrium exits. (As a  test, I created a shortcut to my PRE- bat file and set it to run "minimized". But doing that prevented it from displaying my prompt message to turn on the drive.) I will try a real WiIn32 program instead of a BAT or CMD file and have it prompt for and test for the drive to come on line, then have it call Macrium, then prompt me to turn off the drive and close the already open reminder window. That will make it all one program (not just a batch file). Since the two programs to prompt me are already Win32, I can merge them. That's a LOT more work to help me dump Acronis, but may be worth it.

HOWEVER, I know that Macrium uses a separate context to run the backups and scripts, so they can't write to the screen, but I will test to see if they can still pop the window and let IT appear on the screen.
By DocDJ - 4 April 2022 1:04 PM

DocDJ - 4 April 2022 12:24 PM
Beardy - 4 April 2022 11:29 AM
Well something needs to specify it, can be the schedule, can be something in your script (batch powershell or other). Can even be a parameter passed to your script.
Reflect doesn't determine what backup type to do, with the exception that if there's none you get a full.

Retention policies won't help you, they're about storage space management, and what history you retain, not what type of backup is done on any particular run.

I'm not sure we can be a great deal more help without knowing exactly what you want to happen, some things would be easy to simply schedule..
e.g. Full every Monday, diff every other day
Full first Monday of month, Diff every Tuesday, inc other days..
Pretty much any regular pattern is doable, peculiar ones are tedious (or can be) to set up & might need the Windows scheduler to do.

Some things will take some scripting to actually figure out what you have, so say "Every time I've accumulated 9 diffs I want a new full" regardless of producing backups on an ad-hoc basis with no regularity would take measures like parsing a directory listing & counting them to set a parameter based off the findings.. you'd have to do that the hard way.  But that's pretty much your only option if you do sporadic irregular backups & want a set pattern of so many diffs after each full.

Frankly (my opinion) you're far better deciding a regular schedule you can just set..

@Beardy I understand the scheduling operation and I think I may be stuck with creating Windows-scheduled programs which call Macrium via command-line and tell it what kind of backup to do. HOWEVER, I don't think that batch files will solve my problem of getting my pre-and post window dialogs to show up at the right times. Yes, I can handle the pre-backup popup in the batch file that STARTS Macrium, but in order to create the POST- backup window, that batch file must remain on my screen until the backup is finished and Macrium exits. (As a  test, I created a shortcut to my PRE- bat file and set it to run "minimized". But doing that prevented it from displaying my prompt message to turn on the drive.) I will try a real WiIn32 program instead of a BAT or CMD file and have it prompt for and test for the drive to come on line, then have it call Macrium, then prompt me to turn off the drive and close the already open reminder window. That will make it all one program (not just a batch file). Since the two programs to prompt me are already Win32, I can merge them. That's a LOT more work to help me dump Acronis, but may be worth it.

HOWEVER, I know that Macrium uses a separate context to run the backups and scripts, so they can't write to the screen, but I will test to see if they can still pop the window and let IT appear on the screen.

Well, unless I messed up the script, the test failed - bummer. So I guess it all has to be done outside Macrium. It would be nice if Macrium had an option to run the backup in the users' context (not just user's authority), so the commands called by the scripts could output to the console.
Here's how I modified my starting script:
$strRunAtStartApp = 'E:\DJs Documents\my_vs_executables\running_program-reminder.exe;
$strRunAtStartArgs = 'testing"';
By jphughan - 4 April 2022 1:24 PM

No, the technique I described would not have required you to have your batch file decide which backup to run. You would have had two separate tasks, one for a Full and one for a Diff. The tasks would have called the same batch file, but each task would have been configured to pass a different PARAMETER to that batch file. This mimics what Reflect does when you define schedules within it. The reason you’d have been creating a scheduled task manually would have been to configure one that can run interactively.
By DocDJ - 4 April 2022 2:58 PM

jphughan - 4 April 2022 1:24 PM
No, the technique I described would not have required you to have your batch file decide which backup to run. You would have had two separate tasks, one for a Full and one for a Diff. The tasks would have called the same batch file, but each task would have been configured to pass a different PARAMETER to that batch file. This mimics what Reflect does when you define schedules within it. The reason you’d have been creating a scheduled task manually would have been to configure one that can run interactively.

Thanks. I understand a little better now. My needs DO require the interactive activity pre- and post- backup. So I am going to try it with 2 Windows -scheduled tasks calling the same batch file passing it "Full" or "Diff" based on my needs and letting IT do the pop-ups. I think part of my thick-headedness has been thinking of "schedule" as "when AND how to run the backup" as opposed to Macrium's when/what to purge. 

By jphughan - 4 April 2022 3:21 PM

A scheduled task DOES handle "when and how to run the backup".  The retention policy purges occur as part of the backup job; they are not a mechanism that must be run separately.  So I'm not sure it's entirely clear yet.  But I'll try again.

A basic schedule setup involves two main components.
  • An XML definition file, which descries the backup job in terms of WHAT to back up, WHERE to store the backups, WHAT (if any) retention policy to enforce, etc.  But it does NOT handle WHEN to run it (the schedule), nor does it handle the TYPE of backup to perform (Full/Diff/Inc).
  • One or more scheduled tasks.  These are entirely separate entities from the XML definition file.  A scheduled task is configured to perform an action on a particular schedule.  In a regular case, the "action" is to run the Reflect executable.  In addition to running that application, the task itself can pass parameters to that application.  One of those parameter will be the path to the XML definition file, so you're saying "Whenever this scheduled task fires on whatever schedule has been configured for it, tell Reflect to run this job".  Then you can also optionally specify a backup TYPE parameter so you can say "Specifically, run a Differential backup of this job".  So if you want to schedule various backup types of the same job, then you'll have one scheduled task for each type, and those scheduled tasks will be identical EXCEPT for the single parameter that each task is configured to send to Reflect.  There are other parameters that may be appropriate to use when calling Reflect, documented here.
But in YOUR case where you want to run Before and After batch files, you'd have some additional complexity.  You now need to have a script that does all of that, i.e. a) calls your Before batch file, b) runs Reflect, including passing it the appropriate parameters, and c) running your After batch file.  And if you want to use the same "master script" for both types of Reflect backups, then that "master script" itself needs to support receiving a backup type parameter and be configured to pass that to Reflect at the point in that script where it runs Reflect.  (The other parameters that should be passed to Reflect, such as the XML definition file and anything else you might want to use in the documentation, can probably be hardcoded into your script as part of the command it runs at that point.)  So now, you'll still have one or more scheduled tasks as above, but instead of calling Reflect and providing all Reflect-appropriate parameters, they will instead call your "master script" and specify only a backup type parameter, because the script will have the rest hardcoded to pass to Reflect.

I don't know if I've made it clearer or more confusing at this point, but hopefully it's the former.
By Beardy - 4 April 2022 7:18 PM

@Beardy I understand the scheduling operation and I think I may be stuck with creating Windows-scheduled programs which call Macrium via command-line and tell it what kind of backup to do. HOWEVER, I don't think that batch files will solve my problem of getting my pre-and post window dialogs to show up at the right times. Yes, I can handle the pre-backup popup in the batch file that STARTS Macrium, but in order to create the POST- backup window, that batch file must remain on my screen until the backup is finished and Macrium exits. (As a  test, I created a shortcut to my PRE- bat file and set it to run "minimized". But doing that prevented it from displaying my prompt message to turn on the drive.) I will try a real WiIn32 program instead of a BAT or CMD file and have it prompt for and test for the drive to come on line, then have it call Macrium, then prompt me to turn off the drive and close the already open reminder window. That will make it all one program (not just a batch file). Since the two programs to prompt me are already Win32, I can merge them. That's a LOT more work to help me dump Acronis, but may be worth it.

I'm still struggling to see the issue I run Reflect from a batch file, that batch file does things both before and after Reflect runs, if I create or have reflect generate the batch, add such actions as I need before and after the backup.. Momentarily forget Reflect exists (even though the batch runs it) and use Windows task scheduler to create a task to run that batch job as Myself only when I'm logged in, when it runs, it opens a command window with which I can interact, (and for testing purposes I have had PAUSE and even CHOICE in there in a couple of places so I had to hit a key to continue)...  Actual command window pre & post backup dialogues if you will (do you actually need a dialogue that's external to the batch job even?).
The reflect window not showing is no never mind.. reflect puts enough in standard out you can see the progress in the command window..
You can see & interact with any program you have the batch run before or after the backup..
Or at least I can here.
For the life of me I can't begin to see why you need two batch files.. one generated by reflect but scheduled by you as you doesn't exit till everything you have happen after the backup completes, it doesn't even move on to the next command in sequence till the last completes, unless that's an external program which you've started by "Start program.exe" rather than running it in the normal way for a batch file... if you simply put notepad.exe in there notepad will open & the batch will wait till you close notepad before continuing or whatever other program you need to manipulate shares or connect a drive, or whatever.. you can even make following actions contingent on the exit status of that program..

The only catch I can see is if your personal account lacks admin, because reflect will need the run with highest privileges checkbox set in the scheduler (I think, I've not tried without or from a limited account).

I must be missing something pretty basic here because I'm not seeing an issue that can't be overcome or worked around fairly easily.
By jphughan - 4 April 2022 7:37 PM

Beardy - 4 April 2022 7:18 PM
The only catch I can see is if your personal account lacks admin, because reflect will need the run with highest privileges checkbox set in the scheduler (I think, I've not tried without or from a limited account).

I don't think that's correct in this case.  Reflect is capable of prompting for elevation if it wasn't invoked from an elevated context, and if the scheduled task is designed to run in a way that will allow interaction, then it would be possible to respond to that UAC prompt rather than needing to make sure Reflect runs "pre-elevated" as would be necessary for appropriate background execution.  Whether anything ELSE in the batch file might require execution from a context that's already elevated is of course a separate question.
By Beardy - 4 April 2022 8:00 PM

@jphughan You're probably correct, I did qualify that by saying I'd not tried it...

@DocDJ  I'm clearly missing something.. Sometimes it's easier to approach helping set up Reflect knowing exactly what the ultimate goal is, someone stating:
I'd like X to happen, followed by a backup, followed by Y & this sequence of backups... Different people might come up with various approaches, & some might prove easy..
On the other hand, trying to troubleshoot a specific snag a user hits, while trying to set up a certain approach can sometimes muddy the waters, particularly as it can result in suggestions which might solve that bit while breaking the ultimate aim.

I'd probably do better if I understood clearly what your aim is.. Maybe I'm just being a bit slow today.. it wouldn't be the first time. But we appear to be trying to solve two issues at once, what type of backup, & how to prompt for something unrelated to that, & I'm afraid at the moment I can't see the wood for the trees.
By DocDJ - 4 April 2022 11:43 PM

Beardy - 4 April 2022 8:00 PM
@jphughan You're probably correct, I did qualify that by saying I'd not tried it...

@DocDJ  I'm clearly missing something.. Sometimes it's easier to approach helping set up Reflect knowing exactly what the ultimate goal is, someone stating:
I'd like X to happen, followed by a backup, followed by Y & this sequence of backups... Different people might come up with various approaches, & some might prove easy..
On the other hand, trying to troubleshoot a specific snag a user hits, while trying to set up a certain approach can sometimes muddy the waters, particularly as it can result in suggestions which might solve that bit while breaking the ultimate aim.

I'd probably do better if I understood clearly what your aim is.. Maybe I'm just being a bit slow today.. it wouldn't be the first time. But we appear to be trying to solve two issues at once, what type of backup, & how to prompt for something unrelated to that, & I'm afraid at the moment I can't see the wood for the trees.

Thanks to all of you for your really valuable comments. Please let me try to explain in more detail what I want to happen. Fr script, below, read "batch file" (.BAT, .CMD, .VBS or .PS1), NOT a Macrium script, unless I say it is.
1. a scheduler (probably Windows' scheduler, from what I've learned here) starts a script, passing it a parameter (e.g.; Full)
2. the script prompts me (via whatever mechanism - a window, msgbox, etc -is available) to turn on the switch for the drive
3. the script tests (in a loop) for the drive to come online (I have a 5TB drive that takes about 30 sec's to become ready)
4. the script issues the Windows "start" command WITHOUT the "WAIT" option which causes a small window to popup that says "backup running, don't close this window". The purpose is to remind me not to reboot, not to turn off the drive, or do anything else that may interfere with the backup. This window stays onscreen until later but I want the SCRIPT window to disappear).
5. the script starts Macrium using the Windows "WAIT" option of the start command. The backup runs and my script is waiting for it to finish.
6. now the script pops up another window where the program running it tells me to turn off the switch, then loops until the drive is no longer online
7. the new window says "the backup is done.Close me." When I click, it tells Windows to locate the other window by its Window class and Window name,  then the script tells THAT window to close, then closes itself.. The Window class and name were defined by me (when I wrote those 2 popup programs). 

I could probably do all this with ONE program having the 2 popup windows. I tried to get Macrium to run these 2 programs as part of my before/after scripts, but the windows never appear (because Macrium runs in its own context, which is therefore forbidden (by Windows) to interact with my login session)

I would LIKE the script window not to be visible, but I've tried to get that to work and not had success. So I might need an actual program instead of a script, because a program does not have to have a window unless it pops one to communicate (or uses a msgbox).. 
PS. I have been using this technique in Acronis with before/after batch files to do it and it has kept me happy for a year. Now I want to do the EQUIVALENT with Macrium.
I hope this clears up what I'm trying to do. I have the individual pieces working. I just need to find the best way to combine them to do it with Macrium. I'm currently doing it with batch files and WIn32 .exe programs. As I said, I might need to do it all with a single large Win32 program.
.
By jphughan - 4 April 2022 11:57 PM

I can't speak to how Acronis works since I haven't used it in a long time, and judging by the fact that they're now rebranded their imaging product as a cyber protection suite with forcibly installed and not entirely defeatable general purpose malware protection, I have no intention of touching it.  But you've got the right idea.  If you were using PowerShell, you would be able to control the visibility of the script window.  The scheduled task can be configured to run an instance of PowerShell that provides parameters for calling the script file, specifying the backup type, and hiding its own window.  But I haven't looked into how to achieve that when using batch files.
By Beardy - 5 April 2022 2:34 AM

I wouldn't try hiding a batch window, you get some very odd behaviour & some most strange exit codes, can possibly be done, but if PS has means to hide itself it'd be the way to go, that or VBS, or something similar like autoscript, which runs in the system-tray unless you've got it popping up a window or message.

The basic functionality could be done in a batch, & it could have run minimised until Microsoft ripped NET SEND out of windows for the messages, then also deprecated or removed (unsure which) the message function they replaced it with...  now if I were trying with a batch I'd probably just have the command window open & interact with it directly.

I'd probably use AutoScript, or learn enough powershell to get the job done with the requirement for being unobtrusive though.

You might get a batch to work, but you'd need to schedule it (outside of Reflect) to run as yourself, the Windows scheduler can certainly run a batch within a given user's session, with which it can then interact, but it's awkward making it even run minimised (in that Microsoft's documented method to do so doesn't actually work, & it runs in an open window you have to manually minimise regardless), much less hidden.
By DocDJ - 5 April 2022 10:18 AM

jphughan - 4 April 2022 11:57 PM
I can't speak to how Acronis works since I haven't used it in a long time, and judging by the fact that they're now rebranded their imaging product as a cyber protection suite with forcibly installed and not entirely defeatable general purpose malware protection, I have no intention of touching it.  But you've got the right idea.  If you were using PowerShell, you would be able to control the visibility of the script window.  The scheduled task can be configured to run an instance of PowerShell that provides parameters for calling the script file, specifying the backup type, and hiding its own window.  But I haven't looked into how to achieve that when using batch files.

I'll have a look at PS, thanks.
By DocDJ - 5 April 2022 10:28 AM

Beardy - 5 April 2022 2:34 AM
I wouldn't try hiding a batch window, you get some very odd behaviour & some most strange exit codes, can possibly be done, but if PS has means to hide itself it'd be the way to go, that or VBS, or something similar like autoscript, which runs in the system-tray unless you've got it popping up a window or message.

The basic functionality could be done in a batch, & it could have run minimised until Microsoft ripped NET SEND out of windows for the messages, then also deprecated or removed (unsure which) the message function they replaced it with...  now if I were trying with a batch I'd probably just have the command window open & interact with it directly.

I'd probably use AutoScript, or learn enough powershell to get the job done with the requirement for being unobtrusive though.

You might get a batch to work, but you'd need to schedule it (outside of Reflect) to run as yourself, the Windows scheduler can certainly run a batch within a given user's session, with which it can then interact, but it's awkward making it even run minimised (in that Microsoft's documented method to do so doesn't actually work, & it runs in an open window you have to manually minimise regardless), much less hidden.

yes, batch is painful. The biggest problem with it is how to get the post-backup one to run and interact when the backup is done. My scenario seems to require one big batch file or program running in the background. And making a batch invisible seems to be a losing proposition. All these problems seems to be driving me toward Power Shell (as suggested by @Beardy or a Win32 program.scheduled outside of Macrium.