Very Strange Scheduling Anomaly


Author
Message
Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
Greetings! I'm dealing with a very strange phenomena. First, some history on machine configuration... I just switched from using a W10 Enterprise LTSC Edition (20H2) to a 21H2 edition of the same OS, and, at the same time, switched from an Intel i7 Haswell (4th gen, 4-core, 8-thread, RAM=12gB) to an AMD Ryzen 7 3700X (8-core, 16-thread, RAM=32gB). Moved the REFLECT license from the old to the new machine. It's basically the same build, same software, etc. Reflect is running a fairly basic GFS configuration as well as an IntraDaily HOURLY Incremental from 9am 'til 8pm into the same backup chain.  Using the Macrium Scheduler, tried the Windows Scheduler with same result...

Observations...
1. Perfect operation prior to System switch.
2. No issues with GFS-based scheduled backups (probably not enough to see the issue).
3. IntraDaily scheduled backups run haphazardly some of the time. Sometimes they'll start but not finish for many (20+) minutes, and the operation LOG says they actually started 20+ minutes after the scheduler LOG says they did. The LOG says they ran normally except for the start time which was 20+ minutes past the logged scheduled start time. The 20+ minute time difference varies from a few minutes to 2-1/2 hours later. The questionable operations run as expected once they actually start. During that delayed operation time, any other REFLECT operation seems to never start or gets hung waiting for the delayed operation to finish.

I've been looking through the Event System looking for System anomalies but nothing has jumped out as of yet. It's truly a random thing... some days it runs fine, other days it messes up almost continuously. I even had 1-day where the delayed start anomaly was exactly the same delay time for many consecutive IntraDaily Incrementals... almost as though there was some fixed delay being offered the task before the operation actually commenced. The scheduler LOG would show the task starting on the hour and finishing up 21-min later... the actual task LOG would show it starting up 21-min after the hour and finishing up normally in about a minute.

If anyone would like to jump in... feel free w00t

Edited 15 May 2022 5:55 PM by Froggie
Drac144
Drac144
Guru
Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)
Group: Forum Members
Posts: 900, Visits: 3.2K
It does sound like the issue is with Reflect.  The scheduler started the process at the correct time and Reflect did SOMETHING for 20 minutes before actually running the task.  Have you looked at the specific steps in the scheduler history and the scheduled task definition (properties) screens to verify you have not specified some condition that would be causing the schedule to delay sometimes (like only run when not on battery - this is just an EXAMPLE of something that could be intermittent). 
Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
I've looked through the Event mgr for any System thing which could delay Reflect once it started. nothing stands out.  I have seen where I've tried to manually start a version of my IntraDaily definition and it never made it through the initial disk scan and move on to the PROMPT menu that I have set up for it to do (to add comments)... this is why I think the issue may be System related (maybe the disk scan is hanging for some reason?).

I just turned on any kind of notification and/or LOG associated with the IntraDaily setup (some stuff is off by DEFAULT) and we'll see what that produces.  The puzzling thing is that the change in operation definitely occurred when the hardware changed.  Other than the computer itself (which is now a barebones Alienware hardware collection), the main OS disk is now a NVME SSD (Disk #3 in the Reflect scan rather than Disk #1 as in the previous hardware config) rather than a SATA III SSD from before... all the other disks are the same (SATA SSDs).

Edited 15 May 2022 10:37 PM by Froggie
Andrew
Andrew
Macrium Representative
Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)Macrium Representative (153 reputation)
Group: Moderators
Posts: 90, Visits: 960
Hi Froggie,

After you've found out whether turning on all the other IntraDaily logging / notifications gives any further information, can you please create a support ticket at https://www.macrium.com/support so we can investigate this issue further?

Kind Regards

Andrew

Macrium Development

Next Webinar

See our reviews on

Trustpilot Logo
Trustpilot Stars


Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
Froggie - 15 May 2022 5:51 PM
Observations...
2. No issues with GFS-based scheduled backups (probably not enough to see the issue).


Well, that turns out not to be the case.  A single daily GFS Differential (different schedule than the IntraDaily) had a 19-min delay just like the IntraDaily experience.


Edited 16 May 2022 12:52 PM by Froggie
L. W. "Dan" Danz
L. W. "Dan" Danz
Guru
Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)Guru (1.3K reputation)
Group: Forum Members
Posts: 576, Visits: 6.1K
An off-the-wall possibility, but submitted for your thought process to consider:   Could it be that two jobs are started in the same minute, and the second one is stacked and waits for the first to finish?  


 L W "Dan" Danz, Overland Park KS 
MR v8.0.6786+    Win10.21H2-19044.1706+    Win11.21H2-22000.778+




Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
Since this happens 99+% of the time during scheduled jobs, I've checked both the Reflect queue and the Windows queue for possible duplicates, etc. and found nothing unusual.  If that was really occurring, it would be unusual since none of the jobs I run, except for my FULL every 4-weeks, are that long in their duration... especially that 2.5-hr delay that occurred recently.  Most of my scheduled tasks are at the less than a minute level up to maybe 2.5-min for an occasional Differential.

The other strange observation, in relation to the manual operation mentioned above, is that I start a manual Prompt-based Incremental (usually based on my IntraDaily definition) and no Prompt screen appears (at all when it has happened before).  I then jump into the "Task Mgr" (to see what's going on) and as soon as the TM window appears, the Prompt window shows up... like clockwork.  This has happened a coupla times.  It seems that bringing up TM breaks the logjam holding up the Prompt Incremental... which may be the same logjam causing the scheduled tasks to have their anomaly.

Edited 16 May 2022 3:19 PM by Froggie
Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
Andrew - 16 May 2022 10:46 AM
Hi Froggie,

After you've found out whether turning on all the other IntraDaily logging / notifications gives any further information, can you please create a support ticket at https://www.macrium.com/support so we can investigate this issue further?

Andrew, Ticket #209114 has been issued.  Have been running for 24-hours without issue, all DEFAULT IntraDaily options have been disabled.

Froggie
Froggie
Macrium Hero
Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)Macrium Hero (2.4K reputation)
Group: Forum Members
Posts: 1.4K, Visits: 14K
Starting on 22May2022 I will turn all the IntraDaily DEFAULT options back on and see if the issue resumes... no incidents since the turn off on 15May2022.

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Login

Explore
Messages
Mentions
Search