What is sitecore exception of "alarm clock event subscriber"

Hi Team,

Today i will share one of the exception we started seeing in one of our CDs, We all like to keep our logs clean right, any exception can be symptoms of something going wrong

With my surprise, i was seeing following which was being continuously logged

Problem

The exception is

"ERROR Exception in alarm clock event subscriber"

It would have not caught my eyes, But because i was troubleshooting something else and anything in the logs looked suspicious to me and i researched it further and what happens is below

Cause

Sitecore documentation has explained very clear working of what goes behind the scene when Tracker is trying to submit a session to xConnect and what if xConnect is down

Refer : Sitecore documentation

"When the tracker submits a session to xConnect, but xConnect is temporarily unavailable, the tracker classifies this as a transient error (meaning it assumes xConnect becomes available later). The tracker adds the session to a submit queue that, by default, serializes the session to disk. The submit queue service later re-submits the session to xConnect at configurable intervals"

So that means if xConnect is unavailable there is some mechanism where the files gets stored on a file system,  which can be later picked up by the system when xConnect becomes available

Now in this process if xConnect becomes unavailable for sometime and it could be possible that those file which were created on file system by tracker might get stale or corrupt and that is from where the above exception takes birth.

Solution

First of all i wanted to check where on the file system the "Submit Queue" is being created and wanted to double check do i have any files there

and where the tracker will create a file on the disk, that path is mentioned in the \App_Config\Sitecore\Marketing.Tracking\Sitecore.Analytics.Tracking.config

and "dataFolder" is "sc:variable" and can be found in sitecore.config file from where you can get the actual path 

It pointed me to App_Data/Submit Queue folder on my CD server, and because i was getting the exception on my CD0, i went to that path and i saw following files there in queue waiting to be processed

i saw files from 2020 and after further research i took back up of those files and removed those file from the queue, because they were just there and not processed due to some reason

after doing this i went to my CD0 log stream, and i was happy to see that there was no exception and it worked !!!

Conclusion : We all want to keep our logs clean, so some exceptions might not be harming from the processing or performance point of view but its good to get it cleaned, you could also set some automation where if there are files older than some months, you can clean that up to keep your logs clean from these exceptions