Skip to main content

Package installation is taking huge time even if few items

Hello Folks,

Last week we have had couple of deployments on different environments, and we have similar standard environments like Dev, Stage, Production and so on

Our DEV environment is something all of our developer uses and we have set up automated deployments on azure apps via octopus deployment and DEV environment generally has lot of deployments happening as and when developers finish their work

Stage and Production are something i have a control over, so i make sure that i do all branching and merging and deployment plans for it and do the deployments (of course we do it as a team :) ), and i never faced this issue in those environments.

Now from last week i was just observing that our developers are triggering the deployment and the DEV deployment was taking longtime, Our last deployment took around 6+ hours which is completely insane, So i start digging to see why it is happening and i also observed that gradually as the weeks passed, the deployment was taking longer time, so initially it was taking 3+ hours, later on 5+ hours, So i decided to dig and see what could be the issue 

Problem

Octopus deployment for sitecore items was taking huge time to finish the deployment around 5+ hours

Troubleshooting Steps

As our deployment was happening from octopus and octopus deployment was configured to do around 25+ steps, so one by one it executes different deployment steps and i was more interested in which step is actually talking long time? and very quickly i figured out that it is the Sitecore package installation which was consuming almost all the time

Now the next step was to find why it is taking so long, So i compared this same steps with other environment for which deployments were happening smoother and for the same package it was taking few mins to complete, now this was eyebrow raising.

So, It was time to observe the logs to find out what is going wrong because now it was evident that nothing is wrong from octopus side, but something is fishy on Sitecore package installation process

One thing was observed in the log that, after every item it tried to install it was showing below log 

and it was filled with these logs but they were warnings but that definitely pointed me to some extra search if there was something wrong with the index strategy, so went on to google to find something about it and i found few really helpful articles which were pointing into this direction 

https://sitecore.stackexchange.com/questions/478/sitecore-index-update-jobs-getting-stuck

https://community.sitecore.net/developers/f/8/t/7764#pi214filter=all&pi214scroll=false

1) These links have different scenarios but issue was same, so i went ahead and changed the strategy for my master index, to following (change from syncMaster to intervalAsyncMaster)

<strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />
</strategies>

2) Also changed CheckForThreshold in Sitecore.ContentSearch.DefaultConfiguration.config to false from true

3) This is where it hit me that, this CheckForThreshold flag is used for EventQueue and one of the thread was also talking about EventQueue size limit, and Sitecore recommends not to have more than 1000 items in that table for optimized performance, Also i have observed in my deployments that it gradually started taking time so in my back of the mind EventQueue was already started shaping that because of the table is growing, The deployments might be taking longer time depending on entries it had

 So i went in and hit the query to see the size of EventQueue table in core, master and web sql tables

and with my surprise it was huge and was having couple of thousands of records, i kept data only of last week and deleted all the records from it, and now there were only around 300+ records 

Now it was time to hit the deployment again, so i restarted my azure web apps after these changes, stopped on going deployment and once CM was up, i started the deployment again and i was waiting for that step to come where it installs the packages 

This time it was giving me same WARN logs but it took only and 1.5 hour to finish the whole process which was taking humongous 5+ hours  

Solution

It was definitely that event queue which was causing this package installations to go on and on and on, it is always best practice to configure Sitecore to have only EventQueue to certain days if really needed, so Sitecore can be configured to clean the queue automatically after few days

I hope this will also help someone who is running into the same issue.

Labels:

Comments

Post a Comment

Popular posts from this blog

401.1 Unauthorized with windows authentication error code 0xc000006d

How many of you have faced this hosting issue when you do everything what it takes to run the site with windows authentication but still you are getting the same error again and again? If you think you also have faced the same issue and you tired of reading MSDN KBs for it and still have not found the issue (If KB has solved the issue, well and good, if not you can try this trick),Please Read below Typical scenario In typical hosting with IIS, i did every possible things like enabling windows authentication, changing it in web.config, configuring connection pool, authorization rules, it asks me for window authentication login and despite of entering correct credentials it always fails and keeps on asking for login, and when pressed cancel it gives 401.1 with 0xc000006d error code Solution (Which worked for me at-least after trying for almost 6-9 hrs) You need to change the Loop Back Check in registry so that it allows the host names which you are giving in url are allowed and au...
2 comments
Read more

One or more exceptions occurred while processing the subscribers to the 'item:creating' event

I was recently installing the packages from one of the QA environment to my local Sitecore instance, "Media library package" to be precise, And it started giving me this below error One or more exceptions occurred while processing the subscribers to the 'item:creating' event Looking at the sitecore logs, it gave me more info on the context and the actual inner exception was following Solution: 'Name' should consist only of letters, digits, dashes or underscore Now it was evident that some of my file names were violating the naming rule, I could see in the log just before exception from where the installer stopped creating items, and that file name had round braces "(" and ")" at the end of it, with my surprise I was able to create the item with those name in the content tree, but below was the solution for it, Solution I am using SC 9.0.1 and in that Go to Sitecore.Marketing.config file residing in "App_Config\Sitecore\Mar...
9 comments
Read more

Zero to Hero - A real life RCA of exact issue in Sitecore Managed Cloud environment

Hello All, The purpose of today's post is to share a real life burning and escalated scenario which was new to me and how did I approach it and how big the escalations were and what was the outcome Sitecore's goodwill was at stack not because Sitecore is not capable of handling it but just because our environment was Sitecore Managed Cloud, and any issue that comes if its infra, back end code, front end code will be first pointed as Sitecore issue and that is where our consultancy and experience will play a role to prove that it is not Sitecore issue.  Issue we faced Out of the blue our site started giving "504 Gateway Time-out", and it was reported that almost everyone is getting this error, but when we used to browse the site, everything looked good and never 504. 504 Gateway Time-out error tells that, That the request went to Content Delivery servers of Sitecore from gateway, but gateway did not get response in time from those CDs and hence it gave time out error. ...
2 comments
Read more