Skip to main content

Package installation is taking huge time even if few items

Hello Folks,

Last week we have had couple of deployments on different environments, and we have similar standard environments like Dev, Stage, Production and so on

Our DEV environment is something all of our developer uses and we have set up automated deployments on azure apps via octopus deployment and DEV environment generally has lot of deployments happening as and when developers finish their work

Stage and Production are something i have a control over, so i make sure that i do all branching and merging and deployment plans for it and do the deployments (of course we do it as a team :) ), and i never faced this issue in those environments.

Now from last week i was just observing that our developers are triggering the deployment and the DEV deployment was taking longtime, Our last deployment took around 6+ hours which is completely insane, So i start digging to see why it is happening and i also observed that gradually as the weeks passed, the deployment was taking longer time, so initially it was taking 3+ hours, later on 5+ hours, So i decided to dig and see what could be the issue 

Problem

Octopus deployment for sitecore items was taking huge time to finish the deployment around 5+ hours

Troubleshooting Steps

As our deployment was happening from octopus and octopus deployment was configured to do around 25+ steps, so one by one it executes different deployment steps and i was more interested in which step is actually talking long time? and very quickly i figured out that it is the Sitecore package installation which was consuming almost all the time

Now the next step was to find why it is taking so long, So i compared this same steps with other environment for which deployments were happening smoother and for the same package it was taking few mins to complete, now this was eyebrow raising.

So, It was time to observe the logs to find out what is going wrong because now it was evident that nothing is wrong from octopus side, but something is fishy on Sitecore package installation process

One thing was observed in the log that, after every item it tried to install it was showing below log 

and it was filled with these logs but they were warnings but that definitely pointed me to some extra search if there was something wrong with the index strategy, so went on to google to find something about it and i found few really helpful articles which were pointing into this direction 

https://sitecore.stackexchange.com/questions/478/sitecore-index-update-jobs-getting-stuck

https://community.sitecore.net/developers/f/8/t/7764#pi214filter=all&pi214scroll=false

1) These links have different scenarios but issue was same, so i went ahead and changed the strategy for my master index, to following (change from syncMaster to intervalAsyncMaster)

<strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />
</strategies>

2) Also changed CheckForThreshold in Sitecore.ContentSearch.DefaultConfiguration.config to false from true

3) This is where it hit me that, this CheckForThreshold flag is used for EventQueue and one of the thread was also talking about EventQueue size limit, and Sitecore recommends not to have more than 1000 items in that table for optimized performance, Also i have observed in my deployments that it gradually started taking time so in my back of the mind EventQueue was already started shaping that because of the table is growing, The deployments might be taking longer time depending on entries it had

 So i went in and hit the query to see the size of EventQueue table in core, master and web sql tables

and with my surprise it was huge and was having couple of thousands of records, i kept data only of last week and deleted all the records from it, and now there were only around 300+ records 

Now it was time to hit the deployment again, so i restarted my azure web apps after these changes, stopped on going deployment and once CM was up, i started the deployment again and i was waiting for that step to come where it installs the packages 

This time it was giving me same WARN logs but it took only and 1.5 hour to finish the whole process which was taking humongous 5+ hours  

Solution

It was definitely that event queue which was causing this package installations to go on and on and on, it is always best practice to configure Sitecore to have only EventQueue to certain days if really needed, so Sitecore can be configured to clean the queue automatically after few days

I hope this will also help someone who is running into the same issue.

Labels:

Comments

Post a Comment

Popular posts from this blog

Why SitecoreAI - Getting into the shoes of the customer how to select right CMS

Hi Team, Lately, I have been talking to lot of our customers / potential customers and having pre-sales demos where one question always comes is "Why Sitecore" ?  Now this question can be for any product which is out for sell. And as a technician I always get into product technical features, but at the same time as a pre-sales guy, it also makes me think, surely all competitive products have same features, so definitely answer to this is not in the technicalities.  If you step back and think, we are also a customer in our daily life and buy lot of things, what is that process we go through? When we buy, how can your customer decide if this is a right fit for you or not, why we select A over B? Is it price? is it service? Is it a brand? Is it about features? Is it about brand loyalty?  When it is a technical product, I am sure it cannot start with the technicalities of the product or selecting product itself, 100% not, I feel decision is always business strategy first and ...
1 comment
Read more

Hell of sitecore aliases pipeline breaking the site with 500 error

Hello Friends, I belive this blog post is very important for everyone because, It has some very serious effect on working of your headless website, i will share my experience what we faced and how we resolved it Issue we started facing Our site started giving "Key cannot be null or empty" with YSOD like following  Side affect Because of this 500 error, Our site pages were showing 500 custom error page intermittently and our MAU (Monthly Active User) drop rate increased. Sitecore KB There is already Sitecore KB article talking about this error but the patch which is provided on this link is confusing as well as very huge and it could bring other issues along with it as that upgrade patch also has lot of other things too which i did not want to introduce in our stable CMS. Known Issues - Retrieving the child items of resource items is not thread-safe Observation Though the surfaced exception was looking similar and giving same error and behavior given on this article, We looked...
Post a Comment
Read more

401.1 Unauthorized with windows authentication error code 0xc000006d

How many of you have faced this hosting issue when you do everything what it takes to run the site with windows authentication but still you are getting the same error again and again? If you think you also have faced the same issue and you tired of reading MSDN KBs for it and still have not found the issue (If KB has solved the issue, well and good, if not you can try this trick),Please Read below Typical scenario In typical hosting with IIS, i did every possible things like enabling windows authentication, changing it in web.config, configuring connection pool, authorization rules, it asks me for window authentication login and despite of entering correct credentials it always fails and keeps on asking for login, and when pressed cancel it gives 401.1 with 0xc000006d error code Solution (Which worked for me at-least after trying for almost 6-9 hrs) You need to change the Loop Back Check in registry so that it allows the host names which you are giving in url are allowed and au...
2 comments
Read more