Skip to main content

Package installation is taking huge time even if few items

Hello Folks,

Last week we have had couple of deployments on different environments, and we have similar standard environments like Dev, Stage, Production and so on

Our DEV environment is something all of our developer uses and we have set up automated deployments on azure apps via octopus deployment and DEV environment generally has lot of deployments happening as and when developers finish their work

Stage and Production are something i have a control over, so i make sure that i do all branching and merging and deployment plans for it and do the deployments (of course we do it as a team :) ), and i never faced this issue in those environments.

Now from last week i was just observing that our developers are triggering the deployment and the DEV deployment was taking longtime, Our last deployment took around 6+ hours which is completely insane, So i start digging to see why it is happening and i also observed that gradually as the weeks passed, the deployment was taking longer time, so initially it was taking 3+ hours, later on 5+ hours, So i decided to dig and see what could be the issue 

Problem

Octopus deployment for sitecore items was taking huge time to finish the deployment around 5+ hours

Troubleshooting Steps

As our deployment was happening from octopus and octopus deployment was configured to do around 25+ steps, so one by one it executes different deployment steps and i was more interested in which step is actually talking long time? and very quickly i figured out that it is the Sitecore package installation which was consuming almost all the time

Now the next step was to find why it is taking so long, So i compared this same steps with other environment for which deployments were happening smoother and for the same package it was taking few mins to complete, now this was eyebrow raising.

So, It was time to observe the logs to find out what is going wrong because now it was evident that nothing is wrong from octopus side, but something is fishy on Sitecore package installation process

One thing was observed in the log that, after every item it tried to install it was showing below log 

and it was filled with these logs but they were warnings but that definitely pointed me to some extra search if there was something wrong with the index strategy, so went on to google to find something about it and i found few really helpful articles which were pointing into this direction 

https://sitecore.stackexchange.com/questions/478/sitecore-index-update-jobs-getting-stuck

https://community.sitecore.net/developers/f/8/t/7764#pi214filter=all&pi214scroll=false

1) These links have different scenarios but issue was same, so i went ahead and changed the strategy for my master index, to following (change from syncMaster to intervalAsyncMaster)

<strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />
</strategies>

2) Also changed CheckForThreshold in Sitecore.ContentSearch.DefaultConfiguration.config to false from true

3) This is where it hit me that, this CheckForThreshold flag is used for EventQueue and one of the thread was also talking about EventQueue size limit, and Sitecore recommends not to have more than 1000 items in that table for optimized performance, Also i have observed in my deployments that it gradually started taking time so in my back of the mind EventQueue was already started shaping that because of the table is growing, The deployments might be taking longer time depending on entries it had

 So i went in and hit the query to see the size of EventQueue table in core, master and web sql tables

and with my surprise it was huge and was having couple of thousands of records, i kept data only of last week and deleted all the records from it, and now there were only around 300+ records 

Now it was time to hit the deployment again, so i restarted my azure web apps after these changes, stopped on going deployment and once CM was up, i started the deployment again and i was waiting for that step to come where it installs the packages 

This time it was giving me same WARN logs but it took only and 1.5 hour to finish the whole process which was taking humongous 5+ hours  

Solution

It was definitely that event queue which was causing this package installations to go on and on and on, it is always best practice to configure Sitecore to have only EventQueue to certain days if really needed, so Sitecore can be configured to clean the queue automatically after few days

I hope this will also help someone who is running into the same issue.

Labels:

Comments

Post a Comment

Popular posts from this blog

Sitecore Technical Workshops - Top FAQs customers asked on XM Cloud

Hi Readers, I want to talk to you about interesting things which we have been doing which is "Technical Workshops" for our customers, so here goes the scenarios. So, we have been doing multiple types of technical workshops.  1) Training customer and their Sitecore technical team on latest and the greatest technologies like XM Cloud & Another composable stack and try enabling them for new Sitecore tech stack. 2) Customers / Potential Customers have their agenda of existing pain points, and we take a workshop on topics around them with best practices etc. little on new technologies, so they also know the future. Basically, we prepare custom targeted presentations & demos for individual workshops, and make sure it helps them answer their questions and they get insights of where Sitecore eco systems has to offer from their versatile toolset and try to keep them up to date with it. So, Purpose of this blog is, because in all these customer & their technical team's
Post a Comment
Read more

An error occurred while receiving the HTTP response to This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.

You have noticed many times that everything was working fine and suddenly the below error starts coming and you find no way to work it out An error occurred while receiving the HTTP response to This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. The reason for this is the receiving size of WCF service is smaller then the data which is coming from service It was working before because it was small,So you will have to try to increase the receiving setting in your end point,Possible settings can be following maxStringContentLength="2147483647" maxReceivedMessageSize="2147483647" maxBufferSize="2147483647" maxArrayLength="2147483647" That would definately help you!!!
1 comment
Read more

Set up leprechaun code generation with Sitecore XM Cloud Starterkit

Hi Sitecorians, It has been amazing learning year so far and with the change in technology and shift of the focus on frontend frameworks and composable products, it has been market demand to keep learning and exploring new things. Reasons behind this blog Today's topic is something that was in my draft from April-May, and I always thought that there is already a good documentation out there for  Leprechaun  and a blog post is not needed, Until I realized that there was so many of us facing same kind of issues and same kind of problems and spending same amount of time, That is where I thought, if I could write something which can reduce that repetitive troubleshooting time, That would really help the community. 1)  In a project environment, if we get into some configuration issues, we resolve them, we make sure we are not blocked and continue, but if you think same issue, same step and same scenario will come to other people, so if we can draft it online, it will help other people 2
Post a Comment
Read more