The Zuora integration experienced intermittent issues with calculations in December 2014. This was particularly noticeable on December 1st and later in the month, leading up to December 26th, 2014.
On December 1st, 2014, there was a planned hold on all batch calculations (this includes regular tax calculations from Zuora). This planned hold lasted from 6 - 6 PST. The reason for the hold was that December 1st was Cyber Monday. After the feedback from customers, especially customers using Zuora, we do not plan to do anything similar to this in the future.
Improvements to the AvaTax infrastructure lead to intermittent failures to calculate with Zuora. This was caused by the new servers not having the necessary properties to process Calc API batch calculations calls. This was resolved on December 26, 2014 at noon PST.
Below is the root cause analysis from the Avalara Site Reliability Engineering team.
From the afternoon of 11/24 through 12/22 at 11:13 AM, loading old batches and saving new batches was partially impacted.
Since the 8 new web servers were added on 11/24, Admin Console connections to them that tried to load an old batch or create a new one failed. The 6 existing servers worked fine for those actions. All other functionality worked on all of the servers.
In our TUK datacenter, there are two domains. The long term goal is to just have one because having two is overly complex, leads to incidents and has more problems than benefits. The 8 new web servers were created in the AVA domain instead of the WEB domain which is where the old web servers are but is also going to be retired.
The Calc website runs as the default user (ApplicationPoolIdentity) which can access network resources like the file share for batches just fine. The problem is that it cannot access cross-domain network resources and the batch file share is in the other domain. This causes the new web servers to error when trying to read or write from the batch file share when running as the default user. The fix will be in SRE-9273 which is to update the Calc website to run as a newly created service account. There is domain trust between the WEB and AVA domains allowing this to work and was verified during our troubleshooting.