Notice Details

---

7/16/2019 9:05:00 PM - Service Slow Down

RESOLVED

Yesterday and today we experienced issues that affected many, if not all, of our customers. We greatly regret having these problems and are sorry for the stress, extra work and pressure it added upon you, your teams and your customers. Keeping our systems running fast and fully available is one of our most important goals and for the past two days we failed to deliver.

We are taking several steps to improve ourselves and to ensure the same issues don’t recur. These steps include:
• A better internal escalation and communication plan to provide you with proactive notices and issue handling.
• A full review of our infrastructure to improve how we maintain and operate our systems.
• An investigation of processes that consume a lot of system resources to find optimizations that reduce overhead.

WHAT HAPPENED
On Monday, July 15 starting around 1:46 PM PST and lasting until approximately 3:05 PM PST we experienced a slow-down in our production database systems.
On Tuesday, July 16 starting around 6:30 AM PST and lasting until approximately 9:21 AM PST we experienced a similar slow-down in our production database systems.

During both of these events some customers reported the inability to search for forms and generate forms. The end of both of these events was due to fixes we made to our systems that were discovered at the time.


ROOT CAUSE
Although we performed an update of some REST services and two turnkey applications on Friday, July 12, the update was not the primary cause of the issues we experienced, though it may have contributed. The slow-down was generally caused by a few factors.

1. Inefficient maintenance processes that run daily
2. Inefficient form search and form generation queries
3. High demand of our systems

The inefficient processes, searches and queries have been in operation on our systems for multiple years each, and our systems have generally run fine. However, we’re growing and with an increase in demand of our systems, especially on a new application that uses the form search and form generation routines more frequently than expected, our database system resources were being consumed too fast. Add in higher demand during peak hours of usage, and the combination overwhelmed our systems.

On Monday we narrowed down the issue to a key process that maintains form records for customers by running every 10 minutes. This process was taking upwards of 5-7 minutes to run. We re-wrote that process and now it runs in 2 seconds – a huge improvement that greatly freed up system resources.

On Tuesday we narrowed the problem down to the form search process as the key bottleneck in the system. When we removed a single feature from that process the entire database system cleared up and went back to normal. Later, we rebuilt that feature to run 99% faster and our database systems have been operating normally ever since.

OUR COMMITMENT TO YOU
We are continuing to closely monitor our systems and to refactor processes that run slowly. Some of our updates will take several days to a few weeks to implement that will further improve speeds. Our infrastructure project will take a few months to implement changes that can contribute to higher availability and/or to dynamically scale better to meet peak loads.

Again, we are sorry for the issues, the stress and the frustration you, your team members and your customers felt as a result of our issues. We are committed to doing better and living up to our name of being both quick and efficient at all times.

Thank you for understanding and choosing to be our partner.

Last Updated 7/16/2019 11:09:16 PM
Last User SysAdmin

Back