UPDATE: There was another outage on Friday, March 24 for a few hours. Another response from Intuit on 3/25/11 is posted below.
Intuit has issued the following in response to the recent service outage affecting QuickBooks Online, QuickBooks Merchant Services, QuickBooks Online Payroll and other Intuit services which were down on March 21 and 22, 2011.
A Letter from Kiran Patel- 3/24/11
To our customers,
I apologize for the frustrations caused by our service outages and for letting you down this week. We didn’t meet our own high standards for dependability and customer service. There is no excuse for this performance.
We want to help those affected by this outage. If you need further assistance, please contact us at the following numbers between 8:00 a.m. and 5:00 p.m. Pacific time, Monday through Friday:
- QuickBooks Online: 800-286-6800
- QuickBooks Online Payroll: 888-537-7794
- Intuit Payments Solutions: 800-558-9558
What Happened 
Over the course of two days we had service outages due to two separate errors made during scheduled maintenance.
In both cases we made network configuration changes that inadvertently blocked customer access to a portion of our servers. The initial disruptions lasted just a few minutes while we undid the change. But in both cases, restoring full service took longer than we would have liked. After fixing the problem, a surge in traffic overloaded the servers when we restored connectivity. We then decided to restore service at a purposefully measured pace to closely monitor the system and prevent another overload.
Be assured that there was no security breach or attack on our servers. At this time, we do not believe there is any damage or loss to your data.
Our commitment to you
We understand the important role our services play in your business. And we take that responsibility very seriously.
Again, we apologize for the disruption to our service and are working to improve the resiliency of our systems.
We appreciate your business and your patience. We will work hard to regain your trust.
Sincerely,
Kiran Patel
Executive Vice President and General Manager
Intuit Small Business Group
*********************************
UPDATE: Intuit Response on 03/25/11
This was a disappointing week, both for you and for Intuit. And yes, that may be an understatement.
You’re probably aware of how several of our online small business products weren’t available when you needed them. For many of you, it goes beyond disappointment and beyond frustration. You’re wondering how and why this happens, and what we’re doing to fix it – all legitimate questions.
Let’s start with an apology. Quite simply, this shouldn’t happen and there’s no excuse for this performance. We didn’t meet our own high standards for dependability and customer service. If your business was affected by these outages, we want to do right by you. If you were affected, call:
- QuickBooks Online: 800-286-6800
- QuickBooks Online Payroll: 888-537-7794
- Intuit Payments Solutions: 800-558-9558
What happened
On Monday and Tuesday, March 22-23, we had service outages triggered by separate errors made during scheduled maintenance. On Friday, March 25, we lost those services again. We’re still working to identify the cause of that last disruption.
In the first two instances, blame human error. We changed our network configuration and inadvertently blocked customer access to a portion of our servers. Those initial disruptions lasted just a few minutes while we undid the change. But in both cases, restoring full service took longer than we would have liked. After fixing the problem, a surge in traffic overloaded the servers when we restored connectivity. We then decided to restore service at a purposefully measured pace to closely monitor the system and prevent another overload.
On Friday, the same services went down. We restored them much more quickly, but are still trying to determine why that happened. In all cases, we know there was no security breach or attack on our servers. And we know that your data was not lost or damaged.
What we’re doing
Explanations and apologies don’t mean much if we don’t fix the underlying problems. With our services restored, we’re going to get to the bottom of this and solve it.
Here’s what we’re doing:
- We’re building resiliency and redundancy into our new products and data centers, so we can keep your services up and running in the event of a failure with minimal downtime.
- We’re improving the way we operate our data centers to reduce the risk of human error.
This is a work in progress. When it goes right, no one notices. When it goes wrong, it’s obvious. We’re committed to doing this as quickly as possible and, I hope, with no disruption to you and your business. If that happens, we’ll work to restore services quickly while staying in touch with you along the way.
As I said previously, we understand the important role our services play in your business. And we take that responsibility very seriously. While we do apologize, we want to do much more. We’d rather provide you the great service you expect. And we will. We appreciate your business and your patience.
Sincerely,
Kiran Patel
Executive Vice President and General Manager
Intuit Small Business Group
What good is an apology?
can i take it to the bank and cash it in for the thousands of dollars in business
Intuit’s incompetence has cost my business?
They put me on hold for hours, and finally transferred me to a supervisor who said he would give me a case number to submit my complaint with and hang up on me. I have never experienced more incompetence in my career.
I will be submitting a letter of complaint to headquarters requesting compensation next.
I haven’t been able to log onto Merchant Services for most of the day today. I keep getting the message “Try again in 2 minutes” (or 4 minutes). Looks like Intuit has not solved this problem. Intuit should take a lesson from Quark. Quark was the only game in town for page layout for many years, so they provided substandard service and got away with it. Until a strong competitor (Adobe) stepped in and destroyed them with a superior product. QuickBooks does the job (so did Quark Express), but someone like Apple could develop an app that blows it away if they don’t improve.
“When it goes right, no one notices. When it goes wrong, it’s obvious.”
I’m afraid you’re mistaken. For many years I was a loud and loyal proponent of QuickBooks Online Edition. As a long-time Certified ProAdvisor I’ve turned several clients onto QBOE and sung its praises far and wide. Because it worked, because I and my clients got excellent support and because the online product grew into a nearly-elegant solution for many of my micro-biz clients. No more.
Your poor performance is a symptom of what’s wrong at the top. Since the Intuit focus shifted away from solid products and spectacular customer support toward frantic growth and acquisitions, you’ve lost your focus on the market that made you “great,” namely small businesses and your network of dedicated professionals. You’ve made choices about what to do with your resources, and those choices have directly resulted in the poor showing you’ve made in recent months…proximate causes notwithstanding.
You’ve lost so much credibility I can no longer advocate your products. I support them when the client insists, but my days of “selling Intuit” are over.
I just had another outage yesterday, while I’m in the middle of researching the info to respond to an IRS notice. Did the incident on the 21 & 22 repeted yesterday or another issue this time? The service was back before the day ended and was able to call the IRS before they close their office. A redundant service wil really make sense or make it convenient for us to download the files so we can build our own data back-ups. At least, the research issue will be taken cared of, not the SaaS issue.
We are always talking up SaaS computing & Intuit Merchant Services to our clients and prospects. It’s embarrassing and we lose credibility when these outages occur.
Redundant servers in a different location seems to make alot of sense.
These outages are no longer an exception to the rule. This statement speaks volumes… “After fixing the problem, a surge in traffic overloaded the servers when we restored connectivity.” This tells us how the infrastructure is operated… there are two schools of thought on redundancy; one school creates a mission critical environment that is intended to survive not just the known, but the unknown risks – like an artificial heart where failure is not an option. The second school relies on a term called “uptime” which balances the cost of redundancy against the cost of being down by having higher capacity utilization, fewer failsafes and less redundancy – cell phone service is a good example of the second school.
This is not an indictment of QuickBooks Online, I’m just saying that if your mind is in the first school, it’s time to change your expectations. Intuit should offer a “pro” online version that runs on a mission critical infrastructure.
Mark, I agree wholeheartedly with what you said. I did want to acknowledge however that this is the best response Intuit has given customers thus far. I am hopeful that it is a step in the right direction.
What we all expect is along the lines of Wayne’s comment.
Cindy Steed
These outages happen way too frequently. I am reluctant to recommend any online Intuit service especially Intuit Payment Solutions to my clients.
The only thing that will restore my trust is the establishment of redundant servers located in a different time zone with instant auto switch-over. Until then I must advise potential subscribers to select Intuit online services only if they can tolerate a 2-day outage.
Keith -This is Laura from Intuit. We worked together a few years back. Please do call the numbers Kiran listed to let us help you. I understand that this caused a significant disruption. For that, I apologize. Sincerely!
This is not acceptable. I was in the middle of a session with a client in West Seattle that had to end prematurely which means I lost money. At a minimum, Intuit should credit everyone’s account for one week’s service.
Cindy, that kind of an apology is meaningless unless backed up with steps so it doesn’t happen again. It’s like my kids being forced to say they are sorry then they start fighting again. True repentance starts with being sorry then making sure it doesn’t happen again. And trust me, it will. This is their track record.
This is the type of communication and apology I expect to see from Intuit. I am happy to see that they are taking responsibility and keeping us informed. Now we need to see improvement in the form of outages being almost non-existent.
A competent network engineering team should have recognized that there would have been a surge in network traffic at the second the service was restored and planned accordingly.