At about 7:26 AM to 7:30 AM this morning, one of our datacenters lost network connectivity.

We are investigating with the provider as to the cause and will identify any opportunities for us to prevent it in the future. While the downtime was short (four minutes), we are notifying users because it was affected a majority a minority of our Phoenix-based users.

The data center is located in Tempe, AZ and our systems team is in contact with them. At this time, they confirm full network connectivity has been restored.

—
Updates:

7/7/2010 10:11 AM – We have determined the impact of this outage was isolated to only between the data center and the Cox network in the Phoenix, Arizona region. As such, a very small number of users may have been affected.

Today, we discovered an issue where some applications are experience higher than normal memory usage. We will be selectively rebooting various front-end servers until we resolve the underlying issue causing more memory usage. (High memory usage could severely impact performance.)

We use multiple applications on our servers so if one app behaves abnormally, others may be affected. To prevent adverse effects, our Systems team will identify troublesome servers and reboot them to release the memory. They’re super smart and dedicated and will be doing this in a way to minimize impact to you.

During the process of our server reboots, users will be logged out from their active Infusionsoft sessions and will need to log in again. From that point, there may be a short delay from being logged out to logging in again.

We’re aware of these issue as we are initiating the reboots. There is no need to contact support for this matter. Our Systems team will continue to monitor our servers and application performance and will be making improvements to keep things running smoothly.

If you are writing lengthy emails in your templates, we recommend to save a copy on your computer locally and to save frequently in the application until this issue is resolved. We’ll post updates on the blog post here so you can check back to know when we determine this to be fully resolved.

Thanks for your patience and understanding. Updates will be posted below.

—

UPDATE 5/14/10 1:31 PM: *RESOLVED* Our Systems team has identified the culprit and isolated the performance to a very small group of applications. As such, we will not be rebooting server today. Enjoy your awesome uptime and performance.

Today at 1:10PM, we became aware of a number of users reporting unexpected slowness and performance when sending emails. This issue is of a critical nature and our systems team is working to move all backend email systems to a separate system.

We believe the scope of this issue impacts all users, but we haven’t confirmed that. You may observe very slow email sending performance at this time while we investigate this and fix it.

This post will be updated with the latest details as we have them. Thank you for your patience and sorry for any inconvenience this may have caused.

—
UPDATES:

1:43 PM — We’re working with our email system provider to get this fixed. At this time it appears this impacts all customers and emails may not be sending properly. The result you might observe is an “ERROR” for the status.

3:36 PM — We have identified the issue and narrowed it down to a performance bug in our MTAs. Basically the Infusionsoft application was attempting to send mails to an overloaded mail server. We are assessing all options to restore reliable email delivery. Right now we are using our secondary (failsafe) email system which emails are being sent, but slower than normal. You do not need to re-queue new batches (and we don’t recommend that, either). We plan on delivering all the failed email broadcasts users have already sent and excluding duplicate ones users may have inadvertently queued up. Standing by for further updates.

3:52 PM — We have restored our email sending capabilities to “normal.” At a technical level, we found an incompatibility between our MTA and our DNS software that properly routes messages to the right destinations on the Web (and internally). We are currently working on process failed batches We do not recommend you re-queue them again. Our support team is working closely with our systems team to restart these broadcasts that have stalled during this outage. The impacted customers of this appear to be less than 8% of our users. Thanks for your patience.

4:52 PM — Issue resolved. We are currently processing previously failed Follow Up Sequence Step Emails (emails in a FUP sequence) as well as interrupted email “batch” broadcasts during our earlier outage. Our email servers are back to a normal, healthy volume and we are not noticing any related issues. For broadcasts that did not send at all, they will remain unsent. To check this, go to your Email Batch Status Report, go the specific broadcast and check the status. If ALL the messages failed, you can be free to send the broadcast again as if nothing ever happened. ***HOWEVER*** Due to the number failed email messages, we are unable to anticipate how quick these are done. Please exercise care in monitoring that report. The intended result would be a failed email broadcast would later change to “delivered” without worrying about sending the broadcast again.

6:02 PM — Issue resolved. All failed broadcasts have been delivered and email volume is normal.

RESOLVED 3:48PM — We have confirmed that systems are stable and working properly for all customers. We have narrowed the outage period between 1:50PM and 3:15PM Pacific Time today. We did experience a spike in support-related call volume that indicates this. If you have have any problems with your application related to this issue, please provide a detailed Fusebox ticket so we can handle it properly and refer to this post.We apologize for the inconvenience you may have experienced, but with our quick response, no long-term damage occurred.

####

UPDATE 3:32PM — Our systems team isolated the problem that caused this issue. Long story short, an master email server (akin to routing emails), became unresponsive. We rebooted the machine and attempted to bring the system back up, but what resulted was the database was taking unusually long to become alive. This email “router” is responsible for telling applications which email servers to use and permits or denies the emails to be process. As a result anything that required to send an email (such as Web Form confirmation emails, order confirmations, etc.) would hang and not process.

At this time, all applications and systems are coming back online and restoring their stability and resuming normal processing. Stay tuned for further updates.

####

At this time, all systems and applications are not working properly. A database binding (relating to connectivity between databases and other servers) have been severed and we’re working to bring systems back online.

Because e-mail is down, anything related to it is not functioning.

As a result users will observe the following services to be not functional:

• E-Mail, Broadcast Email, Follow-up Sequences
• Order Forms
• Shopping Carts
• Web Forms
• Anything else that relates to email in the app.

It does appear clients are able to log into the app, but that’s about it.

We have all systems and development teams working on bringing the system back online. If you have diagnostic data to report, please log into your app and create a case in the Fusebox with any data you can provide — that is, how you’re impacted and any evidence of the system not functioning correctly.

Please review the updates above for updates.