Today at 1:10PM, we became aware of a number of users reporting unexpected slowness and performance when sending emails. This issue is of a critical nature and our systems team is working to move all backend email systems to a separate system.
We believe the scope of this issue impacts all users, but we haven’t confirmed that. You may observe very slow email sending performance at this time while we investigate this and fix it.
This post will be updated with the latest details as we have them. Thank you for your patience and sorry for any inconvenience this may have caused.
—
UPDATES:
1:43 PM — We’re working with our email system provider to get this fixed. At this time it appears this impacts all customers and emails may not be sending properly. The result you might observe is an “ERROR” for the status.
3:36 PM — We have identified the issue and narrowed it down to a performance bug in our MTAs. Basically the Infusionsoft application was attempting to send mails to an overloaded mail server. We are assessing all options to restore reliable email delivery. Right now we are using our secondary (failsafe) email system which emails are being sent, but slower than normal. You do not need to re-queue new batches (and we don’t recommend that, either). We plan on delivering all the failed email broadcasts users have already sent and excluding duplicate ones users may have inadvertently queued up. Standing by for further updates.
3:52 PM — We have restored our email sending capabilities to “normal.” At a technical level, we found an incompatibility between our MTA and our DNS software that properly routes messages to the right destinations on the Web (and internally). We are currently working on process failed batches We do not recommend you re-queue them again. Our support team is working closely with our systems team to restart these broadcasts that have stalled during this outage. The impacted customers of this appear to be less than 8% of our users. Thanks for your patience.
4:52 PM — Issue resolved. We are currently processing previously failed Follow Up Sequence Step Emails (emails in a FUP sequence) as well as interrupted email “batch” broadcasts during our earlier outage. Our email servers are back to a normal, healthy volume and we are not noticing any related issues. For broadcasts that did not send at all, they will remain unsent. To check this, go to your Email Batch Status Report, go the specific broadcast and check the status. If ALL the messages failed, you can be free to send the broadcast again as if nothing ever happened. ***HOWEVER*** Due to the number failed email messages, we are unable to anticipate how quick these are done. Please exercise care in monitoring that report. The intended result would be a failed email broadcast would later change to “delivered” without worrying about sending the broadcast again.
6:02 PM — Issue resolved. All failed broadcasts have been delivered and email volume is normal.
Cox Communications Outage Impacting CA & AZ Users (Resolved)
UPDATE 2/2/2010: Cox Communications restored service for their southwestern region during the night and we confirm the issue has be resolved. Yay for fast Internet!
—
This afternoon our office was impacted by an outage from Cox Communications. This outage is at the tier-1 backbone level for Cox customers in California and Arizona. You may experience abnormally slow Internet performance as a result.
This does not impact any Infusionsoft production servers as those are hosted offsite from the Cox Communications network. Basically, we’re slow in our offices and if you’re a Cox Internet subscriber, you may observe slow Internet performance.
This is the statement posted by Cox Communications on their Network Status page:
(Posting this message to you at about a 56K speed. Sorry for the delay.)