At about 7:26 AM to 7:30 AM this morning, one of our datacenters lost network connectivity.

We are investigating with the provider as to the cause and will identify any opportunities for us to prevent it in the future. While the downtime was short (four minutes), we are notifying users because it was affected a majority a minority of our Phoenix-based users.

The data center is located in Tempe, AZ and our systems team is in contact with them. At this time, they confirm full network connectivity has been restored.

—
Updates:

7/7/2010 10:11 AM – We have determined the impact of this outage was isolated to only between the data center and the Cox network in the Phoenix, Arizona region. As such, a very small number of users may have been affected.

As we continue to grow, so do our network needs. We will be making major upgrades to our network infrastructure and this will cause a planned downtime for all users of Infusionsoft during our lowest of usage periods.

Infusionsoft will be unavailable on Saturday, March 13, 2010 between 12:00 AM and 3:00 AM EST. (Click here to see what time in your time zone.) We don’t plan our network upgrade to take this long however, we are prepared for any delays or issues that may come up.

During this period, you will not be able to access or use Infusionsoft. In addition, scheduled actions in Infusionsoft will not be ran including follow-up sequences, broadcasts or any other scheduled actions during this time period. (They will run directly following the downtime, requiring no action on your part.)

IMPORTANT! During this during this downtime period…

• eCommerce pages such as store fronts, order forms and any other public-facing pages will not be accessible;
  
• Web Form submissions will be non-functional;
• Trackable Links will be non-functional;
• Affiliate redirects will be non-functional;
• Open-rate tracking will be non-functional;
• Applications will be inaccessible completely to anyone who accesses it;
• API connections will not function;
• All applications (even our own) will be impacted by this downtime.

This blog will be updated with details prior to and after the network upgrade, and we’ll also keep you posted on our @InfusionStatus Twitter account.

Thanks for your understanding and patience!

Update: *RESOLVED* Our maintenance was complete at 12:28 AM with no issues.

UPDATE 2/2/2010: Cox Communications restored service for their southwestern region during the night and we confirm the issue has be resolved. Yay for fast Internet!

—
This afternoon our office was impacted by an outage from Cox Communications. This outage is at the tier-1 backbone level for Cox customers in California and Arizona. You may experience abnormally slow Internet performance as a result.

This does not impact any Infusionsoft production servers as those are hosted offsite from the Cox Communications network. Basically, we’re slow in our offices and if you’re a Cox Internet subscriber, you may observe slow Internet performance.

This is the statement posted by Cox Communications on their Network Status page:

Arizona and San Diego — Connection Slow — Customers in Arizona and San Diego are experiencing slow connections at this time. Our technicians are aware of the issue and are working to resolve it as quickly as possible. Currently, there is no estimated time of repair.

(Posting this message to you at about a 56K speed. Sorry for the delay.)

We’ll undergo a brief loss of connectivity this evening during the hours of 9PM and 10PM MST for planned maintenance. This is needed so we can upgrade and replace a power distribution unit for our servers. During this period, you might not be able to access your application, but should hardly be noticed by any clients.

This notice was sent to all customers when they login to their application:

At Infusionsoft, we pride ourselves on bringing you 99.9% uptime with your Infusionsoft application. We are continually striving to provide you that top level of service everyday. Because of our commitment to your success, Infusionsoft will be performing maintenance tonight, Thursday, June 4, 2009, between 9:00 – 10:00 PM MST.

We are not making any updates or changes to your application.

We will be replacing a power distribution unit in order to ensure the highest quality of service.

There may be a network disruption lasting between 2 and 5 minutes during this maintenance window.

We apologize for the inconvenience.

Infusionsoft Customer Service and Support
1-866-800-0004

This morning, several customers have reported the inability to access their application, reporting a Network Timeout error. We assume this is the error being reported from Firefox. This is probably due to a DNS change or a some hop in the route to our servers.

We don’t have any details, but the problems occurred since about 5AM Pacific Time. Our own network monitoring services have not detected any outages and the servers appear to have the same expected load from customers using our software.

Please read this post, Is Infusionsoft Down? and report back to us in this post. We’re still looking into the cause and touching base with those customers who reported it to us this morning.

Thanks!

~Joe

[UPDATE 10:26am]
Our systems team has identified the failing network. It’s Limelight Networks and only about 5% of our customers cruise in on that network. We’ve opened up tickets with our ISPs on getting this resolved quickly. We apologize for the inconvenience and look forward to getting this resolved as quickly as you are.

[UPDATE 11:57am]
We have confirmation that the affected network’s router has been fixed. All Web traffic should be processed as normal and all clients should now be able to access their Infusionsoft application.  Thanks for your support and interest in this matter! As a result, we’re classifying this as resolved.

~Joseph