RESOLVED 3:48PM — We have confirmed that systems are stable and working properly for all customers. We have narrowed the outage period between 1:50PM and 3:15PM Pacific Time today. We did experience a spike in support-related call volume that indicates this. If you have have any problems with your application related to this issue, please provide a detailed Fusebox ticket so we can handle it properly and refer to this post.We apologize for the inconvenience you may have experienced, but with our quick response, no long-term damage occurred.
####
UPDATE 3:32PM — Our systems team isolated the problem that caused this issue. Long story short, an master email server (akin to routing emails), became unresponsive. We rebooted the machine and attempted to bring the system back up, but what resulted was the database was taking unusually long to become alive. This email “router” is responsible for telling applications which email servers to use and permits or denies the emails to be process. As a result anything that required to send an email (such as Web Form confirmation emails, order confirmations, etc.) would hang and not process.
At this time, all applications and systems are coming back online and restoring their stability and resuming normal processing. Stay tuned for further updates.
####
At this time, all systems and applications are not working properly. A database binding (relating to connectivity between databases and other servers) have been severed and we’re working to bring systems back online.
Because e-mail is down, anything related to it is not functioning.
As a result users will observe the following services to be not functional:
- E-Mail, Broadcast Email, Follow-up Sequences
- Order Forms
- Shopping Carts
- Web Forms
- Anything else that relates to email in the app.
It does appear clients are able to log into the app, but that’s about it.
We have all systems and development teams working on bringing the system back online. If you have diagnostic data to report, please log into your app and create a case in the Fusebox with any data you can provide — that is, how you’re impacted and any evidence of the system not functioning correctly.
Please review the updates above for updates.
