UPDATE 9:06pm: This issue has been resolved and customers have been notified with additional details in their application. Login to your application to read the latest on the Broadcast email matter (it doesn’t deviate much from the info below). We narrowed the affected customers to under 200 apps. We paused (and resumed as of this update) all outbound email in order to minimize the impact on customers, subscribers and to maintain email delivery reputation. Outbound email was resumed at 7:00PM MST across all applications and all systems are running well. Our customer notification tool has been fixed as well, so when you access your Infusionsoft application, you will see important notices from Infusionsoft, as expected.
At this point, we have declared this as an “ALL CLEAR.” Our mail servers are currently crunching away at tons of backlogged email to be delivered to recipients and it’s expected it will result in a temporary delays in delivery of email. Thanks everyone for their patience as we resolved this issue.
–Joe
####
This morning our support staff arrived to see several customers reporting their recipients on batch (broadcast) email received multiple copies of email messages. This was observed by customers on the morning of Sunday, September 6, 2009. The resulting effect was recipients received numerous copies of the same email. The additional after affects would be increased opt-outs and spam complaints.
Important: While only a handful of customers have reported the issue, we believe this is system-wide and have taken measures to prevent further problems for our customers. We temporarily paused all future email messages for broadcasts and follow-up sequences until this is resolved. This means, email will not be able to be sent from the application until we enable it. You do not need to send additional emails, we have merely paused it and will resume (hopefully, this afternoon).
Our development team is actively investigating the cause of this issue and working hard to prevent this from happening again. They are confident it should be resolved by the end of the day today (as of this post). Updates and more information will be provided as it is available.
Also, our internal client notification tool is unavailable, so we are unable to inform our customers via their login to their application.
If you are impacted and wish to provide samples, please submit a ticket through the Fusebox. However, at this time we have identified the issue and we’re in progress to resolving it.
Geek Stuff: The issue appears to have been caused between the scheduling server and the email server. When an application is sending email through multiple threads, the scheduler would loop through until a thread is terminated and send mail. This resulted in a variable amount of messages being sent to recipients — about three to ten each. We paused the scheduler and will reschedule broadcasts to go out when the issue is fixed, and messages will be sent in the priority and order it was received.
We sincerely regret any inconvenience this may have caused you or your customers.
