This morning, a number of support questions were elevated from customers who observed abnormal performance from the Infusionsoft application. Specifically, longer loading times for doing functions involving contact records within the app. This will also add latency to logging into the app.

We can’t confirm, but estimate this to impact all users of Infusionsoft. We’ll have a better idea as we research the issue.

All other functions that do not relate specifically to updating contact records should be running well.

We do not have an ETA yet as we are still investigating the cause. Our systems team is analyzing all components of our application servers, components and databases to identify any potential causes for this.

We’ll keep this post updated with the progress of the situation. Please check back for updates. We apologize for any inconvenience this may have caused.

—
UPDATES:

1:50 PM — Our systems team is still investigating the cause for the degraded performance. Earlier, we did narrow the scope down to anything that actively involves contact records (adds/updates/edits). We have also heard reports of emails being sent slowly; it appears to be related to the bandwidth provided to our MTAs. If you have an issue with this – please submit a support case with details, Batch ID and any additional details that will aid us in tracking this issue. Our support team is working to review these and communicate them with our systems team for swift resolution.

6:19 PM — Our systems team has identified the cause and has a proposed patch in the works. We are performing further testing before we deploy it to ensure system stability and reliability.

8:00 PM — We successfully deployed the patch after testing it for reliability that stabilized. This issue was *separate* from any issues with email delivery latency that many of you have commented on this post. This issue will be tracked separately. Thanks again for your patience and understanding as we worked through this technical matter.

Today at 1:10PM, we became aware of a number of users reporting unexpected slowness and performance when sending emails. This issue is of a critical nature and our systems team is working to move all backend email systems to a separate system.

We believe the scope of this issue impacts all users, but we haven’t confirmed that. You may observe very slow email sending performance at this time while we investigate this and fix it.

This post will be updated with the latest details as we have them. Thank you for your patience and sorry for any inconvenience this may have caused.

—
UPDATES:

1:43 PM — We’re working with our email system provider to get this fixed. At this time it appears this impacts all customers and emails may not be sending properly. The result you might observe is an “ERROR” for the status.

3:36 PM — We have identified the issue and narrowed it down to a performance bug in our MTAs. Basically the Infusionsoft application was attempting to send mails to an overloaded mail server. We are assessing all options to restore reliable email delivery. Right now we are using our secondary (failsafe) email system which emails are being sent, but slower than normal. You do not need to re-queue new batches (and we don’t recommend that, either). We plan on delivering all the failed email broadcasts users have already sent and excluding duplicate ones users may have inadvertently queued up. Standing by for further updates.

3:52 PM — We have restored our email sending capabilities to “normal.” At a technical level, we found an incompatibility between our MTA and our DNS software that properly routes messages to the right destinations on the Web (and internally). We are currently working on process failed batches We do not recommend you re-queue them again. Our support team is working closely with our systems team to restart these broadcasts that have stalled during this outage. The impacted customers of this appear to be less than 8% of our users. Thanks for your patience.

4:52 PM — Issue resolved. We are currently processing previously failed Follow Up Sequence Step Emails (emails in a FUP sequence) as well as interrupted email “batch” broadcasts during our earlier outage. Our email servers are back to a normal, healthy volume and we are not noticing any related issues. For broadcasts that did not send at all, they will remain unsent. To check this, go to your Email Batch Status Report, go the specific broadcast and check the status. If ALL the messages failed, you can be free to send the broadcast again as if nothing ever happened. ***HOWEVER*** Due to the number failed email messages, we are unable to anticipate how quick these are done. Please exercise care in monitoring that report. The intended result would be a failed email broadcast would later change to “delivered” without worrying about sending the broadcast again.

6:02 PM — Issue resolved. All failed broadcasts have been delivered and email volume is normal.

Some users may experience a delay accessing and using their application due to the new User Home that was deployed. Our systems and development team are investigating and resolving the issue now.

It appears complex queries performed/saved on the User Home cause the application to perform them in excess causing performance issues. A majority of gadgets and searches have not caused issues, so feel encouraged to continue using the feature.

Users may switch back to the ‘old’ User Home by clicking the corresponding link located at the top.

Thanks for your patience and understanding, we’ll update you when this is resolved.