This morning, we became aware of a system-wide issue impacting anyone using a 1.19 (Summer Release) involving HTML email templates. At this time we are aware of the full scope of the issue and there is unfortunately no workaround. We’re working hard to address this critical issue as quickly as we can.

There are two main issues at hand: Certain text like apostrophes, registered trademarks, etc. in email would turn into “'” and respective XHTML entities and not be actually rendered as designed. Another impact of the issue is when there are styling around text such as bold or emphasis, it will remove the space between the words.

From what we have tested, it appears all (UPDATE, read below!) few outbound HTML emails are impacted. Including ones from the Classic HTML email editor and the new Email Builder. We’ve isolated the cause and are testing a solution at this time.

We will update this post with progression and resolution. We apologize for this inconvenience and appreciate your patience. We aim to have this resolved over the weekend.

—

UPDATES:

1:21 PM 6/11/2010: Our development team narrowed the issue down to only users who edit a template AND click the Source tab. At this time we have a fix in place and are currently testing it. The workaround is to not click the Source tab. One caveat — we are not able to restore the spaces between words that may have been modified by HTML styles. We strongly recommend that users who edited their HTML email templates recently to confirm they are rendering the way it should.

1:27 PM 6/11/2010: We feel confident the changes will pass QA and deployed tonight. Much better than a weekend.

8:04 PM 6/11/2010: The issue has been resolved and deployed to affected applications.

8:21 AM 6/18/2010: Based on a couple reports from customers, we have re-opened this case and have an additional patch that will go out at the next maintenance release. If you’re still having trouble, it is important to contact support so they can properly diagnose the situation.

9:00 AM 6/22/2010: This has been resolved and corroborated by the lack of customer reports of the issue. Thanks for your patience.

At 10:03 PM on 6/10/2010, we’ve successfully deployed our fourth phase of our 2010 Summer Release to users. Approximately 1700 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.29

At 9:38 PM on 6/09/2010, we’ve successfully deployed our third phase of our 2010 Summer Release to users. Approximately 800 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.25

At 8:46 PM, we’ve successfully deployed our second phase of our 2010 Summer Release to users. Approximately 500 applications have been upgraded. Users of those specific applications have been notified via notification following successful login.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

Version: 1.19.1.24

We’ve successfully deployed the latest production version of our 2010 Summer Release to our Beta users. After hitting a few snags late last week, we’re on pace to deploy to all users in the next couple weeks.

To learn more about the new features and enhancements, read about the 2010 Summer Release on the Infusionsoft blog.

This applies to you if you have 1.19.1.24 in the lower left corner of your application.

Today, we discovered an issue where some applications are experience higher than normal memory usage. We will be selectively rebooting various front-end servers until we resolve the underlying issue causing more memory usage. (High memory usage could severely impact performance.)

We use multiple applications on our servers so if one app behaves abnormally, others may be affected. To prevent adverse effects, our Systems team will identify troublesome servers and reboot them to release the memory. They’re super smart and dedicated and will be doing this in a way to minimize impact to you.

During the process of our server reboots, users will be logged out from their active Infusionsoft sessions and will need to log in again. From that point, there may be a short delay from being logged out to logging in again.

We’re aware of these issue as we are initiating the reboots. There is no need to contact support for this matter. Our Systems team will continue to monitor our servers and application performance and will be making improvements to keep things running smoothly.

If you are writing lengthy emails in your templates, we recommend to save a copy on your computer locally and to save frequently in the application until this issue is resolved. We’ll post updates on the blog post here so you can check back to know when we determine this to be fully resolved.

Thanks for your patience and understanding. Updates will be posted below.

—

UPDATE 5/14/10 1:31 PM: *RESOLVED* Our Systems team has identified the culprit and isolated the performance to a very small group of applications. As such, we will not be rebooting server today. Enjoy your awesome uptime and performance.

A few users have inquired about the anomaly where  some messages associated with a contact record would be shown to the user with a panic-inducing “No Body Found” message. Our team was made aware of this issue and we’ve identified the cause and are currently pursing a patch to be deployed sometime next week.

Not all users are impacted. No content has left our network with “No Body Found,” and it is only cosmetic in nature to application users.

It appears this is how the application is pulling the content from the database and is displaying it to application users. The data being displayed is in the HTML view when the messages are in plain-text and that’s why it’s not showing right now. It will be fixed when the patch is deployed.

You do not need to contact support for this issue – our developers are working on this. Thanks for your patience and understanding.

UPDATE 5/14/10 11:01 AM: This issue has been resolved for the most part. We’ve seen instances where messages with attachments might not display properly. In those instances, please submit the specific message to our support team so they can track it and research it further. A majority of messages though, this is not a problem.

This morning, a number of support questions were elevated from customers who observed abnormal performance from the Infusionsoft application. Specifically, longer loading times for doing functions involving contact records within the app. This will also add latency to logging into the app.

We can’t confirm, but estimate this to impact all users of Infusionsoft. We’ll have a better idea as we research the issue.

All other functions that do not relate specifically to updating contact records should be running well.

We do not have an ETA yet as we are still investigating the cause. Our systems team is analyzing all components of our application servers, components and databases to identify any potential causes for this.

We’ll keep this post updated with the progress of the situation. Please check back for updates. We apologize for any inconvenience this may have caused.

—
UPDATES:

1:50 PM — Our systems team is still investigating the cause for the degraded performance. Earlier, we did narrow the scope down to anything that actively involves contact records (adds/updates/edits). We have also heard reports of emails being sent slowly; it appears to be related to the bandwidth provided to our MTAs. If you have an issue with this – please submit a support case with details, Batch ID and any additional details that will aid us in tracking this issue. Our support team is working to review these and communicate them with our systems team for swift resolution.

6:19 PM — Our systems team has identified the cause and has a proposed patch in the works. We are performing further testing before we deploy it to ensure system stability and reliability.

8:00 PM — We successfully deployed the patch after testing it for reliability that stabilized. This issue was *separate* from any issues with email delivery latency that many of you have commented on this post. This issue will be tracked separately. Thanks again for your patience and understanding as we worked through this technical matter.

We are under way of releasing our Winter Release (version 18) to Infusionsoft users. This will be a steady, paced roll out to ensure optimal user experiences. This post will be updated with details and status of the deployment.

—

1/28/2010 10:33AM — Last night around 11PM, we deployed an update to a very small sampling of users. So far,  so good. We’re doing more testing until we expand this release to more users. These users were notified of their application update at 1AM this morning.

2/02/2010 6:26PM — Everyone who already has the Winter Release will be receiving an update for refinements. This refinement will noticed by the “18.1″ version number in the lower left corner. No new features from the Winter Release have changed, other than a number of back-end optimizations.

2/04/2010 12:19AM — 112 more applications received their Winter Release update. From yesterday, we did not observe any system performance or issues from users. (Don’t worry, we’ll be releasing it in wider, larger batches on future deployments from this.)

2/08/2010 5:29PM — 1189 more applications received their Winter Release update during this weekend. Our fourth of five phased updates will occur this evening. We have not identified any critical system issues or performance as a result and the release is working great for our customers!

2/09/2010 9:44PM — About 1300 more applications received their update yesterday evening. We haven’t had any critical issues stemming from this release and are confident to continue the deployment to the rest of the users this week. The one application performance issue that people have noticed (and is noted on this blog) is unrelated to the Winter Release and rather the underlying architecture that we have identified and are resolving shortly.

2/11/2010 8:10PM — At this time we are holding off on the final deployment due to a few system issues that would negatively impact customer experience. Once we’re confident things are running smoothly, we’ll deploy to the rest of users. Thanks for your patience!

2/16/2010 8:35PM — After much tweaking, monitoring and testing to the code base, we have identified and resolved the previous impairments that prevented us from scaling our Winter Release to customers. Good news is that we’re finally caught up and will be releasing to another group of our customers. This evening, we will be rolling out to 187 applications. Stay tuned for an update following the release.

RESOLVED 3:48PM — We have confirmed that systems are stable and working properly for all customers. We have narrowed the outage period between 1:50PM and 3:15PM Pacific Time today. We did experience a spike in support-related call volume that indicates this. If you have have any problems with your application related to this issue, please provide a detailed Fusebox ticket so we can handle it properly and refer to this post.We apologize for the inconvenience you may have experienced, but with our quick response, no long-term damage occurred.

####

UPDATE 3:32PM — Our systems team isolated the problem that caused this issue. Long story short, an master email server (akin to routing emails), became unresponsive. We rebooted the machine and attempted to bring the system back up, but what resulted was the database was taking unusually long to become alive. This email “router” is responsible for telling applications which email servers to use and permits or denies the emails to be process. As a result anything that required to send an email (such as Web Form confirmation emails, order confirmations, etc.) would hang and not process.

At this time, all applications and systems are coming back online and restoring their stability and resuming normal processing. Stay tuned for further updates.

####

At this time, all systems and applications are not working properly. A database binding (relating to connectivity between databases and other servers) have been severed and we’re working to bring systems back online.

Because e-mail is down, anything related to it is not functioning.

As a result users will observe the following services to be not functional:

• E-Mail, Broadcast Email, Follow-up Sequences
• Order Forms
• Shopping Carts
• Web Forms
• Anything else that relates to email in the app.

It does appear clients are able to log into the app, but that’s about it.

We have all systems and development teams working on bringing the system back online. If you have diagnostic data to report, please log into your app and create a case in the Fusebox with any data you can provide — that is, how you’re impacted and any evidence of the system not functioning correctly.

Please review the updates above for updates.