At about 7:26 AM to 7:30 AM this morning, one of our datacenters lost network connectivity.

We are investigating with the provider as to the cause and will identify any opportunities for us to prevent it in the future. While the downtime was short (four minutes), we are notifying users because it was affected a majority a minority of our Phoenix-based users.

The data center is located in Tempe, AZ and our systems team is in contact with them. At this time, they confirm full network connectivity has been restored.

—
Updates:

7/7/2010 10:11 AM – We have determined the impact of this outage was isolated to only between the data center and the Cox network in the Phoenix, Arizona region. As such, a very small number of users may have been affected.

This morning, we encountered an issue where users will notice a cosmetic issue with the chart display on Email Batch Status Report. What happens is users access the report and notice only one contact has opened the message and everyone else hasn’t taken action with email messages sent.

Example:

Luckily, there’s a workaround.

To see who interacted with your message, click the pie chart to see the accurate results. Alternatively, you can do an email batch report and select the appropriate message (or period of time). From there you will be able to view the contacts under the conditions of what you’ve searched for.

We’re aware of the issue and plan to have this issue resolved tomorrow evening. This data is all there and is merely a display issue on the Email Batch Status/Broadcast Report.

Thanks for your patience and understanding in this matter; we’ll update this post with further developments.

Updates

7:00 PM 6/16/2010: This issue has been resolved.

This morning, we became aware of a system-wide issue impacting anyone using a 1.19 (Summer Release) involving HTML email templates. At this time we are aware of the full scope of the issue and there is unfortunately no workaround. We’re working hard to address this critical issue as quickly as we can.

There are two main issues at hand: Certain text like apostrophes, registered trademarks, etc. in email would turn into “'” and respective XHTML entities and not be actually rendered as designed. Another impact of the issue is when there are styling around text such as bold or emphasis, it will remove the space between the words.

From what we have tested, it appears all (UPDATE, read below!) few outbound HTML emails are impacted. Including ones from the Classic HTML email editor and the new Email Builder. We’ve isolated the cause and are testing a solution at this time.

We will update this post with progression and resolution. We apologize for this inconvenience and appreciate your patience. We aim to have this resolved over the weekend.

—

UPDATES:

1:21 PM 6/11/2010: Our development team narrowed the issue down to only users who edit a template AND click the Source tab. At this time we have a fix in place and are currently testing it. The workaround is to not click the Source tab. One caveat — we are not able to restore the spaces between words that may have been modified by HTML styles. We strongly recommend that users who edited their HTML email templates recently to confirm they are rendering the way it should.

1:27 PM 6/11/2010: We feel confident the changes will pass QA and deployed tonight. Much better than a weekend.

8:04 PM 6/11/2010: The issue has been resolved and deployed to affected applications.

8:21 AM 6/18/2010: Based on a couple reports from customers, we have re-opened this case and have an additional patch that will go out at the next maintenance release. If you’re still having trouble, it is important to contact support so they can properly diagnose the situation.

9:00 AM 6/22/2010: This has been resolved and corroborated by the lack of customer reports of the issue. Thanks for your patience.

Today, we discovered an issue where some applications are experience higher than normal memory usage. We will be selectively rebooting various front-end servers until we resolve the underlying issue causing more memory usage. (High memory usage could severely impact performance.)

We use multiple applications on our servers so if one app behaves abnormally, others may be affected. To prevent adverse effects, our Systems team will identify troublesome servers and reboot them to release the memory. They’re super smart and dedicated and will be doing this in a way to minimize impact to you.

During the process of our server reboots, users will be logged out from their active Infusionsoft sessions and will need to log in again. From that point, there may be a short delay from being logged out to logging in again.

We’re aware of these issue as we are initiating the reboots. There is no need to contact support for this matter. Our Systems team will continue to monitor our servers and application performance and will be making improvements to keep things running smoothly.

If you are writing lengthy emails in your templates, we recommend to save a copy on your computer locally and to save frequently in the application until this issue is resolved. We’ll post updates on the blog post here so you can check back to know when we determine this to be fully resolved.

Thanks for your patience and understanding. Updates will be posted below.

—

UPDATE 5/14/10 1:31 PM: *RESOLVED* Our Systems team has identified the culprit and isolated the performance to a very small group of applications. As such, we will not be rebooting server today. Enjoy your awesome uptime and performance.

This morning we identified an issue where users would send themselves a test email in a template and they would receive an older copy of the message. Our developers have been made aware of this and are working to resolve it quickly.

This issue won’t impact any emails that sent via broadcast or from follow-up sequence. It is isolated to only the “Save and Send Test” button found on the email editor.

There is a workaround to use in the meantime so you can test your email messages. Simply go to your own contact record and send yourself an email and choose the desired template you wish to test. Alternatively, you can also send a broadcast and target yourself as the recipient.

Sorry for the inconvenience and we appreciate your patience as we fix this.

—

UPDATE 5/13/10 2:48 PM: At this time we have identified the cause and a patch to resolve this. We anticipate to deploy the fix this evening to all applications. We apologize for the delay but thank you for your patience and understanding.

RESOLVED 5/14/10: The issue has been resolved.

A few users have inquired about the anomaly where  some messages associated with a contact record would be shown to the user with a panic-inducing “No Body Found” message. Our team was made aware of this issue and we’ve identified the cause and are currently pursing a patch to be deployed sometime next week.

Not all users are impacted. No content has left our network with “No Body Found,” and it is only cosmetic in nature to application users.

It appears this is how the application is pulling the content from the database and is displaying it to application users. The data being displayed is in the HTML view when the messages are in plain-text and that’s why it’s not showing right now. It will be fixed when the patch is deployed.

You do not need to contact support for this issue – our developers are working on this. Thanks for your patience and understanding.

UPDATE 5/14/10 11:01 AM: This issue has been resolved for the most part. We’ve seen instances where messages with attachments might not display properly. In those instances, please submit the specific message to our support team so they can track it and research it further. A majority of messages though, this is not a problem.

We’re currently investigating an unusual software defect that causes some email addresses to be identified as invalid. Only a few customers have observed this and of those, only a few contacts in their application.

This software defect requires additional investigation and we do not have an ETA yet. We have elevated this in priority so we will be resolved soon. We do not know of a workaround, either.

No, we’re not intentionally trying to break RFC standards.

Thanks for your patience and understanding — updates will be posted below.

Update 3/25/2010: *RESOLVED* This issue has been resolved last night during our maintenance release. Thank you for your patience.

It appears our delivery issues to GoDaddy-hosted email servers has appeared again. We were confident it was fixed previously, but we’re being impacted by the issue again and unfortunately the issue rests in the hands of GoDaddy to resolve. We’re working with GoDaddy’s staff to investigate and resolve this, but it appears this issue is not easily diagnosed or resolved due to the complexity of email spam filters.

Here’s what we know:

• Users who send mail to any GoDaddy-hosted email servers may not be received.
• We’ve seen this universally across transactional, single opt-in and double-opt in.
• Not all email messages are impacted, but many are.

Here’s what is happening:

• Infusionsoft, like all email providers transmits messages to recipient ISPs.
• During transmission, most receiving email servers tell our servers “It’s cool,” “Come back later,” or “No way.”
• Our servers log all transactions for technical assurance, security and provide monitoring data for our Systems team to manage.
• We are transmitting the messages to GoDaddy, their servers tell our servers “It’s cool” and that’s it.
• Our servers can only conclude it was successfully received and the connection terminates.
• (At this point the message leaves our possession and we can’t control what happens to it.)
• UNCONFIRMED: It appears what happens next is GoDaddy is using additional filtering technologies that reject our messages and not giving an error code.
• UNCONFIRMED: Whatever the issue is, the servers are not communicating with each other advising of what the issue is.
• UNCONFIRMED: Some messages may be delivered to the Inbox, some to the “Junk” and many not at all. (!!)

Here’s what we’re doing about it:

• For the past few weeks, we’ve been in contact with GoDaddy and our Systems’ team is monitoring and researching the issue.
• We’ve narrowed the issue to ONLY impact GoDaddy-hosted email servers.
• We’ve escalated this matter to a few different staff members at GoDaddy.
• We’re telling our users about it; assuring them they do not need to call Support. We’re aware and working to resolve this as soon as possible.

What we expect:

• It’s clear this issue will remain unresolved for quite a while until we can confirm it’s resolved.
• Once our issue is properly investigated, we will determine if we need to make any changes. (Unknown ETA)
• We will test those changes before deploying them. (3 Days ETA)
• We will deploy the necessary changes, if applicable. (1 Day ETA)
• We will rejoice!

We apologize we don’t have any better news about the situation, but we want to inform you that we’re working to get this resolved as quickly as possible.  Thanks for your patience and understanding. Further updates on this will be posted below.

—

UPDATE 3/25/2010: We had successful contact with the escalation team at GoDaddy and they have helped us narrow down the cause. We’re still investigating and working with SenderBase to resolve — we’ve made substantial progress to get this resolved. Thank you for your patience and sorry for any inconvenience.

UPDATE 5/14/2010: We have resolved this issue since the end of March. We have confirmed all messages are being properly handled by GoDaddy-hosted domains. We appreciate the help we received from their escalation team to resolve this issue. Thanks for your patience.

This afternoon, we have confirmed a defect in our software from last night’s maintenance release that causes Email Notifications to multiple users at once to not function. In addition, because this function also powers our Fulfillment Reports, those are not expected to function correctly, either.

The problem at this point is solely related to the fact that multiple email addresses are configured for the “send a notification of this … to:” section. If you jump into your Web Form, Order Form and Shopping Cart settings, you will be able to change this quickly.

From our research, the email notifications are not sending due to a defect in the software. The workaround is to make sure you’re sending those email notification to only one recipient, not multiple. We have confirmed Web Form Submission notifications are impacted by this.  The email notifications from both shopping cart sales and order form submissions would also be impacted.

The fulfillment lists and reports are impacted as well. The workaround (if you’re affected) is to make sure you only send reports to one recipient and/or show the Fulfillment List on the User Home. That will enable you to do what you need to until we get this resolved.

Our development team is working the best they can to resolve the issue. We anticipate this will be ready for deployment (fix) tomorrow afternoon. We apologize for any inconvenience you may have experienced.

—

UPDATE, 3/22/10: We have identified a related matter that has held up our deployment of this fix. We’re planning to have this resolved within the next day or so. We’re sorry for the inconvenience as we focus on improving the reliability and functionality of the software.

UPDATE, 3/25/10: *RESOLVED* We have confirmed this issue was resolved and was updated in last night’s maintenance release. Thank you for your patience as we improved our application.

As previously posted, the network upgrade we planned is now complete. The systems team completed this at 12:28 AM and there are no issues.

Thank you for your patience and understanding.