Past Incidents

The Tucows engineering team have so far restored 91% of the affected mail services.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution and we will continue to keep you updated on progress.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+Into+Webmail

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

The Tucows engineering team have so far restored 43% of the affected mail services. We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution and we will continue to keep you updated on progress.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+Into+Webmail

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

The Tucows engineering team have so far restored 35% of the affected mail services.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution and we will continue to keep you updated on progress.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+Into+Webmail

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

The Tucows engineering team have so far restored 12% of the affected mail services.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution.

We will continue to keep you updated on progress.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+Into+Webmail

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

Update @ 3:08 AM MDT: The Tucows engineering team have so far restored 25% of the affected mail services.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution.

We will continue to keep you updated on progress.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this. https://www.midphase.com/knowledgebase/display/MID/How+To+Log+Into+Webmail

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

Tucows' engineering team have recovered the underlying system affecting 15% of users and are currently performing restarts of the customer interfaces for IMAP and SMTP.
We apologize for any inconvenience this may cause.

We are aware of issues affecting email functionality. We continue to work with Tucows' engineering team to restore this service.

This issue is not affecting all customers and is currently impacting approximately 15% of our mailbox users. We apologize for any inconvenience this may cause.

We are aware of issues affecting email functionality. We are working with our 3rd party email service provider (Tucows) to resolve the issue. We apologize for any inconvenience this may cause.

We can confirm the Tucows engineering team have now restored all mail services.

We sincerely apologise for the inconvenience this has caused our customers. We remain in close dialogue with Tucows around this incident and next steps.

We will be distributing a full Root cause analysis to all affected customers.

If you have any questions, please contact our support team who are here 24/7 and happy to help.

This servers are undergoing emergency maintenance and will be available again as soon as possible. We have experienced an unexpected hardware issue with the chassis that houses these four servers and are working to get it resolved.

Update @ Thu Jul 23 08:05:23 MDT 2020: The failed hardware has been replaced and these servers are online and available for use once again.

The Tucows engineering team have continued to make good progress this morning with 80% of mail services now back online. Services are being restored by domain name in alphabetical order.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution.

We will continue to keep you updated on progress. A full Root cause analysis will be provided to all affected customers once service has been fully restored.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

The Tucows engineering team have made good progress overnight with over half of the mail services now back online. Services are being restored by domain name in alphabetical order. Tucows estimate all mail services will be restored by lunchtime.

We sincerely apologise for the inconvenience this continues to cause our customers. We remain in close dialogue with Tucows and continue to press them for urgent resolution.

We will continue to keep you updated on progress. A full Root cause analysis will be provided to all affected customers once service has been fully restored.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

We are aware that there are issues affecting this server. We are working to get them resolved.

Update @ Thu Jul 23 00:31:24 MDT 2020: The server is online and available for use once again.

The Tucows engineering team have finished recovery of the platform and will now start to restore mail services. They have estimated that all mail services should be back online tomorrow morning.

We sincerely apologise for the inconvenience this is causing our customers. We remain in close dialogue with Tucows and continue to press them for an urgent resolution.

We will continue to keep you updated on progress. A full Root cause analysis will be provided to all affected customers once service has been fully restored.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

UPDATE: At 03:51:53 MDT, the server became unresponsive and has been rebooted. It is now online and available for use once again.

INCIDENT: We are aware that there are issues affecting this server. Technical teams have been engaged and are investigating as a priority.

The Tucows engineering team are completing final data integrity checks. Once that has been completed they will start work to restore mail services. We will notify our customers when that starts and give an indication of when mail services should be back online.

Again, we apologise for the inconvenience this is causing our customers. This is being worked on as an absolute priority and we will continue to keep you updated on progress. A full Root cause analysis will be provided to all affected customers once service has been fully restored.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

We continue to work with Tucows following their service failure on Friday 17th July. This issue is not affecting all customers and is currently impacting approximately 15% of our mailbox service users.

Please note: this issue will not result in data loss.

The service interruption was due to shared storage metadata corruption limited to customers that operate on the new shared storage infrastructure.

The Tucows engineering team have been performing multiple recovery and repair steps. They are in the final phase of this and when this completes successfully, they will begin the final steps to bring mail services back online.

We will receive an update from them at 6 PM UTC, after which we will post an update here.

Again, we apologise for the inconvenience this is causing our customers. This is being worked on as an absolute priority and we will continue to keep you updated on progress. A full Root cause analysis will be provided to all affected customers once service has been fully restored.

As a workaround, clients are still able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

INCIDENT UPDATE

We continue to work with Tucows following their service failure on Friday 17th July. This issue is not affecting all customers and is currently impacting approximately 15% of our mailbox service users.

Please note: this issue will not result in data loss.

We will receive an update from them at 3PM UTC, after which we will post an update here.

Again, we apologise for the inconvenience this is causing our customers. This is being worked on as an absolute priority and we will continue to keep you updated on progress.

As a workaround, clients are able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc.) but, you will not be able to send/receive from your client at this time.

A full Root cause analysis will be provided to all affected customers once service has been fully restored.

Title: Planned Server Maintenance - slmp-550-86, slmp-550-87, slmp-550-88, slmp-550-89
Date: July/21/2020
Start time: 12 AM MDT
End time: 6 AM MDT
Duration: 5-6 hours

To provide you with the most reliable and robust hosting environment available, we will be upgrading the operating system on the following servers:

slmp-550-86.slc.westdc.net
slmp-550-87.slc.westdc.net
slmp-550-88.slc.westdc.net
slmp-550-89.slc.westdc.net

This maintenance window will start on July 21st, 2020 between the hours of 12 a.m. - 6 a.m. MDT. It is expected there will be around 2-3 hour hours of downtime during this window. The upgrade process will take up to 5-6 to fully complete.

If you have any questions or concerns, please contact our support team.

Update @ Tue Jul 21 05:37:31 MDT 2020: The maintenance has been successfully completed.

We're currently working through some emergency maintenance on our website building platform. During this time, access to services on these systems will be unavailable. We apologize for any inconvenience and will ensure further updates are posted as soon as this maintenance is complete

UPDATE - The emergency maintenance is complete. The website builder is back online.

We are currently experiencing issues with the telephone service of our Billing team. Our technical teams are currently investigating the issue as a priority. We apologise for any inconvenience you might experience.

UPDATE:

We continue to work with the technical teams at our 3rd party service provider (Tucows). This issue is currently impacting approximately 15% of our mailbox service users.

Please note: this issue will not result in data loss.

They continue to work on service restoration and are currently estimating this could take up to 2 more business days. Again, we apologise for the inconvenience this is causing our customers. This is being worked on as an absolute priority and we will continue to keep you updated on progress.

As a workaround, clients are able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this.

https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc). but, you will not be able to send/receive from your client at this time.

A full Root cause analysis will be provided to all affected customers once service has been fully restored.

INCIDENT:

From 02:30 UTC on July 17th 2020, our technical team became aware of a 3rd party issue affecting some of our clients connecting to the email platform via IMAP & POP3.

No data-loss is expected as a result of this issue.

Resolution steps have been ongoing to resolve these issues. However, this has not progressed as quickly as expected. We continue to monitor the situation and will continue to provide regular updates.

As a workaround, clients are able to send and receive emails from their account(s) via webmail. Please use the following link for the knowledge article explaining how to do this. https://www.midphase.com/knowledgebase/display/MID/How+To+Log+into+Webmail+from+CHI

Unfortunately, access to your old emails is only possible if you have previously configured and accessed your emails via any mail client (such as Outlook, Mail, Thunderbird, etc). but, you will not be able to send/receive from your client at this time.

We apologise for the inconvenience caused by this issue. If you have any other questions, please let us know and we will be happy to help!

Title: Planned Server Maintenance - slmp-550-82, slmp-550-83, slmp-550-84, slmp-550-85
Date: July/20/2020
Start time: 12 AM MDT
End time: 6 AM MDT
Duration: 5-6 hours

To provide you with the most reliable and robust hosting environment available, we will be upgrading the operating system on the following servers:

slmp-550-82.slc.westdc.net
slmp-550-83.slc.westdc.net
slmp-550-84.slc.westdc.net
slmp-550-85.slc.westdc.net

This maintenance window will start on July 20th, 2020 between the hours of 12 a.m. - 6 a.m. MDT. It is expected there will be around 2-3 hour hours of downtime during this window. The upgrade process will take up to 5-6 to fully complete.

If you have any questions or concerns, please contact our support team.

Update @ Mon Jul 20 05:53:43 MDT 2020:
The maintenance has been completed. All servers except for slmp-550-85 have been successfully upgraded.

We have experienced an unexpected disk failure on slmp-550-85 and had to cancel the upgrade on this server due to degraded disk performance and possible issues it may cause. We will re-schedule this maintenance to a later date as soon as the server completes RAID rebuild.