Network Status News & Information

We are currently working on a DNS issue that is affecting services on the following servers:

• cpanel01.reliablewebhosting.ca
• cpanel02.reliablewebhosting.ca
• cpanel03.reliablewebhosting.ca
• cpanel04.reliablewebhosting.ca
• cpanel05.reliablewebhosting.ca
• cpanel06.reliablewebhosting.ca
• cpanel07.reliablewebhosting.ca
• cpanel08.reliablewebhosting.ca
• cpanel09.reliablewebhosting.ca

This may result in intermittent access of services and DNS probe errors in your browser.

This notice will be updated once more information is available or the issue is resolved. We appreciate your patience and understanding in this matter.

UPDATE: 09/02/2019 11:19AM

Most services have been restored at this time. Web and e-mail access have been re-enabled. cPanel and FTP are currently disabled while remaining work is completed.

UPDATE 09/02/2019 1:00AM

The restoration is approximately 80% complete and we expect for the server to be back online and accessible by noon.

UPDATE 09/01/2019 2:00PM

The verification has completed and a manual comparison was performed of the known state of the file system prior to beginning the process and it was determined that the corruption had resulted in quite a bit of data loss. A calculation puts it at around 700GB.

At this time, we have made the decision to invoke our Disaster Recovery plan (initiated at 2PM Eastern Time) and have queued up all cPanel accounts on the server to be restored from known good backups that were generated approximately 12 hours prior to the original maintenance period. We afraid that, due to the nature of having to restore from an remote backup and the amount of data involved, we have reason to believe that the completion of this process may take at least 24 - 48 hours. During that time, e-mail and web services will be unavailable as the full resources of the server will be dedicated to getting this resolved as soon as posslble. We are investigating potential optimizations to the restoration process, but ultimately we are likely limited by the performance of network connections between 2 locations and the backup storage devices being used.

UPDATE 09/01/2019 7:45AM

There was a power outage during the maintenance. As a result, the file system on the server has experienced some corruption and is in the process of being scanned and corrected. Unfortunately this can take several hours or more on servers like this. Rest assured we are working diligently to get this resolved and will post further updates as they become available.

ORIGINAL REPORT:

Due to unforeseen circumstances, we will need to perform emergency maintenance on the raid array on the Zeus server. Without getting too technical, one of the filesystems needs to be resized as it has grown unexpected quickly and is close to running out of available space. Unfortunately this will require taking the server offline for a period of several hours as it is unsafe to do this while running. Efforts will be made to ensure that the resizing is sufficient to avoid this occurring again.

The estimated window for this maintenance is in the Start Date and End Date values shown above - 7:00 PM Newfoundland Time on Saturday, August 31st to 9:00 AM Newfoundland time on Sunday, September 1st . We expect that the work may be completed before then, however we've chosen a conservative estimate to be safe.

This timeslot was chosen as it is generally the slowest traffic period each week and we hope it will lessen any impact on your services.

We appreciate your patience and understanding in this matter.

UPDATE: 11:00PM EST:  This has been corrected and all known related issues have been resolved at this time. If you continue to experience any issues, they are likely unrelated and we recommend submitting a support request for assistance.

UPDATE 7:30PM EST: a new battery was installed, but appears to be defective. Our team will be trying another one shortly. This is also causing a license issue with cPanel and should be corrected when this is done.

While performing maintenance on the Zeus server earlier today, we observed that the time off by several minutes. After testing, this was determined to be due to a failing battery on the motherboard and has been scheduled for an emergency replacement at 4:30pm EST (6PM Newfoundland). We hate having to do this on such short notice, however incorrect time information can impact many aspects of the services provided, including incorrect times on e-mails, any database-driven content, etc and we feel it is important to get this resolved immediately.

Our datacentre team has advised that the window for this work is approximately 1 hour and likely to be completed sooner, barring any unforeseen circumstances.

We appreciate your patience and understanding in this matter. We will update this announcement if there are any changes and once it has been resolved.

UPDATE: network issue has been identified and corrected. This notice is left in place for reference.

We are aware of an issue with intermittent connectivity loss to the Zeus server and have our system management team investigating at this time. The cause is currently unknown. Any information that becomes available will be added to this notice. We thank you for your patience and understanding.

We have received reports of e-mail delivery issues to Hotmail/Live/Outlook.com users and are currently in communication with Microsoft to determine the cause and get it resolved. We are working to get this resolved as soon as possible and thank you for your patience.

UPDATE: we've been informed by our system administrator that this has been resolved. If you continue to have isses, please contact our support team to debug further as your issue may be unrelated.

We've had some reports of PHP errors/issues with the Zeus server and have contacted our system administrator to investigate and resolve. This will be updated as soon as it has been. We appreciate your patience.

Due to an unforseen glitch in the software for the hard drives in the RAID array on the affected server, performance is currently degraded and it will need to be shutdown for approximately 1 to 2 hours around midnight CST time while the drives are replaced with newer ones that correct the issue. We appreciate your patience and understanding in wanting to give you the best quality service possible.

We have received reports of e-mail delivery issues to Hotmail/Outlook.com users and are currently in communication with Microsoft to determine the cause and get it resolved. We are working to get this resolved as soon as possible and thank you for your patience.

Due to an unforseen glitch in the software for the hard drives in the RAID array on the affected server, performance is currently degraded and it will need to be shutdown for approximately 1 to 2 hours later today while the drives are replaced with newer ones that correct the issue. We appreciate your patience and understanding in wanting to give you the best quality service possible.

It has been reported that there are intermittent connectivity issues with the Zeus server and speaking with our datacenter, this is due to an upgrade of the firmware on their Cisco routing equipment and it should be resolved shortly. Thank you for your understanding. If you have any questions, please don't hesitate to contact us!

Due to an unforseen conflict with the new network card installed earlier this evening, the Aries server is currently offline. All efforts are being made to get it back up as soon as possible. Thank you for your patience.

UPDATE: Yahoo has advised us that the new IP has now been added to their database to make deliveries to Yahoo users. This means that these messages should stop shortly and e-mail will be delivered as soon as it is sent. We will leave this notice up for the next 24 hours to monitor and ensure there are no further reports.

If you send an e-mail to a Yahoo user and get a bounce message similar to the following, do not be alarmed. After today's IP change to deal with the Hotmail problem, Yahoo will see a sudden surge (from their perspective) of e-mails coming from a new address and will limit them for the next 24 hours.

host mta7.am0.yahoodns.net [209.191.88.254]: 421 4.7.1 [TS03] All messages from 184.22.145.41 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

Despite what it says, it will succeed once their filters update and become familiar with the normal flow of e-mails to their users.

Due to the recent change to restore deliverability to Hotmail, cPanel is temporarily offline until they update our license information. This does not affect normal web or e-mail traffic in any way, and we expect the required update to be completed within the next few hours.

UPDATE: We are now migrating all accounts to a new IP range, as Hotmail support were unwilling to fix the problem on their end. Unfortunately, you may experience up to 60-90 minutes of service downtime as this change is taking place. We apologize for any inconvenience and thank you for your patience at this time.

UPDATE: We are still working with Hotmail on this, and it appears a resolution is near. However, if the problem is not resolved by Monday evening, we do have a contingency plan that can be enacted. This will only be done as a last resort, as it can result in some downtime of both web and e-mail services.

It has come to our attention that some clients hosted on the Aries server are experiencing difficulties e-mailing people whose addresses are hosted by Hotmail (mostly Hotmail.com and Live.ca). Generally, any attempt will result in a failure message with content similar to below

xxx@hotmail.com
SMTP error from remote mail server after MAIL
FROM: SIZE=4854:
host mx4.hotmail.com [65.54.188.72]: 550 SC-001 (BAY0-MC1-F39)
Unfortunately, messages from 66.197.155.85 weren't sent. Please contact your
Internet service provider since part of their network is on our block list.
You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.

We are aware of this issue and have been in contact with Hotmail to get it resolved. At this time, there is nothing to suggest that there was any spam being sent to them. This appears to simply be an overreaching block on their part and may be related to a new spam filter they introduced on Monday.

Updates will be posted here as information becomes available. We appreciate your patience.

UPDATE: We have been advised by AT&T that the problem has been resolved on their end, and e-mail delivery is now restored.

It has come to our attention that some clients hosted on the Aries server are experiencing difficulties e-mailing people whose addresses are hosted by AT&T and their various affiliates (mostly SBC and Prodigy). Generally, any attempt will result in a failure message with content similar to below

xxxx@sbcglobal.net
(ultimately generated from xxxx@xxxx.com)
SMTP error from remote mail server after MAIL FROM::
host sbcmx9.prodigy.net [207.115.37.23]: 553 5.3.0 nlpi152 DNSBL:ATTRBL 521< 66.197.155.85 >_is_blocked.__For_information_see_http://att.net/blocks

We are aware of this issue and have been in contact with AT&T to get it resolved. Updates will be posted here as information becomes available. We appreciate your patience.