Approximately at 7AM EST we have detected an unscheduled power reset event at one of our server racks affecting following servers; CLE02WEB09 CLE02WEB10 CLE02WEB11 CLE02WEB12 CLE02WEB13 CLE02WEB14 CLE02SQL03 (mssql09.ihostasp.net) CLE02SQL04 (mssql10.ihostasp.net) CLE02VMN02 This issue is likely caused by a faulty PDU (power distribution unit). We have attempted to bring servers back online however the power has tripped again. To prevent any damage to the servers, we will not turn on power until our engineers have replaced the PDU unit that provides power to the servers mentioned above. We have dispatched an engineer with a replacement PDU to the data-center, we anticipate the issue being resolved with in 2 hours. Please follow this thread for updates. We appreciate your patience and understanding.

Power has been restored to a critical switch which was effected by the service issue on this power distribution unit.  The restoration of this switch has restored service to some servers.  The new PDU is being installed and should be complete within approximately a hour.  All services should be restored once the installation of the PDU is complete.

We have replaced the PDU on the rack, it appears electrical arching has caused part of the 30 Amp twist-lock plugin to melt and resulted in loss of power. Currently we have powered up all servers, however MSSQL09 & MSSQL10 database servers have suffered database corruption that are preventing MSSQL database service from starting up. We are currently focused on restoring database service. Please follow this thread for updates.

We have started database service on MSSQL10 server. Currently working on restoring RAID on MSSQL09 database server.

Did the power outtage also affect mssql10? That SQL server is down right now.

Clearly these are still down at this moment :

 

These are still not responding: CLE02WEB10  CLE02WEB08 CLE02WEB08 CLE02WEB09

MSSQL09 Database server is back online. All services should resume normal operation.

 My site CLE02WEB06  and  MSSQL09 is still down and giving server error. I can however log on to my database with SQL server

Same here, CLE03 server still not responding.

Since power came back on, 3 of my 20 sites now show the DotNetNuke Community logo along with the message "This site is currently Unavailable  Please check back later", which is basically the /Install/UnderConstruction.htm page.

2 of the sites are pre-propagation sites on web31 and web35. The other on 207.54.144.30.

Does anyone else have this problem?

Would someone at ihostasp.net kindly acknowledge that you are still looking at this! If you think it is fixed - it isn't. Your recovery actions have altered 3 of my sites.

IF YOU CAN'T FIX THIS, KINDLY LOOK UP MY SUPPORT TICKETS AND RESTORE MY LAST BACKUPS FOR THESE SITES NOW !

 

Please update this space for us nervous customers - numerous site have 6 month old data. I have about 7 at last count all the sites restored to early 2010 status. PLEASE let us know that you can recover our sites and all the updates. Just tell us the truth. Understand if it 100's of sites and takes time to get them all. Just want to know you are in the process of doing it.

Ditto!  My site has data from 5/25/10.

 

 

We are aware of issues impacting users with databases hosted on MSSQL09 database server. The electrical fire that occurred on the main AC PDU circuit this morning has caused some servers RAID to corrupt entire file-system due to electrical surges. We have rebuilt server raid systems using specialized tools and reserve mirror disks to get back online as soon as possible. It appears some MSSQL transaction logs that were being accessed at the time of the electrical surge became corrupted and unrecoverable. When we finally have replaced the failed PDU and powered back the server, MSSQL databases with damaged transaction logs rolled back to a baseline state. The good news, we have full daily backups of all databases up until AM hours of 1/6/11. Our support team has been mobilized and is restoring individual customer databases to their original state. We will restore all affected databases, however this will take some time since we are dealing with possibly several hundred of databases that need to be restored on individual basis. Please, if haven't done so, open a support ticket here https://support.ihostllc.net and include your domain name and database name from the web.config file in the support ticket. If you have multiple sites that are affected, please include them all in that support ticket As result of this outage our phone system is overloaded with calls, please try not to call us every 1/2 hour as it only slows down the progress on getting things back to normal. I sincerely apologize for this unforeseen event, hopefully we will get bulk of the databases fixed by the end of the day tomorrow.

Great news, thank you for taking the time to update this here.