The Hyper-V host node CLE02VMN02 is reporting ECC drive errors in one of the RAID sub units. This is a critical event and cannot be deferred for future maintenance cycle due to potential of data loss and/or corruption that can result if additional drives fail. Our service team will be dispatched to the data center to perform RAID maintenance starting 9:00PM EST 2/3/2010.  All virtual servers hosted on this node will be offline for the duration of this maintenance window. We sincerely apologize for the inconvenience.

 We will post additional information regarding this event as it becomes available.

Virtual machines on CLE02VMN02 node will be taken offline shortly. Please follow this thread for updates.

The SAN volume scan is 36% through, hopefully it will pickup pace. 

Volume scan is currently 47% complete, we are going to let it run for several more hours.

Volume scan completed, restarting Hyper-V Guest servers

We have scheduled another maintenance window for Thursday, February 4th at 10:00PM EST. We will be replacing failed drive units in the SAN array servicing CLE02VMN02 Hyper-V node.

As a precaution we will shutdown all guest VMs serviced by this SAN unit and perform disk swap in powered down state. Please follow this thread for updates.

I'm so frustrated. I don't even know you guys shut down the server today again as I didn't receive any email notice for each of your post. I just got the initial notice.

when will be completed?

 Chris

SAN drive has been replaced and RAID volume reinitialized. We are currently restarting all Hyper-V guest servers.

We will be performing a driver and firmware upgrade today (Sunday, March 28th 2010 starting 10AM EST) on the 3Ware 9650 RAID controller as a follow-up measure. The new firmware release addresses number of issues we have experienced with this particular RAID controller during the last several months.

 As a precaution we have taken full backups of all data. The upgrade process should take approximately 15 to 20 minutes during which time we will have to reboot the server.

 We will be shutting down guest VMs shortly, please follow this thread for updates.

 Maintenance completed, all affected VMs are currently restarting.