Microsoft has recently released a critical security update that was deployed onto our Cle02web06 web hosting server. This security patch is causing technical issues with the IIS services on this web hosting server. Our 2nd level technicians are aware of this issue and working to roll the web hosting server back to before this cirtical update was deployed as quickly as possible. We apologize for the inconvenience this may cause and appreciate your patience as our technicians work to correct this issue as quickly as possible. We will post updates to this issue here as they are available.

We have opened priority A (2 hour SLA) case with Microsoft and are working with the Microsoft engineers right now to get IIS6 service running on CLE02WEB06. It appears one of the recently released server update has cause IIS Admin service to fail. We anticipate Microsoft Windows engineers can help us resolve this issue ASAP.

We have gotten IIS service metabase rebuilt with help of Microsoft engineer. Right now we are running permissions rebuild on all domains hosted on CLE02WEB06.

The control panel server will be disabled to prevent unintended interference while the permissions are being rebuilt.

We are currently rebuilding application pool identities for all sites on CLE02WEB06 server. Unfortunately this process requires us to go through each IIS site manually and update application pool identity, thus slowing down our progress.

The Microsoft patch has managed to corrupt our IIS server metabase and caused application pool identities to reset. We are going to be working through the night and will get all domains functional ASAP. I will be posting updates periodically, so stay tunned and we will get through this mess.

We are currently making a full backup of all websites stored on CLE02WEB06 as a precaution. The next step will involve running scripts that will rebuild entire IIS tree to its original state. Once the IIS sites and corresponding application pools have been rebuilt, the service on CLE02WEB06 server resume.

The recovery process has started. Sites should become available on-line roughly one every 2 minutes in the alphabetical order.

So far we have activated approximately 10% of sites on CLE02WEB06 server. At this rate ETA to full recovery is expected with-in 4 hours.

We have activated approximately 50% of the sites hosted on CLE02WEB06. Our ACL rebuild tool has to traverse each domain and update individual files with a correct ACLs to match the process identity of the IIS work process the site is running in. This is a time consuming task due to large number of files it has to touch.

In addition we manually verify each domain  to make sure it is properly configured and operational. Overall, we are satisfied with the results. Domains are steadily being rebuilt although at a slower pace than what I have initially anticipated.  I must admin, this is first time in four years we have had to do full server recovery on this scale. The disaster recovery plan we have in place is working, however there are few things will certainly need adjustments in the future to facilitate faster server rebuild.

We are approximately 70% complete rebuilding the domains on the Cle02web06 web server at this time and are working diligently to have all domains rebuilt as soon as possible.

We are approximately 92% complete rebuilding the domains on the Cle02web06 web server at this time and will have all domains on this web server plus the Helm control panel operational very shortly.

Our technicians have completed the critical recovery tasks on the Cle02web06 web hosting server and all domains should be operational again at this time.