in

CLE02 Datacenter Outage

Last post 01-15-2011 12:49 PM by skozyuk. 19 replies.
Page 1 of 2 (20 items) 1 2 Next >
Sort Posts: Previous Next
  • 01-12-2011 4:15 PM

    • skozyuk
    • Top 10 Contributor
    • Joined on 10-28-2007
    • Westlake, OH
    • Posts 401
    • Points 1,149

    CLE02 Datacenter Outage

    We have been notified by our upstream carrier regarding outage that is affecting our entire CLE02 network segment. We do not have ETA on resolution at this moment, however we were assured all engineers are working on restoring service.
    • Post Points: 0
  • 01-12-2011 4:17 PM In reply to

    • jclay
    • Top 25 Contributor
    • Joined on 08-23-2008
    • Sandpoint, ID
    • Posts 6
    • Points 49

    Re: CLE02 Datacenter Outage

    OUCH!  When it rains - it pours.  Have you determined if this is weather related or not and what kind of time frame we are looking at?  I am unable to get to email and a couple of site.  Also, it appears yhour support site is in this datacenter because I am unable to get to it.

     Jim

    Jim Clay
    www.jimclay.com
    • Post Points: 0
  • 01-12-2011 4:19 PM In reply to

    • skozyuk
    • Top 10 Contributor
    • Joined on 10-28-2007
    • Westlake, OH
    • Posts 401
    • Points 1,149

    Re: CLE02 Datacenter Outage

    We do not know much right now other than our NOC screen is lit up with alarms. This issue is above us, and is affecting the carrier that we lease the cage space for our racks.
    • Post Points: 0
  • 01-12-2011 5:10 PM In reply to

    • skozyuk
    • Top 10 Contributor
    • Joined on 10-28-2007
    • Westlake, OH
    • Posts 401
    • Points 1,149

    Re: CLE02 Datacenter Outage

    We have received an updated from carrier. It appears one or more of their UPS units have tripped and caused an outage. We do not know yet if our equipment has been affected, we are sending our team to monitor the situation on-site.
    • Post Points: 0
  • 01-12-2011 5:52 PM In reply to

    • skozyuk
    • Top 10 Contributor
    • Joined on 10-28-2007
    • Westlake, OH
    • Posts 401
    • Points 1,149

    Re: CLE02 Datacenter Outage

    Looks like the power is back on. We are seeing most servers have recovered and are now online. Our team has been dispatched to the data-center to physically check the all servers.
    • Post Points: 0
  • 01-12-2011 6:43 PM In reply to

    • wplante
    • Top 50 Contributor
    • Joined on 01-12-2011
    • Posts 3
    • Points 0

    Re: CLE02 Datacenter Outage

    As of 7:03pm I'm showing an "under construction" notice. I understand there are issues but I wanted someone to know that in my case, it's still ongoing. Thanks
    • Post Points: 0
  • 01-12-2011 8:26 PM In reply to

    • wplante
    • Top 50 Contributor
    • Joined on 01-12-2011
    • Posts 3
    • Points 0

    Re: CLE02 Datacenter Outage

    As of 8:46, I'm back up. Thanks.
    • Post Points: 0
  • 01-12-2011 8:41 PM In reply to

    Re: CLE02 Datacenter Outage

    I have two domains on CLE02WEB04 that have been dead for the last 6 hours.
    • Post Points: 0
  • 01-12-2011 8:58 PM In reply to

    • wplante
    • Top 50 Contributor
    • Joined on 01-12-2011
    • Posts 3
    • Points 0

    Re: CLE02 Datacenter Outage

     I spoke too soon. I'm down AGAIN!

    • Post Points: 0
  • 01-12-2011 9:35 PM In reply to

    Re: CLE02 Datacenter Outage

    It's like a yo-yo - up, now back down again - all of them. Guys... please, you're putting me out of business with over 100 sites on your services I'm in the crosshairs of business owners that make their daily livihood off their web shopping carts. If they are down it means hundred of dollars in lost revenue an hour. You think they are not ready to string me up? They don't care about my explainations, those were worn out week ago, before the disaster last week. 

     I need you to be in the hosting business with down time being the exception and not the rule. Yes, frustrated doesn't touch it.

    • Post Points: 0
  • 01-12-2011 10:33 PM In reply to

    Re: CLE02 Datacenter Outage

     It's nearly 11pmEST - massive outages and even your support site is down - please post here to let us know you know. Some faint straw of hope...

    • Post Points: 0
  • 01-12-2011 10:35 PM In reply to

    • jryan
    • Top 10 Contributor
    • Joined on 01-07-2011
    • Posts 16
    • Points 79

    Re: CLE02 Datacenter Outage

     Here's your mantra (from the iHost Advantages on your site) for tomorrow's meeting. Just ignore the typo...

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    5. Uptime

    Despite competitor claims, few DNN hosting vendors understand how to plan for disaster recovery. Servers, like any other equipment, fail. IHost knows has implemented a solid disaster recovery plan.

    Here are some of the highlights:
    • We employ RAID technology on all servers; we can hot-swap failed hard drives without powering down servers.
    • We back up your data off site every day, with a minimum 15-day retention. This means your data is safe from hackers or catastrophic events like fires or flood. In minutes, we can restore an entire server or individual files, providing you a security blanket in case of corruption or malicious attack.
    • We perform proactive maintenance and testing to detect impeding problems and address them before they cause downtime.

     

    • Post Points: 0
  • 01-12-2011 11:07 PM In reply to

    Re: CLE02 Datacenter Outage

    The mail server seems to have been down for a couple of hours now.  Can't submit a support ticket!

    • Post Points: 0
  • 01-13-2011 1:29 AM In reply to

    • skozyuk
    • Top 10 Contributor
    • Joined on 10-28-2007
    • Westlake, OH
    • Posts 401
    • Points 1,149

    Re: CLE02 Datacenter Outage

    I apologize for not posting updates, it's been very hectic last 10 hours. All I want to say right now, all services are operational at this point. The situation we are experiencing is beyond our sphere of control, however it is affecting us and others.

     We will post more information when it becomes available, in the meantime please read my blog here

    http://community.ihostasp.net/blogs/from_the_trenches/archive/2011/01/13/when-it-all-fails.aspx

     

    • Post Points: 0
  • 01-13-2011 8:52 AM In reply to

    Re: CLE02 Datacenter Outage

    Thanks for the post and blog. We are all feeling exactly the same pain as our team was scambling most all night as well over this. While this type of thing is isolated and certainly not at your door step, it would not have been half as bad had not not come on the heals of last weeks all day outage and the previous weeks outages and then a few weeks ago have servers go down literally 4 out of 5 working days at the start of each morning. We come into the office now wondering who's going to be out of commission today by default. It hadn't been that way the previous year.

     It's not the one time here, it's the string of things almost daily and certainly weekly. I leaned 100% of my hosting business in iHost, often adding 2 or 3 sites a week. I see now I have to "mutual fund" approach it, shifting new business elsewhere, moving high demand ecommerce clients so I can get rid of the scorn and lack of confidence in us for choosing to host then with you. There are serious issues that need to be addressed and evidenced by lack of server down time. Moving over 100 sites would kill our productivity and I hope we are not forced to. 

     Please address the underlying issues, infrastructure, whatever. I am pulling for you to do so - It's kiling Clint and Kim, I can hear it in their voice.

    • Post Points: 0
Page 1 of 2 (20 items) 1 2 Next >
2004 - 2008 IHOST, LLC