website<\/a>), however this failed for some reason – we’re still awaiting further details on this.<\/p>\nThis impacted 3 of our 5 core switches on-site. Two were down for approximately an hour, but the final switch required a manual boot command via KVM.<\/p>\n
Most of the problems experienced by our clients related to this failed switch, as the DC responded slowly to our requests to physically inspect the switch, reboot, attach the KVM etc. We usually have quick responses from this specific datacenter, however we believe the slow response in this case related to the overwhelming amount of requests to them from other clients having similar problems.<\/p>\n
The issue with the final switch was resolved at 6PM CDT when the datacenter staff finally attached the KVM device to our switch, enabling us to immediately reboot and reinstate network connectivity to this specific rack, enabling our clients to access their servers again.<\/p>\n
Whilst this was going on, our senior technical team made preparations for a ‘plan B’, whereby we’d physically move the servers to another rack with a backup switch, however waiting for the DC staff would be much quicker than having to co-ordinate the physical moving of servers at such a time.<\/p>\n
We did all we could to resolve this incident as quickly as possible, but the slow response of the DC staff impacted on the resolution time.<\/p>\n
What will you be doing to prevent similar incidents?<\/h3>\n
To ensure we don’t experience issues like this again, we’ll be working with the datacenter on a plan going forward, and will also consider installing additional switches for maximum possible redundancy.<\/p>\n
We’re also looking at ways of better communicating with our clients regarding downtime incidents (scheduled and un-scheduled). This will likely be with the launch of a new status page making it easier to get notified.<\/p>\n
Any clients affected by these issues should contact our support department by submitting\u00a0a ticket<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"As some of you may know, we experienced a power outage at one of our US-based datacenters located in Dallas, TX. This affected those with dedicated servers at this location, along with clients on the following shared\/reseller servers: rs37.abstractdns.com rs41.abstractdns.com rs42.abstractdns.com server41.abstractdns.com server42.abstractdns.com This type of incident is extremely rare for us, and although we […]<\/p>\n","protected":false},"author":913,"featured_media":2686,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"\n
An Apology for Yesterday's Outage at our Dallas, TX Data Center - WebHostingBuzz US Blog<\/title>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"An Apology for Yesterday's Outage at our Dallas, TX Data Center - WebHostingBuzz US Blog\" \/>\n<meta property=\"og:description\" content=\"As some of you may know, we experienced a power outage at one of our US-based datacenters located in Dallas, TX. This affected those with dedicated servers at this location, along with clients on the following shared\/reseller servers: rs37.abstractdns.com rs41.abstractdns.com rs42.abstractdns.com server41.abstractdns.com server42.abstractdns.com This type of incident is extremely rare for us, and although we […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/\" \/>\n<meta property=\"og:site_name\" content=\"WebHostingBuzz US Blog\" \/>\n<meta property=\"article:published_time\" content=\"2015-07-16T17:32:47+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-07-16T17:37:54+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.webhostingbuzz.com\/wp-content\/uploads\/2014\/08\/L31A9240.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"5307\" \/>\n\t<meta property=\"og:image:height\" content=\"3538\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"2 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/#website\",\"url\":\"https:\/\/www.webhostingbuzz.com\/blog\/\",\"name\":\"WebHostingBuzz US Blog\",\"description\":\"Hosting, hosting, more hosting and a little of everything else\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.webhostingbuzz.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/blog.webhostingbuzz.com\/wp-content\/uploads\/2014\/08\/L31A9240.jpg\",\"contentUrl\":\"https:\/\/blog.webhostingbuzz.com\/wp-content\/uploads\/2014\/08\/L31A9240.jpg\",\"width\":5307,\"height\":3538},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/#webpage\",\"url\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/\",\"name\":\"An Apology for Yesterday's Outage at our Dallas, TX Data Center - WebHostingBuzz US Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/#primaryimage\"},\"datePublished\":\"2015-07-16T17:32:47+00:00\",\"dateModified\":\"2015-07-16T17:37:54+00:00\",\"author\":{\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/#\/schema\/person\/d56730c0ad0802b84f8caa335ac9e87f\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/\",\"url\":\"https:\/\/www.webhostingbuzz.com\/blog\/2015\/07\/16\/an-apology-for-yesterdays-outage-at-our-dallas-tx-data-center\/\",\"name\":\"An Apology for Yesterday’s Outage at our Dallas, TX Data Center\"}}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/#\/schema\/person\/d56730c0ad0802b84f8caa335ac9e87f\",\"name\":\"Adam\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.webhostingbuzz.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/457a2842ccd294191c84a4c7ade89057?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/457a2842ccd294191c84a4c7ade89057?s=96&r=g\",\"caption\":\"Adam\"},\"description\":\"Marketing Manager @ WebHostingBuzz. Heading up a few new and exciting projects - watch this space!\",\"sameAs\":[\"https:\/\/adamowen.co.uk\",\"https:\/\/twitter.com\/adamowenit\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/posts\/3052"}],"collection":[{"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/users\/913"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/comments?post=3052"}],"version-history":[{"count":7,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/posts\/3052\/revisions"}],"predecessor-version":[{"id":3059,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/posts\/3052\/revisions\/3059"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/media\/2686"}],"wp:attachment":[{"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/media?parent=3052"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/categories?post=3052"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.webhostingbuzz.com\/wp-json\/wp\/v2\/tags?post=3052"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}