Forum Noamweb
Non sei collegato [Login - Registrati]
Vai alla fine

Versione per la stampa  
Autore: Oggetto: 10/08/2011 - Datacenter di Dallas - Problemi all'impianto elettrico
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 17:58
10/08/2011 - Datacenter di Dallas - Problemi all'impianto elettrico


Gentili clienti,

ci dispiace di comunicarvi che al datacenter di Dallas si stanno verificando dei problemi di natura elettrica che rendono irraggiungibili i server l ubicati.

Il personale del datacenter al lavoro per risolvere il problema, ci scusiamo per il disagio.

:saluto:




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 20:17


Per maggiore trasparenza, pubblichiamo la pagina di emergenza allestita dal personale del datacenter per tenere i clienti aggiornati.

Si trova all'URL http://www.colo4.com/

In questo modo, potrete avere notizie direttamente alla fonte.

:saluto:




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 22:58


Di seguito, riporto lo storico di quanto comunicato da Colo4 (gli orari sono quelli di Dallas):

12:40

Colo4 is currently experiencing a power issue.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 22:59


13:05

There has been an issue affecting one of our 6 service entrances. The actual ATS (Automatic Transfer Switch) is having an issue and all vendors are on site. Unfortunately, this is affecting service entrance 2 in the 3000 Irving facility so it is affecting a lot of the customers that have been here the longest.

The other entrance in 3000 is still up and working fine as well as the 4 entrances in 3004. Customers utilizing A/B should have access to their secondary link. It does appear that some customers were affected by a switch that had a failure in 3000. That has been addressed and should be up now.
This is not related to the PDU maintenance we had in 3004 last night. Separate building, service entrance, UPS, PDU, etc.

We will be updating customers as we get information from our vendors so that they know the estimated time for the outage. Once this has been resolved we also distribute a detailed RFO to those affected.

Our electrical contractors, UPS maintenance team and generator contractor are all on-site and working to determine what the best course of action is to get this back up.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 23:00


13:34

It has been determined that the ATS will need repairs that will take time to perform. Fortunately Colo4 has another ATS that is on-site that can be used as a spare. Contractors are working on a solution right now that will allow us to safely bring that ATS in and use it as a spare while that repair is happening.

That plan is being developed now and we should have an update soon as to the time frame to restore temporary power. We will need to schedule another window when the temp ATS is brought offline and replaced by the repaired ATS.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 23:00


14:53

Thank you for your patience as we work to address the ATS issue with our #2 service entrance. We apologize for the situation and are working as quickly as possible to restore service.

We have determined that the repairs for the ATS will take more time than anticipated, so we are putting into service a backup ATS that we have on-site as part of our emergency recovery plan. We are working with our power team to safely bring the replacement ATS into operation. We will update you as soon as we have an estimated time that the replacement ATS will be online.

Later, once we have repaired the main ATS, we will schedule an update window to transition from the temporary power solution. We will provide advance notice and timelines to minimize any disruption to your business.

Again, we apologize for the loss of connectivity and impact to your business. We are working diligently to get things back online for our customers. Please expect another update within the hour.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 23:04


16:46

Our team and electricians are working diligently to get the temporary ATS installed, wired and tested to allow power to be restored. As the ATS involves high-voltage power, we are following the necessary steps to ensure the safety of our personnel and your equipment housed in our facility.

Based on current progress the electricians expect to start powering the equipment on between 6:15 7:00pm Central. This is our best estimated time currently. We have thoroughly tested and dont anticipate any issues in powering up, but there is always the potential for unforeseen issues that could affect the ETA so we will keep you posted as we get progress reports. Our UPS vendor has checked every UPS, and the HVAC has checked every unit and found no issues. Our electrical contractor has also checked everything.

We realize how challenging and frustrating that it has been to not have an ETA for you or your customers, but we wanted to ensure we shared accurate and realistic information. We are working as fast as possible to get our customers back online and to ensure it is done safely and accurately. We will provide an update again within the hour.

While the team is working on the fix, Ive answered some of the questions or comments that have been raised:

1. ATSs are pieces of equipment and can fail as equipment sometimes does, which is why we do 2N power in the facility in case the worst case scenario happens.

2. There is no problem with the electrical grid in Dallas or the heat in Dallas that caused the issue.

3. Our website and one switch were connected to two PDUs, but ultimately the same service entrance. This was a mistake that has been corrected.

4. Bypassing an ATS is not a simple fix, like putting on jumper cables. It is detailed and hard work. Given the size and power of the ATS, the safety of our people and our contractors must remain the highest priority.

5. Our guys are working hard. While we all prepare for emergencies, it is still quite difficult when one is in effect. We could have done a better job keeping you informed. We know our customers are also stressed.

6. The ATS could be repaired, but we have already made the decision to order a replacement. This is certainly not the cheapest route to take, but is the best solution for the long-term stability.

7. While the solution we have implemented is technically a temporary fix, we are taking great care and wiring as if it were permanent.

8. Colo4 does have A/B power for our routing gear. We identified one switch that was connected to A only which was a mistake. It was quickly corrected earlier today but did affect service for a few customers.

9. Some customers with A/B power had overloaded their circuits, which is a separate and individual versus network issue. (For example, if we offer A/B 20 amp feeds and the customer has 12 amps on each, if one trips, the other will not be able to handle the load.)

As you could imagine, this is the top priority for everyone in our facility. We will provide an update as quickly as possible.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 10/8/2011 at 23:41


17:38

Power has been restored to the distribution gear from the temporary ATS. HVAC units are now all online, and we will be beginning the process of restoring power to UPSs soon, then PDUs, and then customer equipment.

We will update you as the other areas come online. Thank you again for your patience.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 01:11


The power has been restored fully and all customers should be up. If you are a customer and have not come online yet, please open a help ticket for us to handle directly.

In addition, we have deployed a team member to walk the data center and look for any cabinets not powered up. We will reach out to you and coordinate getting your equipment live for any that we observe in this process check.

We will provide customers with a more detailed update upon completion of our after-action review for this incident. Our first goal at this time is to ensure everyone is up safely and all connectivity is restored.
Thank you again for your patience.




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 01:11


Alcuni server sono online, mentre altri sono ancora irraggiungibili. Abbiamo segnalato in problema e siamo in attesa di risposta.



View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 05:12


Sono ancora down ZEUS ed HERMES, stiamo continuando a lavorare per riportarli online.



View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 12:20


Un aggiornamento relativo a ZEUS ed HERMES.

Il problema di ZEUS si sta rivelando di difficile soluzione. Il boot del sistema operativo sembra essersi corrotto e si sta tentando di recuperarlo. In caso di emergenza, stiamo trasferendo il backup di domenica su un altro server, in modo che si possa effettuare un restore qualora non si dovesse riuscire a far partire ZEUS.

Per quanto riguarda HERMES, il problema generale, ossia il server sta manifestando dei malfunzionamenti hardware per cui, molto probabilmente, i dischi verranno inseriti in un nuovo server.

Stiamo lavorando da ieri pomeriggio ininterrottamente, notte inclusa, per poter riportare online i due server prima possibile.

Grazie per la pazienza.

:saluto:




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 13:37


ZEUS tornato online, stiamo lavorando ancora su HERMES.



View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente
Shazan
Super Amministratore
*********


Avatar


Risposte: 1425
Registrato il: 5/10/2004
Utente offline


[*] Inviato il 11/8/2011 at 15:55


Anche HERMES tornato online.

Ricordo ai reseller che i loro piani sono coperti da garanzia di uptime con penale, pertanto, all'atto dell'emissione del prossimo resoconto, qualora l'uptime del server che ospita il proprio account risultasse inferiore al 99%, possono chiedere il credito in base alla tabella pubblicata all'URL http://www.noamweb.com/garanzie-uptime-garantito.shtml

Tutto risolto dunque, grazie per la pazienza :)




View User's Profile Visita la pagina dell utente Scorri tutte le risposte per utente

  Vai all'inizio

Powered by XMB 1.9.11
XMB Forum Software © 2001-2012 The XMB Group