|
Bad day for my host.. if you had issues accessing AtariForge or the
SpareMiNT servers. Rob This morning one of our data centers, experienced major multiple critical power systems failures as follows: * At approximately 0500 Eastern Time, we experienced a total loss of utility power, our backup generators and UPS systems functioned as designed, and this data center was functioning as it should. * At approximately 0700 Eastern Time, the local utility restored power and our building was automatically switched back to utility power. * At approximately 0710 Eastern Time, the local utility power dropped yet again, at this juncture one of our backup generators experienced a failure in both of its two starter motors. Our UPS systems functioned as designed, but exhausted their battery capacity, dropping power to most of the systems in this data center. * At approximately 0730 Eastern Time, the local utility restored power again. * At approximately 0845 Eastern Time, the local utility power dropped yet again, we were in the process of changing the starters on the generator, and at 0930, we were able to re-start our emergency generator. * Although our local utility claims to have corrected their power issues which precipitated this issue, we are still running on generator power until we can perform a coordinated and monitored cut-over back to utility power. This is scheduled to happen at 1430 Eastern Time, as we want to ensure the technicians from the utility are on-site for this event, and will remain on-site to monitor this situation moving forwards. * All UPS Systems are back online, and our emergency generator is functioning as it should be. We apologize profusely for this issue, and can assure you we wish it did not happen. We have been working on an increased staffing level since this incident occurred to ensure we are doing everything possible to address any outstanding issues. |