Category: General Announcement
HCC's datacenter at PKI in Omaha suffered an unexpected power outage the morning of Friday, Oct 19th during a preventative maintenance window.
This type of maintenance has occurred without issue many times in the past and requires the datacenter UPS (battery backup) be bypassed meaning all equipment relies directly on city power. While the bypass was in place there was an issue with the city power feed which caused many servers to reboot unexpectedly and various pieces of networking to fail.
HCC staff has worked throughout the day to restore services and believes we have done so at this time. All services hosted at PKI were affected including:
- ANVIL: Many VMs hosts were rebooted including the instances running on those hosts. Please check your instances and contact hcc-support@unl.edu with your instance ID if you have any problems.
- CRANE / TUSKER : Running jobs were killed and users should check their /home and /work files that may have been open or in the process of being written. Files being written during the power outage are likely lost or corrupted.
- COMMON Filesystem : Users should check their files exist and are accessible. Files being written during the power outage are likely lost or corrupted.
This is the first major power issue at this datacenter in a very long time and we will investigate and take any possible actions to prevent it from happening again. At this time it appears to have simply been a very unfortunate coincidence of being off battery power while the main power feed had an unexpected failure.
Please contact hcc-support@unl.edu with any questions or issues resulting from this outage.
show details...
This type of maintenance has occurred without issue many times in the past and requires the datacenter UPS (battery backup) be bypassed meaning all equipment relies directly on city power. While the bypass was in place there was an issue with the city power feed which caused many servers to reboot unexpectedly and various pieces of networking to fail.
HCC staff has worked throughout the day to restore services and believes we have done so at this time. All services hosted at PKI were affected including:
- ANVIL: Many VMs hosts were rebooted including the instances running on those hosts. Please check your instances and contact hcc-support@unl.edu with your instance ID if you have any problems.
- CRANE / TUSKER : Running jobs were killed and users should check their /home and /work files that may have been open or in the process of being written. Files being written during the power outage are likely lost or corrupted.
- COMMON Filesystem : Users should check their files exist and are accessible. Files being written during the power outage are likely lost or corrupted.
This is the first major power issue at this datacenter in a very long time and we will investigate and take any possible actions to prevent it from happening again. At this time it appears to have simply been a very unfortunate coincidence of being off battery power while the main power feed had an unexpected failure.
Please contact hcc-support@unl.edu with any questions or issues resulting from this outage.