Service Announcement

unscheduled Outage

Emergency shutdown of Auckland VMs and servers due to a Datacentre Cooling Issue

  • Start: Wed, 29 Jul 2026 2:11a.m. (UTC), End: Thu, 30 Jul 2026 4:52a.m. (UTC)

Due to a cooling issue, the University needs to initiate an emergency shutdown of the servers in the data centre.
This will mean that you will not be able to access virtual compute, including virtual desktop services, until the issue is resolved.
Please reach out through nectarsupport@auckland.ac.nz if you have specific concerns.
We will provide an update when we know more. Thank you for your patience!

Current Status:

Completed

Severity:

Severe

Last modification at:
Thu, 30 Jul 2026 4:52a.m. (UTC)

Updates

  • Fixed

    All VMs and Cloud services at Auckland are back up and everything seems operational and stable. There will still be some things to test and verify tomorrow, but the core outage is resolved. Please reach out via a Nectar service request if you notice anything that doesn't look right. Again, thanks for your patience!
    Update posted 1 day, 15 hours ago.

  • Progressing

    The Auckland node is now available. There is a small list of edge case VMs that we're still working on, but the vast majority of instances are now powered on and should be available. Users can now also create and modify existing resources as normal. We will have a final update when the few remaining VMs have been powered on successfully.
    Update posted 1 day, 16 hours ago.

  • Progressing

    Good progress is being made, and we should be in the final step in bringing the Auckland node back online in the Nectar Cloud, but this process does take a while to re-sync. We're expecting perhaps another 30 mins or so to complete. At this point we'll do some quick verification testing, and then turn all instances back on. Next update will be no later than 4:00, and hopefully earlier.
    Update posted 1 day, 18 hours ago.

  • Progressing

    We have brought most of our supporting systems online successfully, and are now working with the Nectar Core Services team to restore our connection to the federated cloud. Once this is complete and we can verify a working state, we will power user VMs back on.

    Still no firm ETA, but we're hoping to start powering VMs back on in the next 60-90 minutes. We will update again on or before 2:30. Thanks for your patience.
    Update posted 1 day, 19 hours ago.

  • Progressing

    We have gotten clearance from UoA to start turning equipment back on. We have a team member en route to the data centre to begin this process. We need to do a slow and controlled resumption of business as usual and perform health checks on the systems as we go.

    We plan to provide another update about the progress and status by 13:00 today.
    Update posted 1 day, 22 hours ago.

  • Progressing

    Vendor onsite has restored two of the three chillers at the datacenter, however there is no redundancy and there still is not a root cause established. UoA have asked us to keep Nectar infrastructure powered off tonight until the third chiller is repaired.
    Update posted 2 days, 14 hours ago.

  • Identified

    Shutting down hypervisors now.
    Update posted 2 days, 18 hours ago.