Service Announcement

unscheduled Outage

OVN Networking issue impacting instance launch

  • Outage start: Thu, 13 Nov 2025 12:30a.m. (UTC), Outage end: Fri, 14 Nov 2025 12:21a.m. (UTC)

A networking issue has been identified in the Nectar Cloud. The system that manages virtual networks (OVN) is having trouble coordinating its control nodes, which is causing delays or failures when launching new instances in some availability zones.

Nectar cloud team is working on, more update will come soon.

Current Status:

Resolved

Severity:

Severe

Last modification at:
Fri, 14 Nov 2025 12:31a.m. (UTC)

Updates

  • Resolved

    All services continue to operate normally, and the previously reported issue has been fully resolved. Instance creation and API calls related to instance networking are functioning as expected. We will maintain ongoing monitoring to ensure stability.


    Update posted 23 hours ago.

  • Fixed

    We believe we have found the cause of the issue and fixes have been applied. Services appear stable and our tests are passing.
    Instance creation and API calls relating to instance networking should now be working again.
    We will continue to monitor and provide further updates.


    Update posted 1 day, 13 hours ago.

  • Progressing

    We have rebuilt our OVN DB, and nova/neutron API is back since around 17:30 pm.

    But Instance launching at some sites is still not working.


    Update posted 1 day, 15 hours ago.

  • Progressing

    Our OVN database is broken, which caused the Nova/Neutron API not working.
    We are still trying to fix the broken OVN DB.


    Update posted 1 day, 18 hours ago.

  • Progressing

    The OVN Database cluster cannot correctly elect the leader which causes the OVN issue.

    We have temporarily make the leader back then we start to see the tempest tests passed.

    The issue is not yet fixed, we are still working on it.


    Update posted 1 day, 22 hours ago.

  • Investigating

    Outage started


    Update posted 1 day, 23 hours ago.