Service Announcement

unscheduled Outage

Magnum Clusters inaccessible

  • Outage start: Tue, 3 Jun 2025 12:00a.m. (UTC), Outage is ongoing

We have received reports that some Magnum clusters are not accessible. Currently, this seems to be only affecting newer clusters (>v1.30) which uses the CAPI driver and the OVN backend for loadbalancers.

We have traced this down to possible corruption the OVN database and are coming up with a fix.

Current Status:

Progressing

Severity:

Significant

Last modification at:
Wed, 4 Jun 2025 2:48a.m. (UTC)

Updates

  • Progressing

    We have fixed a number of OVN load balancers that services Kubernetes API for Magnum clusters. Other OVN loadbalancers may still be affected. For clarity, there are a few types of OVN LBs (OVN LBs are identified by the provider 'ovn', instead of 'amphora')

    1. API LB for Magnum CAPI clusters - this has been fixed
    2. Service LB for Magnum CAPI clusters - ongoing
    3. API and Service LB with provider 'OVN' for Magnum Heat clusters
    4. All other OVN LB


    Update posted 1 day, 11 hours ago.

  • Progressing

    We have identified a workaround for this issue. Deleting the API loadbalancer will allow the backend to re-create one, which will be working.

    We will be rolling out this fix gradually to all the clusters. You may want to try the same if you are keen to get your cluster working again.

    The root cause of this remains under investigation.


    Update posted 2 days, 7 hours ago.

  • Identified

    Outage started


    Update posted 2 days, 13 hours ago.