Service Announcement

unscheduled Outage

Magnum Clusters inaccessible

  • Outage start: Tue, 3 Jun 2025 12:00a.m. (UTC), Outage end: Sat, 7 Jun 2025 12:35a.m. (UTC)

We have received reports that some Magnum clusters are not accessible. Currently, this seems to be only affecting newer clusters (>v1.30) which uses the CAPI driver and the OVN backend for loadbalancers.

We have traced this down to possible corruption the OVN database and are coming up with a fix.

Current Status:

Resolved

Severity:

Significant

Last modification at:
Sat, 7 Jun 2025 12:35a.m. (UTC)

Updates

  • Resolved

    All loadbalancers should be fixed. If you are still experiencing issues, please open a ticket to our helpdesk.


    Update posted 3 weeks, 3 days ago.

  • Fixed

    We have a fix for the rest of OVN Loadbalancers and will be rolling this out gradually over the weekend.


    Update posted 3 weeks, 4 days ago.

  • Progressing

    We have fixed a number of OVN load balancers that services Kubernetes API for Magnum clusters. Other OVN loadbalancers may still be affected. For clarity, there are a few types of OVN LBs (OVN LBs are identified by the provider 'ovn', instead of 'amphora')

    1. API LB for Magnum CAPI clusters - this has been fixed
    2. Service LB for Magnum CAPI clusters - ongoing
    3. API and Service LB with provider 'OVN' for Magnum Heat clusters
    4. All other OVN LB


    Update posted 3 weeks, 6 days ago.

  • Progressing

    We have identified a workaround for this issue. Deleting the API loadbalancer will allow the backend to re-create one, which will be working.

    We will be rolling out this fix gradually to all the clusters. You may want to try the same if you are keen to get your cluster working again.

    The root cause of this remains under investigation.


    Update posted 4 weeks ago.

  • Identified

    Outage started


    Update posted 4 weeks ago.