Updates
-
Completed
Looks like this was less disruptive than first thought. The network between the storage and the VMs do not look to have been badly affected. Network logs show some unusual peaks, but minimal packet loss. Broadcast traffic was amplified by the loop between the two hypervisors involved in the change. This did not get to a level that disrupted the internal network, but looks to have flooded the private network that connects the core switches to the router to the outside world. This affected connections from Melbourne Core Services infrastructure, to the Auckland UnderCloud VMs. Hence the trouble with the MQ and DB, and with users dashboard interactions. It also disrupted connections to/from the internet, for user VMs.
Update posted 2 months ago. -
Investigating
Investigating
Update posted 2 months ago.