Follow up to Network Outage on January 25, 2024 – Root Cause Analysis


Dear members of the Puget Sound community, Technology Services has prepared a Root Cause Analysis for the network incident that occurred on January 25th. The summary below provides additional technical details about the event as well as the course of action taken to restore services. ISSUE AND IMPACT: On January 25th at 2:30 p.m. the campus network began experiencing a level of instability that prevented users from accessing systems and services connected to the internet. During the duration of the network outage, services like Zoom Phone, Canvas, Google Workspace, and Okta were unreachable from within the campus network. ROOT CAUSE: After careful analysis of our event logs, we identified that the initial trigger of the event was caused by a change pushed by our Juniper Mist controller to a series of network devices. This change triggered a network routing convergence across a series of networks which were legacy artifacts of a security policy implementation that was never fully completed by Technology Services. Additionally, these changes triggered a series of events on our Palo Alto Firewalls ultimately causing the devices to no longer accept routing updates from the Juniper Core. At this point, both the Juniper Core and the Palo Alto firewall units were no longer able to successfully pass internet-bound traffic. SOLUTION: Technology Services reviewed all available logging information with our engineering team and determined that the legacy networks highlighted above as well as the associated firewall configuration should be removed from our network. Also, after careful review of the routing protocol used between the Juniper Core and the Palo Alto firewalls, it was decided that we should utilize a static routing protocol between the firewalls and the campus core network. The above stated solution has been scheduled for implementation on Thursday, February 8th at 5:00 a.m. A separate communication will be shared through the Service Announcements page detailing the events that will occur during the outage window. For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.
TS Service Desk Walk-In Support: Tech Center in Collins Library Phone Support: (253) 879-8585 Online Help: https://www.pugetsound.edu/osd (Logon Required) Email Support: servicedesk@pugetsound.edu

Dear members of the Puget Sound community,

Technology Services has prepared a Root Cause Analysis for the network incident that occurred on January 25th. The summary below provides additional technical details about the event as well as the course of action taken to restore services.

ISSUE AND IMPACT:

On January 25th at 2:30 p.m. the campus network began experiencing a level of instability that prevented users from accessing systems and services connected to the internet. During the duration of the network outage, services like Zoom Phone, Canvas, Google Workspace, and Okta were unreachable from within the campus network.

ROOT CAUSE:

After careful analysis of our event logs, we identified that the initial trigger of the event was caused by a change pushed by our Juniper Mist controller to a series of network devices. This change triggered a network routing convergence across a series of networks which were legacy artifacts of a security policy implementation that was never fully completed by Technology Services. Additionally, these changes triggered a series of events on our Palo Alto Firewalls ultimately causing the devices to no longer accept routing updates from the Juniper Core. At this point, both the Juniper Core and the Palo Alto firewall units were no longer able to successfully pass internet-bound traffic.

SOLUTION:

Technology Services reviewed all available logging information with our engineering team and determined that the legacy networks highlighted above as well as the associated firewall configuration should be removed from our network. Also, after careful review of the routing protocol used between the Juniper Core and the Palo Alto firewalls, it was decided that we should utilize a static routing protocol between the firewalls and the campus core network. The above stated solution has been scheduled for implementation on Thursday, February 8th at 5:00 a.m. A separate communication will be shared through the Service Announcements page detailing the events that will occur during the outage window.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help: https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Puget Sound Technology Services

Service Announcements from Technology Services

Follow up to Network Outage on January 25, 2024 – Root Cause Analysis