Campus Network Outage Summary – Friday, January 24

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has resolved the following incident. Details on the issue are available below:

Puget Sound Network: Campus Network Outage

Friday, January 24 | 3:13 p.m. – 10:30 p.m.

Outage Summary:
On Friday, January 24th, at 3:13 p.m., Technology Services observed a campus-wide network outage. Due to the scale of the outage and the campus community's inability to communicate via Zoom or email, our Emergency Response Group elected to issue an Everbridge alert to notify the campus. The team undertook several remedial actions to restore network connectivity, and a follow-up Everbridge alert was issued at approximately 5:15 p.m., indicating the outage had been resolved. Unfortunately, at around 5:50 p.m., the team observed a recurrence of the same network behavior seen during the initial outage. At this point, the campus network became unavailable again.

Following a thorough review of event logs and identifying a viable remediation method, our network engineers, with the support of Juniper, implemented a configuration change to the core and distribution layers of the network. This action stabilized and restored connectivity at approximately 9:30 p.m. We apologize for the disruption caused by this outage and appreciate your understanding as we worked to restore services.

Technical Details:
During the initial onset of the outage, our engineers reviewed network traffic patterns and identified what appeared to be a network loop causing the outage. A network loop, in its simplest form, occurs when data travels in a continuous circle without any mechanism to stop it. This can happen if a data port is connected to another port within the same network, creating an infinite data loop, or if a configuration change inadvertently causes traffic to route in a circular path. While protocols like Spanning Tree Protocol (STP) exist to prevent loops from disrupting an entire network, loops can sometimes originate in areas where STP is not applicable.

The team conducted a comprehensive review of all changes made in the two weeks prior to the incident to determine if any recent network modifications had caused the outage. We are confident that no changes performed by our team were responsible. Additionally, we found no evidence of a cyber event contributing to the outage.

Unfortunately, a series of factors led the team to a false positive regarding the root cause of the outage. During the event, our logging system was overwhelmed with excessive and unnecessary details, which obscured the underlying issue and made it difficult to diagnose the cause. The campus network's routing fabric uses Border Gateway Protocol (BGP) to determine the best paths for data to travel. BGP relies on a supplementary protocol, Bidirectional Forwarding Detection (BFD), to detect routing failures. Upon further analysis of the logs, we discovered an unknown issue causing BFD to continuously reset routing across the campus network, rendering it virtually impossible for devices to communicate.

After consulting with Juniper support, our team confirmed it was safe to disable BFD temporarily to restore network connectivity. Once this configuration change was made, network traffic stabilized immediately, and services were restored. Additionally, packet captures of the network have been obtained to further analyze the outage and identify the root cause of the BFD resets. Once a fix has been identified, we will implement the necessary adjustments and re-enable BFD during the next available routine change window.

Our teams continued to work through the weekend to address residual disruptions caused by the outage and ensure the stability of our data center operations.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help: support.pugetsound.edu 

Email Support: servicedesk@pugetsound.edu

Network Outage: 12/12 Approximately 3:15PM – 4:15PM

This logo identifies the Technology Services department within the University of Puget Sound


Dear members of the Puget Sound community,


On Thursday, December 12 at approximately 3:15 p.m. Technology Services (TS) was made aware of a network outage affecting both Collins Memorial Library and the Facilities Services Complex. While we are still working to identify the root cause of the issue, we believe the network instability may be related to the staging of software for our network infrastructure ahead of the holiday break.


During this time, users would have experienced an inability to access any of the following services:

  • Internet Access
  • Wired and Wireless Network Connectivity (WiFi)
  • Desk Phones Restarting or Dropping Calls

If you experienced any of the outages above, please contact the Service Desk at 253.879.8585 so we can identify those who were affected.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Technology Services Early Closure Notice – Today Dec 12 @ Noon

This logo identifies the Technology Services department within the University of Puget Sound


Dear members of the Puget Sound community,


On Today, December 12 at 12 p.m. Technology Services (TS) and Print and Copy Services (PCS), will be closing early for an on-campus, team holiday gathering.


During this time, limited services will be available from the following areas:

  • Service Desk Support
  • Print and Copy Services (PCS)
  • Media Services

In case of a technical emergency, please contact Security Services at 253.879.3311 and they will reach the appropriate resource within TS. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

AlertUs Desktop Notifications

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

The University of Puget Sound has partnered with Alertus notification systems to enhance the ability to communicate with the community in an emergency. As part of our ongoing effort to improve our emergency preparedness and provide more comprehensive safety and security measures Technology Services and Security Services have selected Alertus to enhance our emergency notification system. For more information on campus emergency management efforts, visit the emergency management page.

We are in the process of combining our existing Everbridge Emergency Communications product with Alertus, creating a robust and redundant communication system. After extensive testing, Alertus will be automatically installed on all university owned Mac and Windows computers. Once installed, you will receive emergency notifications through Everbridge in the manner you have selected and receive an Alertus announcement on your university computer. To remove the message from your screen, simply acknowledge the alert.

Alertus will run on both Mac or PC and, once installed, automatically connects to our emergency notification system. 

Please note the Alertus software will only issue emergency notifications on university issued devices if you are on campus and connected to the university wired or wireless network. To ensure proper installation, make sure all computer system upgrades are installed.

Downloading and installing Alertus is quick, easy and takes just a few minutes. Technology Services will begin remotely installing the software on tomorrow (06/06/2024). Once installed, the software runs in the background and you can verify it is connected by checking the icon in the System Tray (on Windows) or Menu Bar (on a Mac). If you are on a Windows PC the installation will be entirely silent and will not require any user input. However, if you are on a Mac you will likely see a small notification appear in the notification center indicating a background application has been installed. There is no further action necessary to complete the installation.

To learn more about Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Login Required)

Email Support: servicedesk@pugetsound.edu