Dear members of the Puget Sound community, Technology Services (TS) has resolved the following incident. Details on the issue are available below: Puget Sound Network: Campus Network Outage Friday, January 24 | 3:13 p.m. – 10:30 p.m. Outage Summary: Following a thorough review of event logs and identifying a viable remediation method, our network engineers, with the support of Juniper, implemented a configuration change to the core and distribution layers of the network. This action stabilized and restored connectivity at approximately 9:30 p.m. We apologize for the disruption caused by this outage and appreciate your understanding as we worked to restore services. Technical Details: The team conducted a comprehensive review of all changes made in the two weeks prior to the incident to determine if any recent network modifications had caused the outage. We are confident that no changes performed by our team were responsible. Additionally, we found no evidence of a cyber event contributing to the outage. Unfortunately, a series of factors led the team to a false positive regarding the root cause of the outage. During the event, our logging system was overwhelmed with excessive and unnecessary details, which obscured the underlying issue and made it difficult to diagnose the cause. The campus network's routing fabric uses Border Gateway Protocol (BGP) to determine the best paths for data to travel. BGP relies on a supplementary protocol, Bidirectional Forwarding Detection (BFD), to detect routing failures. Upon further analysis of the logs, we discovered an unknown issue causing BFD to continuously reset routing across the campus network, rendering it virtually impossible for devices to communicate. After consulting with Juniper support, our team confirmed it was safe to disable BFD temporarily to restore network connectivity. Once this configuration change was made, network traffic stabilized immediately, and services were restored. Additionally, packet captures of the network have been obtained to further analyze the outage and identify the root cause of the BFD resets. Once a fix has been identified, we will implement the necessary adjustments and re-enable BFD during the next available routine change window. Our teams continued to work through the weekend to address residual disruptions caused by the outage and ensure the stability of our data center operations. For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. |
TS Service Desk Walk-In Support: Tech Center in Collins Library Phone Support: (253) 879-8585 Online Help: support.pugetsound.edu |
Author Archives: fachavez
Network Outage: 12/12 Approximately 3:15PM – 4:15PM
Dear members of the Puget Sound community, On Thursday, December 12 at approximately 3:15 p.m. Technology Services (TS) was made aware of a network outage affecting both Collins Memorial Library and the Facilities Services Complex. While we are still working to identify the root cause of the issue, we believe the network instability may be related to the staging of software for our network infrastructure ahead of the holiday break. During this time, users would have experienced an inability to access any of the following services:
If you experienced any of the outages above, please contact the Service Desk at 253.879.8585 so we can identify those who were affected. |
TS Service Desk Walk-In Support: Tech Center in Collins Library Phone Support: (253) 879-8585 Online Help: https://www.pugetsound.edu/osd (Logon Required) |
Technology Services Early Closure Notice – Today Dec 12 @ Noon
Dear members of the Puget Sound community, On Today, December 12 at 12 p.m. Technology Services (TS) and Print and Copy Services (PCS), will be closing early for an on-campus, team holiday gathering. During this time, limited services will be available from the following areas:
In case of a technical emergency, please contact Security Services at 253.879.3311 and they will reach the appropriate resource within TS. |
TS Service Desk Walk-In Support: Tech Center in Collins Library Phone Support: (253) 879-8585 Online Help: https://www.pugetsound.edu/osd (Logon Required) |
AlertUs Desktop Notifications
Dear members of the Puget Sound community, The University of Puget Sound has partnered with Alertus notification systems to enhance the ability to communicate with the community in an emergency. As part of our ongoing effort to improve our emergency preparedness and provide more comprehensive safety and security measures Technology Services and Security Services have selected Alertus to enhance our emergency notification system. For more information on campus emergency management efforts, visit the emergency management page. We are in the process of combining our existing Everbridge Emergency Communications product with Alertus, creating a robust and redundant communication system. After extensive testing, Alertus will be automatically installed on all university owned Mac and Windows computers. Once installed, you will receive emergency notifications through Everbridge in the manner you have selected and receive an Alertus announcement on your university computer. To remove the message from your screen, simply acknowledge the alert. Alertus will run on both Mac or PC and, once installed, automatically connects to our emergency notification system. Please note the Alertus software will only issue emergency notifications on university issued devices if you are on campus and connected to the university wired or wireless network. To ensure proper installation, make sure all computer system upgrades are installed. Downloading and installing Alertus is quick, easy and takes just a few minutes. Technology Services will begin remotely installing the software on tomorrow (06/06/2024). Once installed, the software runs in the background and you can verify it is connected by checking the icon in the System Tray (on Windows) or Menu Bar (on a Mac). If you are on a Windows PC the installation will be entirely silent and will not require any user input. However, if you are on a Mac you will likely see a small notification appear in the notification center indicating a background application has been installed. There is no further action necessary to complete the installation. To learn more about Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website. Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. |
TS Service Desk Walk-In Support: Tech Center in Collins Library Phone Support: (253) 879-8585 Online Help: https://www.pugetsound.edu/osd (Login Required) |