Follow up to Network Outage on January 25, 2024 – Root Cause Analysis

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,


Technology Services has prepared a Root Cause Analysis for the network incident that occurred on January 25th. The summary below provides additional technical details about the event as well as the course of action taken to restore services.


ISSUE AND IMPACT:

On January 25th at 2:30 p.m. the campus network began experiencing a level of instability that prevented users from accessing systems and services connected to the internet. During the duration of the network outage, services like Zoom Phone, Canvas, Google Workspace, and Okta were unreachable from within the campus network.


ROOT CAUSE:

After careful analysis of our event logs, we identified that the initial trigger of the event was caused by a change pushed by our Juniper Mist controller to a series of network devices. This change triggered a network routing convergence across a series of networks which were legacy artifacts of a security policy implementation that was never fully completed by Technology Services. Additionally, these changes triggered a series of events on our Palo Alto Firewalls ultimately causing the devices to no longer accept routing updates from the Juniper Core. At this point, both the Juniper Core and the Palo Alto firewall units were no longer able to successfully pass internet-bound traffic.


SOLUTION:

Technology Services reviewed all available logging information with our engineering team and determined that the legacy networks highlighted above as well as the associated firewall configuration should be removed from our network. Also, after careful review of the routing protocol used between the Juniper Core and the Palo Alto firewalls, it was decided that we should utilize a static routing protocol between the firewalls and the campus core network. The above stated solution has been scheduled for implementation on Thursday, February 8th at 5:00 a.m. A separate communication will be shared through the Service Announcements page detailing the events that will occur during the outage window.


For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Network Connectivity Issues

The Puget Sound Network was down starting around 2:50 p.m. and has been restored as of 7:11 p.m. Thank you for your patience as we worked through this issue. Our Infrastructure and Operations team identified a routing issue between our data centers and the internet causing severe network instability. The outage we experienced this afternoon ties back to a legacy network design that our teams are actively working to remediate.


Technology Services will be providing a Root Cause Analysis (RCA) for this incident in the next few days which will be available in our Service Announcements page.


For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.


Thank you again for your patience. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.


Technology Services Early Closure Notice – Thurs Dec 14 @ Noon

This logo identifies the Technology Services department within the University of Puget Sound


Dear members of the Puget Sound community,


On Thursday, December 14 at 12 p.m. Technology Services (TS) and Print and Copy Services (PCS), will be closing early for an on-campus, team holiday gathering.


During this time, limited services will be available from the following areas:

  • Service Desk Support
  • Print and Copy Services (PCS)
  • Media Services

In case of a technical emergency, please contact Security Services at 253.879.3311 and they will reach the appropriate resource within TS. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PeopleSoft HR System} Planned Maintenance Complete

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,


Technology Services (TS) has completed the following maintenance activities. Details on the maintenance performed are available below:


Planned Maintenance for the PeopleSoft HR System is Complete. 

REASON FOR MAINTENANCE: 

  • To ensure a secure, reliable computing environment TS updated the PeopleSoft Human Resources Information System. It was unavailable during the maintenance window, which included self service items such as entering time in myPugetSound. 

WHAT YOU SHOULD KNOW:

  • If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conducted this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 


TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PeopleSoft HR System} Planned Maintenance – Fri Nov 10 | 5 p.m. – Sun, Nov 12 | 8 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: PeopleSoft HR System

Friday, November 10 | 5 p.m. –  Sunday, November 12 | 8 p.m.

REASON FOR MAINTENANCE: 

  • To ensure a secure, reliable computing environment TS will be updating the PeopleSoft Human Resources Information System. It will be unavailable during the maintenance window, which includes self service items such as entering time in myPugetSound. 

 

WHAT YOU SHOULD KNOW:

  • Time Entry and Time Approval will not be available during the maintenance window. 

  • myPugetSound will still be available during the maintenance window, but some of the links within myPugetSound may not work if they originate from the PeopleSoft Human Resources System.

  • If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PeopleSoft HR System} Planned Maintenance – Fri Nov 10 | 5 p.m. – Sun Nov 12 | 8 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: PeopleSoft HR System

Friday, November 10 | 5 p.m. –  Sunday, November 12 | 8 p.m.

REASON FOR MAINTENANCE: 

  • To ensure a secure, reliable computing environment TS will be updating the PeopleSoft Human Resources Information System. It will be unavailable during the maintenance window, which includes self service items such as entering time in myPugetSound. 

 

WHAT YOU SHOULD KNOW:

  • Time Entry and Time Approval will not be available during the maintenance window. 

  • myPugetSound will still be available during the maintenance window, but some of the links within myPugetSound may not work if they originate from the PeopleSoft Human Resources System.

  • If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

[RESOLVED] myPugetSound & Employment Page Issues in Google Chrome Browser

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,


Technology Services (TS) has resolved the following issue. Details on the issue and resolution are available below:


myPugetSound: Unable to Access Certain Pages in Google Chrome Browser 

Resolved on Wednesday, October 5 |  5:30 p.m.


Overview

Google released an update to the Chrome browser which impacted faculty, staff, and student ability to access the HR and Finance tiles within myPugetSound. This also impacted the ability to open our Employment Opportunities page from a Google Chrome browser. TS implemented a solution that has resolved these issues. 


If you are still experiencing issues, please follow the below steps: 

  • Clear browser cache 
  • Close all open Google Chrome browser windows 
  • Re-open Google Chrome browser and try again

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.


Thank you for your patience as we worked to resolve this issue. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Puget Sound Website Maintenance 9/30 [COMPLETE]

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,


This weekend Technology Services and Marketing and Communications transitioned our Puget Sound website to a new hosting provider. This maintenance took place earlier today and is complete.


Main Campus Website: Maintenance [Complete]

Saturday, September 30


For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.


For support or questions about this planned maintenance outage, please contact the TS Service Desk at servicedesk@pugetsound.edu or 253.879.8585.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Puget Sound Website Maintenance (No downtime) Sept 9/30 8 a.m. – 12 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

This weekend Technology Services and Marketing and Communications will be transitioning our Puget Sound website to a new hosting provider. There should not be any visible changes during this transition and there should not be any downtime.

Main Campus Website: Maintenance
Saturday, September 30 | 8 a.m – Noon

WHAT YOU SHOULD KNOW:

  • Access to private pages may be limited during the maintenance window
  • There should not be any visible changes during this transition and there should not be any downtime

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

For support or questions about this planned maintenance outage, please contact the TS Service Desk at servicedesk@pugetsound.edu or 253.879.8585.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{RESOLVED] Puget Sound Network: Unable to Connect to Campus Network

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has resolved the following incident. Details on the issue are available below:

Puget Sound Network: Unable to Connect to Campus Network
Friday, September 8 | 6:45 a.m. – 8:30 a.m.

Early this morning our Infrastructure and Operations team identified a network issue that was preventing our systems from connecting to the internet. The issue was identified as a Distributed Denial-of-Service attack (DDoS) to our infrastructure, specifically to our Domain Name System (DNS) servers. This attack consumed much of our available internet bandwidth preventing users from accessing the Puget Sound network.  As of 8:30 a.m. our team was able to block the malicious traffic and restore internet connectivity to the campus.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{myPugetSound} Issue Accessing HR/FN Self Service Pages in Google Chrome Browser

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) is working to resolve the following issue. Details on the issue are available below:

myPugetSound: Issue Accessing HR and Financials Self Service Pages in Google Chrome Browser


Overview

Google has released an update to the Chrome browser which has impacted the use of myPugetSound for some, but not all users. This impacts faculty, staff, and student ability to access the HR and Finance tiles within myPugetSound. This may also impact those in the HR and Finance departments as they access the Financials and Human Resources menu via the Unified Navigation icon in the upper-right-hand corner.

This behavior was introduced in Chrome version 115.0.5790.171 and persists in the recent version 116. TS is working to identify why some people with this update have an error and others do not.

Errors Seen in myPugetSound

Below is the errant behavior seen when clicking on an HR or Finance tile:

  • A pop-up window with this error: custom javascript error blocked a frame with origin 'https://campus.pugetsound.edu' from accessing a cross-origin frame.
  • Another behavior seen in Chrome on a Mac is that this error doesn't pop-up but some of the pages in HR Manager Self-service do not return any results.   

Interim Solution

  • Use an alternative browser such as Firefox or Microsoft Edge
  • If you want to continue in the Google Chrome Browser, please follow the below steps: 
    • Open Chrome 
    • Enter “chrome://flags/#origin-agent-cluster-default”
    • Change the value for “Origin-keyed Agent Clusters by default” to DISABLED. 
    • Then close and re-open Chrome.

Next Steps

  • Technology Services will provide an update via email as soon as we have identified a viable resolution for this issue.
  • If you need additional support related to this issue, please reachout to the Technology Services Desk.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.


Thank you for your patience as we work to resolve this issue as soon as possible. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 


TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{myPugetSound, PeopleSoft Campus} Planned Maintenance, Completed on Sat 7/22 | 3:30 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has completed the following maintenance activities. Details on the maintenance performed are available below:

Planned Maintenance: myPugetSound, PeopleSoft Campus Solutions
Friday, July 21 | 5 p.m. –  Sunday, July 23 | 8 p.m.
Completed on Saturday July 22 

| 3:30 p.m.

Reason for Maintenance
Technology Services updated myPugetSound and PeopleSoft Campus Solutions to a more recent version so we have a secure and reliable environment.

What You Should Know
If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{myPugetSound, PeopleSoft Campus} Planned Maintenance, Fri Jul 21 at 5 p.m. – Sun July 23 at 8 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: myPugetSound, PeopleSoft Campus Solutions
Friday, July 21 | 5 p.m. –  Sunday, July 23 | 8 p.m.


Reason for Maintenance
Technology Services needs to update myPugetSound and PeopleSoft Campus Solutions to a more recent version so we have a secure and reliable environment. These systems will be unavailable during the maintenance window, which includes self service items such as entering time in myPugetSound.

What You Should Know
If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{myPugetSound, PeopleSoft Campus} Planned Maintenance, Fri Jul 21 at 5 p.m. – Sun July 23 at 8 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: myPugetSound, PeopleSoft Campus Solutions
Friday, July 21 | 5 p.m. –  Sunday, July 23 | 8 p.m.


Reason for Maintenance
Technology Services needs to update myPugetSound and PeopleSoft Campus Solutions to a more recent version so we have a secure and reliable environment. These systems will be unavailable during the maintenance window, which includes self service items such as entering time in myPugetSound.

What You Should Know
If you have issues connecting to myPugetSound or PeopleSoft Campus Solutions after the planned maintenance please try clearing your browser cache.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Technology Services Early Close on Monday July 3 @ 2:30 p.m.

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

We wanted to inform you that Technology Services (TS), including Print and Copy Services (PCS), will be closing early in observance of the 4th of July on Monday, July 3 at 2:30 p.m.

The below services will be impacted during this time:

  • Service Desk Support
  • Print and Copy Services (PCS)
  • Media Services

In case of a technical emergency, please contact Security Services at 253.879.3311 and they will reach the appropriate resource within TS.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu