Category Archives: Announcements

Follow up to Network Outage on January 25, 2024 – Root Cause Analysis

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,


Technology Services has prepared a Root Cause Analysis for the network incident that occurred on January 25th. The summary below provides additional technical details about the event as well as the course of action taken to restore services.


ISSUE AND IMPACT:

On January 25th at 2:30 p.m. the campus network began experiencing a level of instability that prevented users from accessing systems and services connected to the internet. During the duration of the network outage, services like Zoom Phone, Canvas, Google Workspace, and Okta were unreachable from within the campus network.


ROOT CAUSE:

After careful analysis of our event logs, we identified that the initial trigger of the event was caused by a change pushed by our Juniper Mist controller to a series of network devices. This change triggered a network routing convergence across a series of networks which were legacy artifacts of a security policy implementation that was never fully completed by Technology Services. Additionally, these changes triggered a series of events on our Palo Alto Firewalls ultimately causing the devices to no longer accept routing updates from the Juniper Core. At this point, both the Juniper Core and the Palo Alto firewall units were no longer able to successfully pass internet-bound traffic.


SOLUTION:

Technology Services reviewed all available logging information with our engineering team and determined that the legacy networks highlighted above as well as the associated firewall configuration should be removed from our network. Also, after careful review of the routing protocol used between the Juniper Core and the Palo Alto firewalls, it was decided that we should utilize a static routing protocol between the firewalls and the campus core network. The above stated solution has been scheduled for implementation on Thursday, February 8th at 5:00 a.m. A separate communication will be shared through the Service Announcements page detailing the events that will occur during the outage window.


For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Network Connectivity Issues

Posted on by

The Puget Sound Network was down starting around 2:50 p.m. and has been restored as of 7:11 p.m. Thank you for your patience as we worked through this issue. Our Infrastructure and Operations team identified a routing issue between our data centers and the internet causing severe network instability. The outage we experienced this afternoon ties back to a legacy network design that our teams are actively working to remediate.


Technology Services will be providing a Root Cause Analysis (RCA) for this incident in the next few days which will be available in our Service Announcements page.


For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.


Thank you again for your patience. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk.


{PUGETSOUND NETWORK}: Warner Network Switch Replacement – Friday 1/26/24 8 a.m. – noon

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: Warner Network Switch Replacement

Friday, January 26 | 8 a.m. – noon

REASON FOR MAINTENANCE: 

  • Technology Services will replace the distribution switch that provides network services to Warner gym.

 

WHAT YOU SHOULD KNOW:

  • Network and WiFi connectivity will be temporarily unavailable in Warner while the network switch is being replaced.

  • Actual downtime is expected to last no longer than 30 minutes.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND NETWORK}: Greek House Unit B (ΣΑΕ) Switch Reboot – Tuesday 1/23/24 noon – 12:30 p.m.

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: Greek House Unit B (ΣΑΕ) Switch Reboot

Tuesday, January 23 | noon – 12:30 p.m.

REASON FOR MAINTENANCE: 

  • Technology Services needs to reset the switch in the Greek house Unit B (ΣΑΕ) to resolve a memory condition.

 

WHAT YOU SHOULD KNOW:

  • Network and WiFi connectivity will be temporarily unavailable in Greek house Unit B (ΣΑΕ) during the maintenance window.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND NETWORK}: Planned Firewall Maintenance – Wednesday 1/17/24 7:00 a.m. – 7:30 a.m. – NEW DATE

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

UPDATE 1/12/2024

  • This work has been rescheduled to Wednesday, January 17, from 7 a.m. – 7:30 a.m. As a reminder, all campus systems are expected to remain available and uninterrupted during this time.


UPDATE 12/19/2023:

  • This work has been postponed due to the availability of our vendor partner. The work will be rescheduled for early next year and we will communicate the new date and time once it is determined. Thank you for your understanding.


Dear members of the Puget Sound community,

Technology Services (TS) will be performing scheduled maintenance on our network security devices. No outage or interruption to services are expected. Details on the maintenance being performed are available below:

Network Security Devices (Firewalls): Configuration Updates

Wednesday, January 17 | 7 a.m. – 7:30 a.m.

WHAT YOU SHOULD KNOW: 

  • All campus network connectivity and hosted systems should remain available.

  • As we continue to transition the university to a more modern network, TS will be changing configurations on our network security devices (firewalls) in preparation for future work.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Planned Maintenance – Harned Hall

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Planned Maintenance: Harned Network Switch Upgrade

Friday, January 5 | 5 p.m. –  6 p.m.

REASON FOR MAINTENANCE: 

  • Technology Services needs to update the Harned network switch software to a more recent version so we have a secure and reliable environment.

 

WHAT YOU SHOULD KNOW:

  • Network and WiFi connectivity will be unavailable during the maintenance window.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{NETWORK, WIFI, PHONES} Planned Campus Maintenance – Tue, Jan 2 – Fri, Jan 12

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound campus community,

Technology Services (TS) will perform critical maintenance to our campus network as we continue to transition the university to a more modern network. During this time, network connectivity on campus will be intermittently unavailable on a building by building basis as various components are moved to the new network infrastructure. No full campus outages are planned.

Network, WiFi, and Phones: On Campus Network Maintenance

Tuesday, January 2 – Friday, January 12

WHAT YOU SHOULD KNOW: 

  • Work will be done one building at a time. Some buildings and houses are connected through multiple locations and may experience multiple outages.

  • Outages are expected to be short in duration, up to a few minutes, as the equipment moves occur.

  • Outages longer than a few minutes are not expected.

  • No full campus outages are planned.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND NETWORK}: Planned Firewall Maintenance – Tuesday 12/19/23 5:00 p.m. – 5:30 p.m. – POSTPONED

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

UPDATE 12/19/2023:

  • This work has been postponed due to the availability of our vendor partner. The work will be rescheduled for early next year and we will communicate the new date and time once it is determined. Thank you for your understanding.


Dear members of the Puget Sound community,

Technology Services (TS) will be performing scheduled maintenance on our network security devices. No outage or interruption to services are expected. Details on the maintenance being performed are available below:

Network Security Devices (Firewalls): Configuration Updates

Tuesday, December 19 | 5 p.m. – 5:30 p.m. – POSTPONED

WHAT YOU SHOULD KNOW: 

  • All campus network connectivity and hosted systems should remain available.

  • As we continue to transition the university to a more modern network, TS will be changing configurations on our network security devices (firewalls) in preparation for future work.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{NETWORK, WIFI, PHONES} Planned Campus Maintenance – Monday 12/18/23 6:00 p.m. – 9:00 p.m. [COMPLETE]

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound campus community,

Technology Services (TS) will perform critical maintenance to our campus core network as we continue to transition the university to a more modern network. During this time, network connectivity on and off campus will be intermittently unavailable as various components are moved to the new core network infrastructure.

Network, WiFi, and Phones: On Campus Network Maintenance

Monday, December 18 | 6 p.m. –  9 p.m. [COMPLETE]

WHAT YOU SHOULD KNOW: 

  • All on campus network connectivity, including access to systems hosted on campus, such as myPugetSound and network shares, as well as Internet, will be intermittently unavailable.

  • Systems requiring network connectivity on the Puget Sound campus, such as physical phones and printers, will be intermittently unavailable.

  • GlobalProtect VPN will be intermittently unavailable. 

  • Internet access (WiFi and Ethernet) will be intermittently unavailable.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{NETWORK, WIFI, PHONES} Planned Campus Maintenance – Monday 12/18/23 6:00 p.m. – 9:00 p.m.

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound campus community,

Technology Services (TS) will perform critical maintenance to our campus core network as we continue to transition the university to a more modern network. During this time, network connectivity on and off campus will be intermittently unavailable as various components are moved to the new core network infrastructure.

Network, WiFi, and Phones: On Campus Network Maintenance

Monday, December 18 | 6 p.m. –  9 p.m.

WHAT YOU SHOULD KNOW: 

  • All on campus network connectivity, including access to systems hosted on campus, such as myPugetSound and network shares, as well as Internet, will be intermittently unavailable.

  • Systems requiring network connectivity on the Puget Sound campus, such as physical phones and printers, will be intermittently unavailable.

  • GlobalProtect VPN will be intermittently unavailable. 

  • Internet access (WiFi and Ethernet) will be intermittently unavailable.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND NETWORK}: Planned Firewall Maintenance – Tuesday 12/19/23 5:00 p.m. – 5:30 p.m.

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) will be performing scheduled maintenance on our network security devices. No outage or interruption to services are expected. Details on the maintenance being performed are available below:

Network Security Devices (Firewalls): Configuration Updates

Tuesday, December 19 | 5 p.m. – 5:30 p.m.

WHAT YOU SHOULD KNOW: 

  • All campus network connectivity and hosted systems should remain available.

  • As we continue to transition the university to a more modern network, TS will be changing configurations on our network security devices (firewalls) in preparation for future work.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

Technology Services Early Closure Notice – Thurs Dec 14 @ Noon

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound


Dear members of the Puget Sound community,


On Thursday, December 14 at 12 p.m. Technology Services (TS) and Print and Copy Services (PCS), will be closing early for an on-campus, team holiday gathering.


During this time, limited services will be available from the following areas:

  • Service Desk Support
  • Print and Copy Services (PCS)
  • Media Services

In case of a technical emergency, please contact Security Services at 253.879.3311 and they will reach the appropriate resource within TS. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND SYSTEMS}: Patch Maintenance, Sun Dec 10, 6 a.m.-6 p.m. [COMPLETE]

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Puget Sound Systems and Applications: Planned Maintenance

Sunday, December 10 @ 6 a.m. – 6 p.m. [COMPLETE]

SYSTEMS IMPACTED: 

  • Campus file servers

  • myPugetSound.edu

  • PeopleSoft Human Resources

  • PeopleSoft Financials

  • PeopleSoft Campus Solutions

  • vDesk

  • Various Puget Sound systems and applications

REASON FOR IMPACT:

  • To ensure a secure, reliable computing environment, TS will be completing routine patch maintenance on our Puget Sound production servers.

 

ADDITIONAL INFORMATION:

  • The university network, internet access, and wireless will NOT be affected.

  • Technology Services schedules regular maintenance windows, generally on the second Sunday of every other month, to ensure a secure, reliable computing environment.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{PUGETSOUND SYSTEMS}: Patch Maintenance, Sun Dec 10, 6 a.m.-6 p.m.

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound community,

Technology Services (TS) has planned the following maintenance activities. Details on the maintenance being performed are available below:

Puget Sound Systems and Applications: Planned Maintenance

Sunday, December 10 @ 6 a.m. – 6 p.m.

SYSTEMS IMPACTED: 

  • Campus file servers

  • myPugetSound.edu

  • PeopleSoft Human Resources

  • PeopleSoft Financials

  • PeopleSoft Campus Solutions

  • vDesk

  • Various Puget Sound systems and applications

REASON FOR IMPACT:

  • To ensure a secure, reliable computing environment, TS will be completing routine patch maintenance on our Puget Sound production servers.

 

ADDITIONAL INFORMATION:

  • The university network, internet access, and wireless will NOT be affected.

  • Technology Services schedules regular maintenance windows, generally on the second Sunday of every other month, to ensure a secure, reliable computing environment.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu

{NETWORK, WIFI, PHONES} Planned Campus Maintenance – Thu 11/16 8 p.m. – 9 p.m.

Posted on by

This logo identifies the Technology Services department within the University of Puget Sound

Dear members of the Puget Sound campus community,

Technology Services (TS) will perform critical maintenance to our campus core network as we continue to transition the university to a more modern network. During this time, network connectivity on and off campus will be intermittently unavailable as various components are moved to the new core network infrastructure.

Network, WiFi, and Phones: On Campus Network Maintenance

Thursday, November 16 | 8 p.m. –  9 p.m.

WHAT YOU SHOULD KNOW: 

  • All on campus network connectivity, including access to systems hosted on campus, such as myPugetSound and network shares, as well as Internet, will be intermittently unavailable.

  • Systems requiring network connectivity on the Puget Sound campus, such as physical phones and printers, will be intermittently unavailable.

  • GlobalProtect VPN will be unavailable. 

  • Internet access (WiFi and Ethernet) will be intermittently unavailable.

For more information on Technology Services' Incident Communication Plan or to view a list of our Critical Systems and definitions, please visit our Service Announcements page on the Puget Sound website.

Thank you for your patience as we conduct this important work. If you have any questions or concerns, please don't hesitate to contact the TS Service Desk. 

TS Service Desk

Walk-In Support: Tech Center in Collins Library

Phone Support: (253) 879-8585

Online Help:  https://www.pugetsound.edu/osd (Logon Required)

Email Support: servicedesk@pugetsound.edu