Mail delivery to and from the Internet during the evening of Sunday, 4/13 suffered several interruptions due to maintenance on email services. Delivery should be back to normal now, but we are still tracking the situation.
A number of university systems were adversely impacted by a time synchronization issue. Correct time synchronization plays a key role in authentication across various systems and an incorrect date from a US Navy time source disrupted authentication. We have changed our servers to synchronize from another time source which has resolved the problem.
If you are still experiencing logon issues, please contact the Technology Service Desk at 253.879.8585 or firstname.lastname@example.org.
The network outage involving WSIPC (and serveral other nearby K-20 sites) appears to have been fixed as of 16:18 this evening. The circuit provider ran a new section of fiber cable around the broken portion to bring them back up. We do not yet know whether that new fiber path is temporary or permanent. (If it’s temporary, there may be a planned outage in the future when they move the circuits back to the original fiber.)
WSIPC’s DNS servers are answering DNS queries again. Their other services should be back up too.
[Update 3/19/09 4:20 PM] Services have been restored. The application services were stopped and restarted. Root cause analysis in underway.
The application server that hosts Cascade is currently unavailable. Other services impacted are Cascade, Portal, Discoverer, CRM, and Views Flash Survey. TS is aware of the problem and it is being investigated.
The Aruba wireless controller, which currently provides wireless service to access points in Harned Hall, was upgraded to ArubaOS 3.1 from version 2.5.3. The upgrade was performed at 10:00, and took approximately 10 minutes to complete.
There were reports last night and this morning that user were having problems connecting to webmail. The problem appears to be linked to the Sophos PureMessage software and the timing of the quarantine digest messages and the slow disk array.
We updated our PureMessage configuration to send on digest messages instead of having each server send individual messages. The result being that all the messages hit the mail server and the same time and overwhelmed it with e-mail and client access with the start of the business day.
We are modifying the interval for digest messages from twice a day to once per day and moving the time to 1:30 am each morining.