The Content Management System stopped publishing content to the main web server at about 9:10am this morning. I believe that this problem has been resolved; however, new content will be unlikely to appear on the main web server until after 3:30pm.
If your changes haven’t shown up on the main web server by 4:00 pm, please publish the pages again. Thank you.
From 6:30-7:30 am on October 16th upgrades to the Sophos Pure Message anti-spam and anti-virus servers will cause delays in outgoing messages (from @ups.edu accounts to Internet users). Service is expected to be fully restored by 7:30am.
Update:
Outgoing mail was down from about 6:45-7:20am, mail is now flowing regularly. Please contact the helpdesk if you see additional delays.
Patch cluster installs and maintenance upgrades were performed on Rainier and Whidbey between 8:00 am and 11:00 pm Sunday morning. These patch clusters are required for the upcoming SAN migration.
All Sophos pure message anti-spam and anti-virus email servers will be down for maintenance from 5:30-7:30am to upgrade the software to the latest version. This upgrade is required to improve spam blocking for UPS email users.
During the upgrade mail should continue to flow between the University and the Internet, but may be slightly delayed at times until the upgrade is complete at 7:30am. The spam quarantine will be unavailable during parts of this outage as well. If the quarantine is not available please check back later in the day.
Update — 10/14/2008 @ 6:54am
All upgrades are complete and working as expected. Due to the upgrade process some email may have been delayed by a half hour or more, but should now be back to normal.
A patch cluster upgrade was performed on Camano from 4:30 am to 6:00 am Thursday morning. This cluster patch is required for the upcoming SAN migration.
Patch cluster upgrades were performed on Grace from 1:00 pm to 3:00 pm Wednesday afternoon. These patches were needed for the upcoming SAN migration.
Patch cluster upgrades were performed on Crystal from 4:00 pm to 5:45 pm Tuesday afternoon. These patches were needed for the upcoming SAN migration.
Starting at 6:30am on Friday, October 10th, the Sophos Spam Quarantine will be down for maintenance. During this time users will not be able to access their spam quarantines, but mail will continue to be delivered as normal. The Quarantine is expected to be available by 8:30am.
Update — 10/10/2008 @ 8:30am
Migration is taking longer than expected waiting on replication of small files. We currently estimate that the Spam Quarantine will be up approximately 9:30am.
Update — 10/10/2008 @ 9:45am
Migration was completed at approximately 9:30am.
CRM fulfillment experienced an error during the processing of an email (rollback segment problem). Restarting the email caused a runaway cpu-intensive session even though the email was subsequently cancelled.
Cause of runaway session: user created content using the wrong content type (admission content type instead of campus email), which relied on admission data being in the system.
Resolution: We bounced the application server and killed the runaway session. User recreated email with correct content type.
Firewall upgrade is complete and no issues encountered.
We will be upgrading our firewall early Friday morning to incorporate a new security patch. The actual down time should be mininal, because a reboot is required to implement the new software.
On Friday 9/26 from 5am – 9:30am a redundant PureMessage server will be down for upgrades. As this server is redundant end users should not notice any downtime. Users may be unable to recover certain messages from the quarantine during this time.
Details:
Due to high iowait times support suggested we switch disks in the hades.ups.edu server to RAID10 (currently RAID5). To achive this we will rebuild the server via our automated installation method and then restore the PureMessage applications and data. Expected downtime for hades is 4.5 hours (1.5hr backup, 1hr disk rebuid, 1 hr OS load, 1hr restore)
Updated 9/25 11:51 AM
We currently are experiencing problems with www2.ups.edu. FTP service, which was not working earlier, has been restored.
We plan to reboot this system tomorrow morning (9/26/08) before 9:00 AM and hope that this will resolve the issues we’re having. More will posted here as we continue investigate the problems.
On Sunday the 21st of September 2008, there was scheduled maintenance on Camano and Grace, for Solaris9 cluster patches as well as J2SE “java” related patches. There was also a hardware replacement of a power supply for the server known as Camano. Both servers were up and running without incident by 10:30 a.m. Sunday.
Library.ups.edu has been experiencing slow response times and dropped connections for several days. We have made some changes to the way the server responds and now are waiting to see if this resolves the problem.