A patch cluster upgrade was performed on Camano from 4:30 am to 6:00 am Thursday morning. This cluster patch is required for the upcoming SAN migration.
Patch cluster maintenance upgrade on Grace
Patch cluster upgrades were performed on Grace from 1:00 pm to 3:00 pm Wednesday afternoon. These patches were needed for the upcoming SAN migration.
Patch upgrades performed Crystal
Patch cluster upgrades were performed on Crystal from 4:00 pm to 5:45 pm Tuesday afternoon. These patches were needed for the upcoming SAN migration.
Spam Quarantine Outage – Friday, October 10th 6:30-8:30am
Starting at 6:30am on Friday, October 10th, the Sophos Spam Quarantine will be down for maintenance. During this time users will not be able to access their spam quarantines, but mail will continue to be delivered as normal. The Quarantine is expected to be available by 8:30am.
Update — 10/10/2008 @ 8:30am
Migration is taking longer than expected waiting on replication of small files. We currently estimate that the Spam Quarantine will be up approximately 9:30am.
Update — 10/10/2008 @ 9:45am
Migration was completed at approximately 9:30am.
CRM application tier bounced at 10:20 am on 10/1/2008
CRM fulfillment experienced an error during the processing of an email (rollback segment problem). Restarting the email caused a runaway cpu-intensive session even though the email was subsequently cancelled.
Cause of runaway session: user created content using the wrong content type (admission content type instead of campus email), which relied on admission data being in the system.
Resolution: We bounced the application server and killed the runaway session. User recreated email with correct content type.
Firewall upgrade is complete!
Firewall upgrade is complete and no issues encountered.
Firewall upgrade scheduled for Friday 26th between 6-7am.
We will be upgrading our firewall early Friday morning to incorporate a new security patch. The actual down time should be mininal, because a reboot is required to implement the new software.
PureMessage Edge Server Maintance 9/26
On Friday 9/26 from 5am – 9:30am a redundant PureMessage server will be down for upgrades. As this server is redundant end users should not notice any downtime. Users may be unable to recover certain messages from the quarantine during this time.
Details:
Due to high iowait times support suggested we switch disks in the hades.ups.edu server to RAID10 (currently RAID5). To achive this we will rebuild the server via our automated installation method and then restore the PureMessage applications and data. Expected downtime for hades is 4.5 hours (1.5hr backup, 1hr disk rebuid, 1 hr OS load, 1hr restore)
Problems with www2.ups.edu
Updated 9/25 11:51 AM
We currently are experiencing problems with www2.ups.edu. FTP service, which was not working earlier, has been restored.
We plan to reboot this system tomorrow morning (9/26/08) before 9:00 AM and hope that this will resolve the issues we’re having. More will posted here as we continue investigate the problems.
Hardware replacement and Patch Maintenance to Unix servers
On Sunday the 21st of September 2008, there was scheduled maintenance on Camano and Grace, for Solaris9 cluster patches as well as J2SE “java” related patches. There was also a hardware replacement of a power supply for the server known as Camano. Both servers were up and running without incident by 10:30 a.m. Sunday.
Problems with library.ups.edu response times
Library.ups.edu has been experiencing slow response times and dropped connections for several days. We have made some changes to the way the server responds and now are waiting to see if this resolves the problem.
Internet outage
Our ISP, had a router issue which caused our internet access to be down during the night. The service was restored at 6:57am this morning.
Database Systems Restored – 09.09.08
Service has been restored to all central database systems (Cascade, Cascade Web, Banner, Famis, Millennium, Basis and CRM) as of 3pm. Please contact the Help Desk, 253.879.8585 or helpdesk@ups.edu, if you continue to experience problems.
All Database Systems Unavailable – 09.09.08
All central database systems are unavailable. This service disruption began at 11AM today and services impacted include: Cascade, Cascade Web, Banner, Famis, Basis, CRM and Millennium. Technology Services has determined the source of the disruption to be a power outage and is working with Facilities Services to resolve as quickly as possible.
UPDATE AS OF 2PM:
Stable power has been restored and Technology Services is now focused on restarting and verifying the database systems. Every effort is being made to restore services as quickly as possible. Our earliest continuation of service is estimated at 3pm today.
Cascade Unscheduled Downtime
At 6 AM this morning, a reload of the core switch caused a failure of the network interfaces on the Cascade Web server. This in turn caused the Cascade web interface to be unavailable from 6 to 8:15 this morning. The server was rebooted and recovered, restoring service at approximately 8:15.
Other affected servers were grace, camano, and crystal, all of which were rebooted to restore service at 7:30 AM. Batch job processing was affected between 6 and 7:30 AM.