On the morning of November 6th at 5:30am messages sent from inside the University to external receipients will be delayed until 7am. The outage does not affect messages from University users to other University users.
—
Update: This outage is compete as of 6:10am on 11/6. All outgoing mail should now be back up to speed. Please contact the helpdesk if you see any irregular behavior.
The Installation of the new storage array and SAN to support upgrades of the database and email systems was completed with the reboots of all database servers from 5:30 to 6:30 AM this morning. CRM was restored at 7:45 AM. The extended timeframe was necessary to ensure that all existing and new LUNs had redundant paths to their hosts in the SAN.
Some users were not getting IP addresses when connecting to Aruba Wireless. This was do to a shortage of IP addresses in DHCP. I expanded the scope on Saturday to make more IP addresses available. This problem should not reoccurr.
At 7:30 this morning, the ezproxy reverse proxy server was moved to the new web server hardware on www5. Downtime for the service was approximately 15 minutes.
Saturday, October 25, Technology Services will be performing important updates on our database systems and storage area network. This will impact our database and email systems. In order to minimize service disruption, there will be two groups of outages:
Network files and shares, and other network services will not be impacted.
The server where sharepoint resides (portal.ups.edu) needs to be rebooted to ensure correct operation of replaced equipment. This is expected to take less than 30 minutes.
Update @ 12:25pm:
SharePoint is now back online, was offline from approximately noon till 12:20pm.
The Content Management System stopped publishing content to the main web server at about 9:10am this morning. I believe that this problem has been resolved; however, new content will be unlikely to appear on the main web server until after 3:30pm.
If your changes haven’t shown up on the main web server by 4:00 pm, please publish the pages again. Thank you.
From 6:30-7:30 am on October 16th upgrades to the Sophos Pure Message anti-spam and anti-virus servers will cause delays in outgoing messages (from @ups.edu accounts to Internet users). Service is expected to be fully restored by 7:30am.
Update:
Outgoing mail was down from about 6:45-7:20am, mail is now flowing regularly. Please contact the helpdesk if you see additional delays.
Patch cluster installs and maintenance upgrades were performed on Rainier and Whidbey between 8:00 am and 11:00 pm Sunday morning. These patch clusters are required for the upcoming SAN migration.
All Sophos pure message anti-spam and anti-virus email servers will be down for maintenance from 5:30-7:30am to upgrade the software to the latest version. This upgrade is required to improve spam blocking for UPS email users.
During the upgrade mail should continue to flow between the University and the Internet, but may be slightly delayed at times until the upgrade is complete at 7:30am. The spam quarantine will be unavailable during parts of this outage as well. If the quarantine is not available please check back later in the day.
Update — 10/14/2008 @ 6:54am
All upgrades are complete and working as expected. Due to the upgrade process some email may have been delayed by a half hour or more, but should now be back to normal.
A patch cluster upgrade was performed on Camano from 4:30 am to 6:00 am Thursday morning. This cluster patch is required for the upcoming SAN migration.
Patch cluster upgrades were performed on Grace from 1:00 pm to 3:00 pm Wednesday afternoon. These patches were needed for the upcoming SAN migration.
Patch cluster upgrades were performed on Crystal from 4:00 pm to 5:45 pm Tuesday afternoon. These patches were needed for the upcoming SAN migration.
Starting at 6:30am on Friday, October 10th, the Sophos Spam Quarantine will be down for maintenance. During this time users will not be able to access their spam quarantines, but mail will continue to be delivered as normal. The Quarantine is expected to be available by 8:30am.
Update — 10/10/2008 @ 8:30am
Migration is taking longer than expected waiting on replication of small files. We currently estimate that the Spam Quarantine will be up approximately 9:30am.
Update — 10/10/2008 @ 9:45am
Migration was completed at approximately 9:30am.
CRM fulfillment experienced an error during the processing of an email (rollback segment problem). Restarting the email caused a runaway cpu-intensive session even though the email was subsequently cancelled.
Cause of runaway session: user created content using the wrong content type (admission content type instead of campus email), which relied on admission data being in the system.
Resolution: We bounced the application server and killed the runaway session. User recreated email with correct content type.