Database Server Reboot 4.17.2008

Email sent to liaisons:

Thursday April 17, Information Services will be performing critical changes on our database systems to improve recent performance concerns.

Access to all database systems including Cascade, Cascade Web, Banner, Famis, Basis and Millennium will be unavailable from 5:30 AM until 6 AM Thursday.

If you have questions or concerns regarding this outage, please contact Jenifer Brauen by telephone at ext. 2725, 253.879.2725, or via email at jbrauen@ups.edu. Thank you for understanding our need to conduct critical systems work.

Thank you.

– Jenifer Brauen

Database Server High Load Problems 4.12-4.14

Our primary database servers providing support for

  • Cascade
  • Cascade Web
  • Millennium
  • Banner
  • Famis
  • CRM
  • Basis

experienced high load April 12 – 14 resulting in performance concerns and periodic service interruptions.  Regular service levels were restored by 5pm Monday, April 14th.

The high load conditions were caused by new data load and warehousing load activity.

¼/p>

April 1 – April 7 listserv Delivery Problems

Beginning April 1st, the Mailman  listservs started failing to deliver messages to their membership.  This happened because one of the real-time blackhole lists that Mailman was subscribed to went out of service permanently, causing Mailman’s mail server (sendmail) to reject all messages outgoing to the list memberships. So the messages were received by Mailman and archived properly, but they were not delivered. This was found and resolved on April 7th.

This coincided with another email problem  on April  2-4, which masked  the listserv issue, which is why it took so long to find and resolve.

Commvault Updates

Wed 2/20 – I updated Galaxy with 7.0 and SP2. Rebooted Galaxy
Thu 2/21 – I updated Alexandria, BE1, BE2, Crystal, Rainier with 7.0
– installed SP2 and updates on Alexandria, BE1, and BE2
Mon 2/25 – I started to install 7.0 on the iDataAgents and to push SP2 and post updates to the client computers. I rebooted Kronos and MSproject to finish their upgrades. I updated Crystal and Rainier with SP2 and post updates. The post updates broke the Online Full backups on Rainier. Crystal did not properly install SP2 so it did not get the broken updates post SP2.
Tue 2/26 – Worked with Commvault on the issue and stopped the Online Full backups for Rainier until the problem is resolved.
Wed 2/27 – Updated Galaxy with post SP2 updates. Uninstalled updates to Rainier and Crystal and reinstalled SP2 only from the disk. Tested the Online Fulls and reenabled the schedules for those backups. Continued to upgrade and update iDataAgents.

Thu 2/28 – Installed 7.0 Base on Shasta