[Resolved] E-mail Delivery Delay Thursday 2/5/2009 5:30-7:30am – Part 2 of 2

On the morning of Thursday, February 5th from 5:30 to 7:30am, TS will be performing upgrades to improve the effectiveness of our spam filtering system. During the upgrade, delivery of email into and out of the college may be delayed.  E-mail sent may not arrive until after 7:30am.  E-mail sent within the University is not affected by this outage (i.e. e-mail from one @ups.edu account to another).

[Update: 2/5 at 9:30am] Mail delivery was restored by 7:10 am; however, the update for the end user quarantine page took longer than expected. All services were restored by 9:30am.

December 23, 2008: Spam Quarantine Unavailable 5-10am

On December 23rd from 5-10am the Sophos PureMessage spam may be unavailable for some messages/users.  Mail will continue to be delivered as per normal.  Should a user be unable to remove or deliver a message from their quarantine they should try again after 10am.

Due to a campus wide power outage from 5-7am on 12/23, this outage may be postponed depending on completion of that outage.

~~~~~~~~ Update 12/23 8:50am ~~~~~~~~~~~

This upgrade is progressing well, however it is taking a bit longer than expected.  We may still make it up by 10am, but 11am is more realistic.

Please note that mail is being delivered normally and without delay, it’s just half of the spam quarantine is unavailable until 11am.

~~~~~~~ Update at 11:20 ~~~~~~~~~

The quarantine is now back online.  From approximately 10:50-11:15am the quarantine was available off and on.  Please contact the helpdesk if you have problems accessing your quarantine.

Database Server High Load Problems 4.12-4.14

Our primary database servers providing support for

  • Cascade
  • Cascade Web
  • Millennium
  • Banner
  • Famis
  • CRM
  • Basis

experienced high load April 12 – 14 resulting in performance concerns and periodic service interruptions.  Regular service levels were restored by 5pm Monday, April 14th.

The high load conditions were caused by new data load and warehousing load activity.

¼/p>

Commvault Updates

Wed 2/20 – I updated Galaxy with 7.0 and SP2. Rebooted Galaxy
Thu 2/21 – I updated Alexandria, BE1, BE2, Crystal, Rainier with 7.0
– installed SP2 and updates on Alexandria, BE1, and BE2
Mon 2/25 – I started to install 7.0 on the iDataAgents and to push SP2 and post updates to the client computers. I rebooted Kronos and MSproject to finish their upgrades. I updated Crystal and Rainier with SP2 and post updates. The post updates broke the Online Full backups on Rainier. Crystal did not properly install SP2 so it did not get the broken updates post SP2.
Tue 2/26 – Worked with Commvault on the issue and stopped the Online Full backups for Rainier until the problem is resolved.
Wed 2/27 – Updated Galaxy with post SP2 updates. Uninstalled updates to Rainier and Crystal and reinstalled SP2 only from the disk. Tested the Online Fulls and reenabled the schedules for those backups. Continued to upgrade and update iDataAgents.

Thu 2/28 – Installed 7.0 Base on Shasta

Internet Outage

Our ISP, Integra, was having some issues last night between 8:10pm and 9:46pm. I called ELI to have them look into the problem. Mark and I came onsite to assist. The problem was resolved by Integra who reset the connection interface on their end. I have not seen any further issues occurring.

Internet performance issues

We are experiencing reduced round trip times to the internet. This is effecting browsing and downloading of files. I am working with Cisco to find the cause and resolve it as soon as possible. You may notice messages when trying to browse to some websites “unable display page”. We are experiencing unusually heavy data traffic, which has brought these issues to the surface. We will work through the problems and keep you posted.