CRM experienced periodic outages the morning of January 19 through 1:30 pm. Information Services was notified of the problem at approximately 11:00 am and began troubleshooting. The cause was identified at 12:15 pm and a resolution implemented by approximately 1:15 pm. The system was fully functional by 2:00 pm.
The cause of the failure was due to uncompiled objects in the database, but the root issue that caused the objects to become invalid is unknown.
Banner payroll was upgraded to 7.2.2 on 1/16/2007
At approximately midnight last night, the University’s Internet connection was unavailable for 30-40 minutes, because of scheduled maintenance on our ISP’s routers.
Due to hardware problems, the spam filter end user interface (the web system where University members check their spam quarantine contents) will be unreliable. Service has been engaged, and we hope to have a resolution today.
The following regulatory release and tax updates have been installed to banner production:
General 7.2
Payroll 7.2.2
Tax Update #100
The Banner system, Banner Tools and the Budget Administration Web application will be unavailable Friday, January 5, at 6:00 p.m., through Monday, January 8, at 7:00 a.m., to allow time for a Banner upgrade.
Camano was upgraded to add new SAN disks
All computer and network services will be unavailable from 7:00 AM until 4:00 PM on Saturday, June 10, 2006.
OIS and Facilities Services staff and outside representatives will conduct a test of the emergency generator that powers the OIS Server and Network Operations Center. All systems will need to be powered down for much of the testing period.
In preparation for emergency generator testing on Saturday, an off-site DNS was added as a tertiary server.
The database systems went offline sometime of 3 June this weekend when rainier and crystal lost connections to several disk volumes. The disks were remounted, and the databases were restarted. The DBA was called at 9 AM this morning.
The www2 webserver also became unresponsive because it lost connection to the database. The webserver was restarted once the database came online, restoring service.
Sometime during the weekend, the ftp server on www2 decided that its license had expired. Until a new license could be obtained, the standard ftp server (wu-ftpd) was run. This led to some slow response (wu-ftpd operates under the xinetd master daemon) for much of the day. The vendor provided a new license key at 4:30 PM. This was installed, and ftp service was restored to normal.
The secondary web server www2 became unresponsive when the SurgeFTP service began consuming 99% of the CPU cycles. We were unable to stop the service or gracefully restart the system. The system was powered down, then back on, after which the system ran normally.
Merlin2 became unresponsive to fileshare access today at 5:30 PM. The console was still responding, and we were able to log on. Access to the disk arrays appeared to be impaired – we were unable to list the disks or view their contents. No pertinent events were logged in the Event Log. The system was rebooted, and was back to normal.
We have made an adjustment to the antivirus software (changed vendors), and will keep monitoring.
The movement of files from the old Merlin2 server was completed successfully. The new server is ready and available.
Data from the Administrative file server Merlin2 will be moved to a new server in an effort to solve the recent instability problems. Merlin2 will be unavailable from 5:30 PM until midnight Friday, May 19th, 2006.