[Update 3/19/09 4:20 PM] Services have been restored. The application services were stopped and restarted. Root cause analysis in underway.
The application server that hosts Cascade is currently unavailable. Other services impacted are Cascade, Portal, Discoverer, CRM, and Views Flash Survey. TS is aware of the problem and it is being investigated.
3/19/2009 11:30am – 12:30pm
Pavan and Jeff disabled the sync profiles and stopped the integration and directory servers on whidbey to install the 1024-bit certificates from rapidssl. We started the servers and enabled the sync profiles, then verified that a change to displayname in OID synced to AD and a password change in AD synced to OID.
We then reversed the process to re-install the previously functioning verisign certificates, and verified that the sync was again working successfully.
[Update 3/6/09 10:41 AM] This error has been resolved; however,
The right navigation on academic departments and programs pages linked from http://www.ups.edu/academicdepartments.xml is not displaying at this time. The missing navigation on these pages includes links to departmental and program Web sites. The problem has been referred to the vendor of our content management system, Ingeniux, and they promise a solution today.
We will add temporary links to the department and program Web sites until this issue is resolved.
I am shutting down the CMS server in order to perform needed maintenance on the system. This is required because some pages are missing their navigation. Folks who use the CMS will only be affected for a few minutes; however, publishing will be slow for the next 90 minutes.
The next scheduled CMS will be the afternoon of March 26.
Gehenna is experiencing a physical memory issue and will be off-line 3/5/09 between 6:45 AM and 7:30 AM for maintenance and troubleshooting. There is no expected impact to campus users as services will be re-routed to another server.
Due to problems on the forms server, we had to bounce it at 2:10 p.m. this afternoon. This kicks all current users off, so I sent an email to them alerting them of the situation. This affected all users in Banner Web Forms and Cascade Web Forms.
We are actively researching the problem and want to get it fixed soon.
On Saturday, February 28th from 6-8am, Alexandria will be offline for storage expansion and maintenance. During the outage, student home directories (H: drive) and department shares will be inaccessible.
This change is required to expand the amount of disk space available on Alexandria and to improve backup performance.
[Update 2/28/2009 @ 8:00am] Normal service on Alexandria was restored by 7:15am this morning.
A component of the events calendar malfunctioned at approxiamtely 3:45 PM causing the site to be unavailable. Service was restored at approximately 4:45 PM.
Cascade Web Forms and Banner Web Forms were down for a little while after 2 p.m. today. We are having unexplained errors on the forms server and we had to bounce it. We are changing the configuration to see if that helps this problem.
Technology Services was alerted around 8:15 am that Alexandria shares were unavailable. Services were restored as of 8:30am. Steps are being taken to 1) improve the stability of Alexandria and 2) better alert TS staff when critical services on Alexandra are unavailable.
The Windows 2000 server known as Whitney was retired today at 2:13pm. It is not known to be used anymore, but writting this entry just in case something is dependant on it and breaks after today. A special one-time commvault backup was done, written to tape 104L3 with 365 days retention.
Banner, Budget Administration, and Your Employment information will be unavailable from 5:00 p.m. on Friday, February 20, until 7:00 a.m. on Monday, February 23 due to the Banner migration to new hardware.
[Update 2/22/09 @ 12:00 noon] All services are back on-line. The upgrade went as planned and ahead of schedule.
At 11:30 am, one of the servers began experiencing performance problems. Banner, CRM, FAMIS, and Cascade batch processing are impacted by this.
Update: 2:05 pm, the performance problem on Camano has been resolved. CRM is still down.
Update: 2:45 pm, service has been restored on CRM
On February 17th from 5:30-6AM batteries in our EMC CX400 Disk Array will be replaced. This is a fully redundant system and no outage is expected. However, as with any change, there is always a risk of outage. If a problem arises with the change, the following services may be impacted:
email
CRM
Database Services (FAMIS, Banner, Millennium, etc.)
Please note that Cascade will not be affected.
Testing