See the post in the “Maintenance Windows” catagory for the public communication. This post is to expand on what servers were patched, and basic changes. All of the following patches were applied by noon.
The following windows servers were patched (Windows updates):
DM-1 to DM-8
WEBSERVER1 (SP2 applied)
WEBSERVER2 (SP2 applied)
Linux (all available OS and Dell RAID FW/Drives as applicable):
Additional changes on Sophos PMX servers:
Turned Perc write cache on (write back), and changed linux readahead:
sudo blockdev –setra 8388608 /dev/sda
adaptive read ahead
[Update 5/31/2009 1:00 PM] All service has been restored, and this maintenance window is complete.
[Update 5/31/2009 12:15 PM] Many services have been restored, with the exception of Database Applications: Cascade, FAMIS, Millennium, and Banner. They are expected to be available by 4:00 PM as planned.
The next production maintenance window is Sunday, May 31, from 8:00 AM to 4:00 PM.
From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:
- Personal and departmental file shares (on Alexandria and Merlin2)
- Database Applications: Cascade, FAMIS, Millennium, and Banner
Please note that Database Applications will continue to be unavailable until 4:00 PM.
[Update 5/29/2009 7:34 AM] Power, network and wireless service have all been restored.
The power was unexpectedly cut to the fieldhouse this morning at 6:00 AM. The campus network and wireless server are unavailable off-line until power is restored.
Upgraded “Norton Symantec Virus” on server Viawarp to “McAfee VirusScan Enterprise.”
We set the debug level back to 63 to troubleshoot the password sync errors due to AD password policy problem. Here are the commands, executed as oracle on whidbey:
oidctl connect=AS1012P server=odisrv instance=1 configset=1 flags=”port=3636 sslauth=2″ stop
oidctl connect=AS1012P server=odisrv instance=1 configset=1 flags=”port=3636 sslauth=2 debug=63″ start
The Active Directory password policy was inadvertently set to reject passwords that did not contain any special (non-alphanumeric) character, such as *#$% etc.
The problem began about 3/21/2009 and was corrected at 3:15pm on 3/26/2009. During this period, anyone changing a password using Windows was instructed to include a special character.
Passwords changed using Cascade Web during this period were not synchronized to Active Directory, so the new password did not work for Webmail, Windows, etc. This can now be corrected by changing either the AD or OID password.
The problem was corrected by deselecting the special character requirement in the AD password policy.
Here is an example of the error in the ActiveExportUsers_Groups.trc log:
Error in executing mapping DIP_LDAPWRITER_ERROR_MODIFY
javax.naming.OperationNotSupportedException: [LDAP: error code 53 - 0000052D: SvcErr: DSID-031A0FC0, problem 5003 (WILL_NOT_PERFORM), data 0
The following non production systems were patched today:
pilchuck (OS, RAID firmware, powerpath)
All patches progressed as expected and no lingering issues are expected. Please see the following share for more information:
Webmail was unavailable for about 15 minutes at 8:30 PM tonight. Technology Services staff was notified and has resolved the problem.
The new Impulse Point NAC server had been implemented for the residential network. It will check machines for virus software, windows updates and shared music and movie files. It will ask to have a profile key installed on the machine for identification, but will be transparent unless a machine is non-compliant.
The login page for Banner Forms was changed early on 5/15/2009 (see Cascade middle tier http server restarted), so Banner Forms users are prompted for Puget Sound username and password instead of Banner Database username, password, and database name.
Initially, this did not work correctly for Macintosh users, who were required to enter their username and password twice.
This problem was resolved Saturday morning, so Macintosh users should now be able to log in to Banner Forms (and create purchase orders), entering their Puget Sound credentials only one time per session.
To fix the problem, Ed changed parameter baseHTML to point to our modified basejpisso_webutil.htm file.
Technology Services are researching the cause and will update as additional information has become available. This affects CASCADE and BANNER.
Form Server is working again.
As part of implementing Banner Password Synchronization, the Cascade middle tier http server was restarted so that the following change in the httpd.conf file would take effect:
#Redirect /banner https://psforms.ups.edu/forms/frmservlet?config=banner
Redirect /banner https://psforms.ups.edu/banner_sso/gokssso.p_login
This directs cascade.ups.edu/banner to the new Banner login page that authenticates against OID.
See also Banner Password Synchronization works for Macintosh users
Our service provider performed some maintenance to our off-campus telephone connection. We lost connectivity Thursday evening, May 13th from approximately 8:00 to 8:30 pm.
The recently released Mac OS X 10.5.7 and Safari 3.2.3 fix the problem where users received “Too many redirects” error when logging in to Cascade Web.
The Technical Help page has been updated to show which versions of Mac OS X and Safari are compatible with Cascade. Browser/OS combinations that are not compatible still receive a message that includes a link to this Technical Help page.
Password synchronization and network/domain password resetting are currently disabled. Technology Services is currently working on restoring this service and we expect it to be restored shortly. We will update you as soon as possible.
UPDATE: Password Synchronization has been re-enabled as of 4:10pm on 5/12.