Monday morning at 7:30 AM, a publish on the CMS system failed, corrupting the University’s web site. The system was restored after about one hour.
Dependency graphs were rebuilt, and the publish target was copied directly to the live web site to expedite recovery
It appears that the Ingeniux application build that we installed in early December (the version with support for IE 7) is much less tolerant of multiple attributes with the same name. The system error is very specific:
ERRORCODE: 0xc00cee3c
REASON: No attribute name may appear more than once in the same start tag or empty element tag.
In this instance, as in another case in late December, the offending attribute name is “NewWindow.” Since this attribute is used in a global export, I have, on the advice of Jason at Ingeniux, commented out lines 1178 and 1434 of CustomUtilities.inc, which should resolve this problem.
If this error recurs we’ll have to look for a different export and fix that one as well.
Due to a problem with an update to the system that was done almost one year ago, the PureMessage quarantine has not been properly expiring, causing a buildup of SPAM messages in the quarantine and an inflation of the PM database. This caused the system to apparently “hold back” certain messages, generally ones that originated from listservs.
In order to correct this issue, we are currently running processes that will properly expire and reindex the quarantine and the metadata in the database. As this proceeds, the held back messages are delivered. Users will see the appearance of old emails in their inboxes. The number of affected messages seems small. Most people are not seeing any old messages appear, but many are. AS stated above, most of the affected messages appear to be from listservs and email subscription services.
The current web statistics for both www.ups.edu and www2.ups.edu are available for viewing at http://www.ups.edu/x13284.xml. New to this set of statistics: browser stats.
The university’s Internet connection will be unavailable for up to 30 minutes starting at 4:00 AM on Wednesday, January 24th. Our Internet provider, Electric Lightwave/Integra, must perform important but routine equipment upgrades during this time.
At approximately midnight last night, the University’s Internet connection was unavailable for 30-40 minutes, because of scheduled maintenance on our ISP’s routers.
Due to hardware problems, the spam filter end user interface (the web system where University members check their spam quarantine contents) will be unreliable. Service has been engaged, and we hope to have a resolution today.
All computer and network services will be unavailable from 7:00 AM until 4:00 PM on Saturday, June 10, 2006.
OIS and Facilities Services staff and outside representatives will conduct a test of the emergency generator that powers the OIS Server and Network Operations Center. All systems will need to be powered down for much of the testing period.
In preparation for emergency generator testing on Saturday, an off-site DNS was added as a tertiary server.
The database systems went offline sometime of 3 June this weekend when rainier and crystal lost connections to several disk volumes. The disks were remounted, and the databases were restarted. The DBA was called at 9 AM this morning.
The www2 webserver also became unresponsive because it lost connection to the database. The webserver was restarted once the database came online, restoring service.
Sometime during the weekend, the ftp server on www2 decided that its license had expired. Until a new license could be obtained, the standard ftp server (wu-ftpd) was run. This led to some slow response (wu-ftpd operates under the xinetd master daemon) for much of the day. The vendor provided a new license key at 4:30 PM. This was installed, and ftp service was restored to normal.
The secondary web server www2 became unresponsive when the SurgeFTP service began consuming 99% of the CPU cycles. We were unable to stop the service or gracefully restart the system. The system was powered down, then back on, after which the system ran normally.
Merlin2 became unresponsive to fileshare access today at 5:30 PM. The console was still responding, and we were able to log on. Access to the disk arrays appeared to be impaired – we were unable to list the disks or view their contents. No pertinent events were logged in the Event Log. The system was rebooted, and was back to normal.
We have made an adjustment to the antivirus software (changed vendors), and will keep monitoring.
The movement of files from the old Merlin2 server was completed successfully. The new server is ready and available.
Data from the Administrative file server Merlin2 will be moved to a new server in an effort to solve the recent instability problems. Merlin2 will be unavailable from 5:30 PM until midnight Friday, May 19th, 2006.