The main university web server (www.ups.edu) experienced errors caused by the Ingeniux (CMS) application this afternoon. Content was restored from backup, although this version of the content is a few days old. When the publishing operation finishes in about 30 minutes, all current content will be restored. If you are a content provider who published pages between 3:00 to 3:30 PM, we recommend publishing them again after 5:40 PM. Thank you.
Category Archives: Unscheduled Outages
BJP down temporarily
BJP was down temporarily on May 7th from about 7:30 pm – 8:30 pm. It took itself down, we could find no evidence of a problem, so we restarted it and it seemed to work fine.
CRM unavailable – 4/25/2008
CRM was temporarily unavailable for for about 40 minutes.
Internet Service Resumes
At approximately 10:30 AM, Internet service was resumed as Integra Telecom moved our routes to a temporary router. Internet service resumed until about 12:30 PM, when Integra moved our routes back to our normal router, causing a brief 5-10-minute interruption in service.
Internet ISP Service DOWN!
A major fiber feed for Integra has been broken. A pole damaged along route 167 has broken the fiber feed for the state of Washington. This feed cannot be repaired until later this afternoon when 167 can be shutdown to restring the fiber across the highway. There is no estimated time of repair.
Reboot Millennium Database
Brought Millennium application off-line, restarted the Oracle database instance and restarted the application. This was done by the recommendation of Sage in coordination with database connectivity changes on 4.11.08.
April 1 – April 7 listserv Delivery Problems
Beginning April 1st, the Mailman listservs started failing to deliver messages to their membership. This happened because one of the real-time blackhole lists that Mailman was subscribed to went out of service permanently, causing Mailman’s mail server (sendmail) to reject all messages outgoing to the list memberships. So the messages were received by Mailman and archived properly, but they were not delivered. This was found and resolved on April 7th.
This coincided with another email problem on April 2-4, which masked the listserv issue, which is why it took so long to find and resolve.
3/18/2008 Internet interruption
There was a 5-10 minute interruption today at approximately 2:15. Our internet service provider increased bandwidth to the University without coordinating the change with us. Configuration changes were made here at the University’s equipment, which restored connection to the internet. The University now has a full DS3(T3) connection at 45 Mbps.
Replacement of network switches in Wyatt.
We replaced the network switches in the south end of Wyatt Hall this week-end. The north end (classrooms) will be replaced this week (during spring break).
09/10 Camano Failure
Camano failed due to process and memory overload caused by a failure to properly process bounceback messages from CRM.
February 19th – CMS Publishing failed
Monday morning at 7:30 AM, a publish on the CMS system failed, corrupting the University’s web site. The system was restored after about one hour.
Dependency graphs were rebuilt, and the publish target was copied directly to the live web site to expedite recovery
February 1st – PureMessage Problems
Due to a problem with an update to the system that was done almost one year ago, the PureMessage quarantine has not been properly expiring, causing a buildup of SPAM messages in the quarantine and an inflation of the PM database. This caused the system to apparently “hold back” certain messages, generally ones that originated from listservs.
In order to correct this issue, we are currently running processes that will properly expire and reindex the quarantine and the metadata in the database. As this proceeds, the held back messages are delivered. Users will see the appearance of old emails in their inboxes. The number of affected messages seems small. Most people are not seeing any old messages appear, but many are. AS stated above, most of the affected messages appear to be from listservs and email subscription services.
Protected: Cascade Web Down 1/24
CRM Down 1/19
CRM experienced periodic outages the morning of January 19 through 1:30 pm. Information Services was notified of the problem at approximately 11:00 am and began troubleshooting. The cause was identified at 12:15 pm and a resolution implemented by approximately 1:15 pm. The system was fully functional by 2:00 pm.
The cause of the failure was due to uncompiled objects in the database, but the root issue that caused the objects to become invalid is unknown.
1/11 Internet Connection Down Briefly Last Night
At approximately midnight last night, the University’s Internet connection was unavailable for 30-40 minutes, because of scheduled maintenance on our ISP’s routers.