WebMail service will be interrupted for approximately 15 minutes from 9:00am to 9:15 am on Wednesday, April 28, 2004. This interruption is needed to reconfigure WebMail and give it a dedicated network interface on the University’s email server for WebMail’s imap sessions.
Electric Lightwave, the University’s Internet Service Provider states that the problem that began at around 5 PM was caused by a lightning strike in Tukwila. The lightning strike caused a section of fiber optic cable to fail, causing internet outages between Seattle and Portland, OR. Internet service was restored to the University at 11:08 PM. Some customers in the area did not have service restored until 3:00 AM April 28.
Most of the campus is experiencing internet connection failure. This is due to a major outage in Seattle (information from Electric Lightwave, the University’s Internet Service Provider). Some areas of campus apparently have an internet connection, so the problem is due to improper routing information across our section of the internet.
No ETA as to the return of service as yet.
In our efforts to identify the cause of recent problems with the WebMail server, we have been at a loss for information. We have tried to discover what has been causing the delays and unresponsiveness in WebMail as of late. We have looked at possible memory leaks in daemons, possible attacks, possible miss configurations. All of these have not lead to a clear answer.
It is believed at this point in time that if Ockham’s Razor holds true we may have found the source of the problem. It was discovered late yesterday that the available disk space of the WebMail server was extremely low. Since WebMail serves as an imap gateway temporarily caching and displaying mail messages via a http server, disk space for temporary files is necessary. This has been the best possible explanation for the problems we have seen thus far.
We have increased available disk space on the server. We have also contacted server individuals who reported problems to determine if the issue still persists.
Whidbey was experiencing problems with the secure shell daemon. During the troubleshooting process we were unable to get the daemon to restart correctly, and the system rebooted at 12:15 pm.
Upon further investigation of the last log, it became clear that the shutdown command issued on April 14 during preventative maintenance froze. In working with the sshd2 daemon, the shutdown completed.
The secure shell daemon is now running correctly.
The imap and pop3 daemon used by various mail clients to read e-mail messages from the University’s mail server have been reconfigured. The have been assigned to seperate network interfaces.
one interface as been assigned on-campus imap traffic
one interface as been assigned on-campus pop3 traffic
one interface as been assigned off-campus imap & pop3 traffic
These daemons also have been assigned new ssl certificates by the University’s certificate authority.
The University’s mail server handling imap requests this afternoon between 3:30 pm and 4:00 pm. The testing was done with the secure imap daemon which confused the server. All imap connections were closed and reopened.
The University’s web server was upgraded to apache 2.0.48.
The application of operating system patches to the directory server ran late. The application of the patches completed at 9:45 am, thus delaying the use of calendar and mail services for 50 minutes. All services have now been restored.
The following servers will be down during the regularly scheduled preventative maintenance (pm) window, 7-9am for software updates:
These updates are recommended software patches to the operating system.
During the pm window the following services will not be available,
The University’s web server will be upgraded to the latest version of the Apache web server (2.0.49) as part of the Office of Information Services’ Preventative Maintenance window.
Intermittent short outages may occur from 6-8 AM.