Internet Connection Down April 27 – entry II – Service Restored

Electric Lightwave, the University’s Internet Service Provider states that the problem that began at around 5 PM was caused by a lightning strike in Tukwila. The lightning strike caused a section of fiber optic cable to fail, causing internet outages between Seattle and Portland, OR. Internet service was restored to the University at 11:08 PM. Some customers in the area did not have service restored until 3:00 AM April 28.

Internet Connection Down April 27 – entry I

Most of the campus is experiencing internet connection failure. This is due to a major outage in Seattle (information from Electric Lightwave, the University’s Internet Service Provider). Some areas of campus apparently have an internet connection, so the problem is due to improper routing information across our section of the internet.

No ETA as to the return of service as yet.

Possible WebMail problem identified

In our efforts to identify the cause of recent problems with the WebMail server, we have been at a loss for information. We have tried to discover what has been causing the delays and unresponsiveness in WebMail as of late. We have looked at possible memory leaks in daemons, possible attacks, possible miss configurations. All of these have not lead to a clear answer.

It is believed at this point in time that if Ockham’s Razor holds true we may have found the source of the problem. It was discovered late yesterday that the available disk space of the WebMail server was extremely low. Since WebMail serves as an imap gateway temporarily caching and displaying mail messages via a http server, disk space for temporary files is necessary. This has been the best possible explanation for the problems we have seen thus far.

We have increased available disk space on the server. We have also contacted server individuals who reported problems to determine if the issue still persists.

Whidbey reboot

Whidbey was experiencing problems with the secure shell daemon. During the troubleshooting process we were unable to get the daemon to restart correctly, and the system rebooted at 12:15 pm.

Upon further investigation of the last log, it became clear that the shutdown command issued on April 14 during preventative maintenance froze. In working with the sshd2 daemon, the shutdown completed.

The secure shell daemon is now running correctly.

Mail clients reconfigured

The imap and pop3 daemon used by various mail clients to read e-mail messages from the University’s mail server have been reconfigured. The have been assigned to seperate network interfaces.

one interface as been assigned on-campus imap traffic
one interface as been assigned on-campus pop3 traffic
one interface as been assigned off-campus imap & pop3 traffic

These daemons also have been assigned new ssl certificates by the University’s certificate authority.

Imap issue

The University’s mail server handling imap requests this afternoon between 3:30 pm and 4:00 pm. The testing was done with the secure imap daemon which confused the server. All imap connections were closed and reopened.

Software updates

The following servers will be down during the regularly scheduled preventative maintenance (pm) window, 7-9am for software updates:

mail
camano
whidbey
rainier
crystal
whistler
charon
bigben

These updates are recommended software patches to the operating system.

During the pm window the following services will not be available,

electronic mail
CASCADE web
CASCADE/SUMMIT/RIDGE databases
Corporate Time
Modem pool
VPN