In our efforts to identify the cause of recent problems with the WebMail server, we have been at a loss for information. We have tried to discover what has been causing the delays and unresponsiveness in WebMail as of late. We have looked at possible memory leaks in daemons, possible attacks, possible miss configurations. All of these have not lead to a clear answer.
It is believed at this point in time that if Ockham’s Razor holds true we may have found the source of the problem. It was discovered late yesterday that the available disk space of the WebMail server was extremely low. Since WebMail serves as an imap gateway temporarily caching and displaying mail messages via a http server, disk space for temporary files is necessary. This has been the best possible explanation for the problems we have seen thus far.
We have increased available disk space on the server. We have also contacted server individuals who reported problems to determine if the issue still persists.
The imap and pop3 daemon used by various mail clients to read e-mail messages from the University’s mail server have been reconfigured. The have been assigned to seperate network interfaces.
one interface as been assigned on-campus imap traffic
one interface as been assigned on-campus pop3 traffic
one interface as been assigned off-campus imap & pop3 traffic
These daemons also have been assigned new ssl certificates by the University’s certificate authority.
The contentDM server was renamed from digitalassets to digitalcollections. This change was made so the name of the server better reflects the services on the server.
For background on this problem, see the January 12 weblog entry.
A new domain, mail.ups.edu, was defined with an MX record pointing mail.ups.edu to the antivirus gateway (AVG). This allows us to deliver messages destined for addresses of the form firstname.lastname@example.org. We will leave this in place until 1 July.
I checked the C-BORD Oyssey backup process this moring to inspect my changes. It was determined that the modifications made on 3/18 were not executed. Upon further inspection it was determined that the original file did not use the correct command-line version switches for the backup log. This was modified and we wait for the scheduled process to run again.
This morning there were questions posed by the operators about the status of the ntbackup process on the C-BORD Odyssey server around the location of the backup log file.
During the course of my investigation, I discovered that the use of the %Odyssey% variable does not work in all situations. So I modified the tapebackup.cmd file to use the absolute path contained in the %Odyssey% variable.
During the implementation of e-mail routing changes on 2/2/04 the MX record for the ups.edu domain on the internal DNS server was inadvertantly disabled.
This problem has been fixed and a backlog of messages, held by the anti-virus gateway are now being delivered.
The University e-mail service has been reconfigured to route as many messages as possible through the anti-virus gateway before they are sent off-campus or recieved by the University’s mail servers.
This additional stop for messages increases the delivery time, but in necessary to reduce the propogation of e-mail viruses and worms.
The HP laserjet 9000 (syslaser1) was tweaked to try and determine source of multiple print jobs. Items changed on printer:
Clearable Warnings (changed from Job to On)
Auto Continue (changed from On to Off)
The Micros server, which controls the cash registers in the CBORD One-card system, ran out of disk space on the C: drive, causing serveral services to crash. This in turn caused the cash registers to go offline.
The pagefile was split among the C: and D: drives in the folowing manner: 500 MB on C: and 1 GB on D:, and serveral temp files and an old log file were deleted.
The NT backup was moved from 3:30 AM to 2:30 AM, because it was not finishing before opening time for the cafe and diner.
The sendmail configuration files on the University’s mail server have been upgraded to version 8.12.10. The early problem with the configuration files and the vacation program have been resolved. The vacation program has also been upgrade to prevent rev-lag.
The sendmail configuration files were also modified to use the currect os_type.
Problems were encountered with the sendmail 8.12.10 configuration file and the auto-responder (vacation) message. The configuration file has been rolled back while further investigation in being done.
Majordomo which has failed to work since September 3, 2003 as the result of unknown reasons has been replaced. Mailman is now up and functional. Most lists that previously existed in majordomo have been recreated in mailman.
These lists can be viewed at, http://mail.ups.edu/mailman/listinfo.
For the same reason as for Shasta, Whitney’s WINS setting were reconfigured to point to porthos and silentbob.
Shasta’s WINS configuration was pointing to nonexistent WINS servers. This was causing some slowing in file listings over SMB. The new setting now point to porthos and silentbob.