Update 8/20/09 @ 9:30am — The change encountered an issue and services were down until 9am.
On the morning of Thursday, August 20th from 7 – 8:30am users will not be able to access their quarantined spam messages on sophos.ups.edu. During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.
This outage is necessary to make performance improvements to this very busy system.
Updated 8/7/09 at 10:45am:
Sophos systems are back up and operational. Unfortunately the maintenance planned had to be backed out, so we will need to re-apply the changes at a future date/time.
Updated 8/7/09 at 10:00am:
We are still experiencing problems with the update. Please stand by for more information.
Updated 8/7/09 at 8:15am:
We experienced some difficulty during maintenance and we expect the window to extend until 9:30am at this time. All services should be up by 10am; sorry for the inconvenience.
On the morning of Friday, August 7th from 6:30 – 9:30am users will not be able to access their quarantined spam messages on sophos.ups.edu. During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.
This outage is necessary to make performance improvements to this very busy system.
See the post in the “Maintenance Windows” catagory for the public communication.
This month the Sophos PMX servers were patched with the latest Linux patches, and upgraded to the latest PureMessage version.
We also did the following non-recuring tasks:
Finally, all production Windows servers were patched:
This month’s outage was completed 1:30PM. All services other than Cascade Web and Banner were available by noon as expected.
The next production maintenance window is Sunday, June 28th, from 8:00 AM to 4:00 PM.
From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:
Please note that Cascade Web and Banner will continue to be unavailable until 4:00 PM.
See the post in the “Maintenance Windows” catagory for the public communication. This post is to expand on what servers were patched, and basic changes. All of the following patches were applied by noon.
The following windows servers were patched (Windows updates):
ALEXANDRIA
BE1
BE2
BE3
CLTACC1
CLTACC2
CMS
DHCP-1
DM-1 to DM-8
EPO
FE1
FE2
ILLIAD
INTCHK
KEEPER
MAURY2
MBS1
MBS2
MBS3
MEDIA
MERLIN2
PROJECTS
SANJUAN
VASHON
VERONICA
VIAWARP
VMMON
WEBSERVER1 (SP2 applied)
WEBSERVER2 (SP2 applied)
VMWare:
vmhost1
Linux (all available OS and Dell RAID FW/Drives as applicable):
orcas
tahoma
purgatory
styx
hades
gehenna
Additional changes on Sophos PMX servers:
Purgatory:
Turned Perc write cache on (write back), and changed linux readahead:
Spam messages quarantined prior to 9:00 AM on April 27, 2009, are no longer available on the PureMessage Web site due to the domain name change which took effect at that time. Messages arriving after 9:00 AM will be available as usual.
If you need messages quarantined before this time, please contact the HelpDesk at helpdesk@ups.edu or x8585.
Today we upgraded sophos puremessage on all servers, and in doing so we had several issues with the end user web ui. First it switched over to IPv6, to solve that:
1. Add this to /etc/modprobe.conf
# added 2/5/09 to disable ipv6 from starting -NJW
alias net-pf-10 off
alias ipv6 off
2. Update “NETWORKING_IPV6=no” in /etc/sysconfig/network
Then in /opt/pmx/etc/manager/httpd2/ssl Update default.conf to this:
# HTTPS Port — see old/ directory for previous/default config -NJW 2/5/2009
Listen *:28080
<VirtualHost *:28080>
SSLCertificateFile etc/manager/httpd2/sophos2007.pem
SSLEngine On
Include etc/manager/httpd2/rpc.conf
</VirtualHost>
Basically just changing to SSL for the port 28080.
Finally restart the server (required for IPv6 kernel?). If you see yourself needing to update the httpd configs, just restart it as the pmx user each time:
pmx-httpd restart
On the morning of Thursday, February 5th from 5:30 to 7:30am, TS will be performing upgrades to improve the effectiveness of our spam filtering system. During the upgrade, delivery of email into and out of the college may be delayed. E-mail sent may not arrive until after 7:30am. E-mail sent within the University is not affected by this outage (i.e. e-mail from one @ups.edu account to another).
[Update: 2/5 at 9:30am] Mail delivery was restored by 7:10 am; however, the update for the end user quarantine page took longer than expected. All services were restored by 9:30am.
On the morning of Tuesday, February 3rd, from 5:30-7:30am, TS will be performing upgrades to improve reliability of our spam filtering system. During the upgrade, some users will be unable to access their quarantined email messages in the Sophos Pure Message system. Should you experience problems accessing quarantined messages, please try again after 7:30am.
[Updated 2/3/2009 @ 7:45AM]
The quarantine is mostly available at this time, but a few users may find that certain messages are not available. We expect this to be resolved within 15-30 minutes.
On December 23rd from 5-10am the Sophos PureMessage spam may be unavailable for some messages/users. Mail will continue to be delivered as per normal. Should a user be unable to remove or deliver a message from their quarantine they should try again after 10am.
Due to a campus wide power outage from 5-7am on 12/23, this outage may be postponed depending on completion of that outage.
~~~~~~~~ Update 12/23 8:50am ~~~~~~~~~~~
This upgrade is progressing well, however it is taking a bit longer than expected. We may still make it up by 10am, but 11am is more realistic.
Please note that mail is being delivered normally and without delay, it’s just half of the spam quarantine is unavailable until 11am.
~~~~~~~ Update at 11:20 ~~~~~~~~~
The quarantine is now back online. From approximately 10:50-11:15am the quarantine was available off and on. Please contact the helpdesk if you have problems accessing your quarantine.
On the morning of November 6th at 5:30am messages sent from inside the University to external receipients will be delayed until 7am. The outage does not affect messages from University users to other University users.
—
Update: This outage is compete as of 6:10am on 11/6. All outgoing mail should now be back up to speed. Please contact the helpdesk if you see any irregular behavior.
On Friday 9/26 from 5am – 9:30am a redundant PureMessage server will be down for upgrades. As this server is redundant end users should not notice any downtime. Users may be unable to recover certain messages from the quarantine during this time.
Details:
Due to high iowait times support suggested we switch disks in the hades.ups.edu server to RAID10 (currently RAID5). To achive this we will rebuild the server via our automated installation method and then restore the PureMessage applications and data. Expected downtime for hades is 4.5 hours (1.5hr backup, 1hr disk rebuid, 1 hr OS load, 1hr restore)
Due to a problem with an update to the system that was done almost one year ago, the PureMessage quarantine has not been properly expiring, causing a buildup of SPAM messages in the quarantine and an inflation of the PM database. This caused the system to apparently “hold back” certain messages, generally ones that originated from listservs.
In order to correct this issue, we are currently running processes that will properly expire and reindex the quarantine and the metadata in the database. As this proceeds, the held back messages are delivered. Users will see the appearance of old emails in their inboxes. The number of affected messages seems small. Most people are not seeing any old messages appear, but many are. AS stated above, most of the affected messages appear to be from listservs and email subscription services.
Due to hardware problems, the spam filter end user interface (the web system where University members check their spam quarantine contents) will be unreliable. Service has been engaged, and we hope to have a resolution today.