[Completed] Sophos SPAM Quarantine Offline 8/20/09 7-8:30am

Update 8/20/09 @ 9:30am — The change encountered an issue and services were down until 9am.

On the morning of Thursday, August 20th from 7 – 8:30am users will not be able to access their quarantined spam messages on sophos.ups.edu.  During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.

This outage is necessary to make performance improvements to this very busy system.

[Complete] Sophos SPAM Quarantine Offline 8/7/09 6:30-9:00am

Updated 8/7/09 at 10:45am:

Sophos systems are back up and operational.  Unfortunately the maintenance planned had to be backed out, so we will need to re-apply the changes at a future date/time.

Updated 8/7/09 at 10:00am:

We are still experiencing problems with the update.  Please stand by for more information.

Updated 8/7/09 at 8:15am:

We experienced some difficulty during maintenance and we expect the window to extend until 9:30am at this time.  All services should be up by 10am; sorry for the inconvenience.

On the morning of Friday, August 7th from 6:30 – 9:30am users will not be able to access their quarantined spam messages on sophos.ups.edu.  During this time all email messages into and out of the University will continue to flow normally and any spam messages captured during this time may be retrieved after 9am.

This outage is necessary to make performance improvements to this very busy system.

6/28/09 Maintenance Window Details

See the post in the “Maintenance Windows” catagory for the public communication.

This month the Sophos PMX servers were patched with the latest Linux patches, and upgraded to the latest PureMessage version.

We also did the following non-recuring tasks:

  • Upgraded Alexandria’s MD3000 Firmware
  • Upgraded AntiVirus to EPO on Webserver1 and Webserver2
  • Cleared stale VSS copies on SanJuan

Finally, all production Windows servers were patched:

  • Alexandria
  • BE1
  • BE2
  • BE3
  • CBORDFSS
  • CLTACC1
  • CLTACC2
  • CMS
  • DHCP-1
  • DM-1 to DM-8
  • EPO
  • FE1
  • FE2
  • ILLIAD
  • INTCHK
  • KEEPER
  • KRONOS
  • LENEL
  • MAURY2
  • MBS1
  • MBS2
  • MBS3
  • MEDIA
  • MERLIN2
  • MICROS
  • MSPROJECT
  • ODYSSEY
  • PROJECTS
  • SANJUAN
  • UPSCA03
  • VASHON
  • VERONICA
  • VIAWARP
  • VMMON
  • WEBSERVER1
  • WEBSERVER2

[Complete] Next Maintenance Window 6/28/09

This month’s outage was completed 1:30PM.  All services other than Cascade Web and Banner were available by noon as expected.

The next production maintenance window is Sunday, June 28th, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

  • E-mail
  • Personal and departmental file shares (on Alexandria and Merlin2)
  • Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Cascade Web and Banner will continue to be unavailable until 4:00 PM.

5/31/09 Maintenance Details

See the post in the “Maintenance Windows” catagory for the public communication.  This post is to expand on what servers were patched, and basic changes.  All of the following patches were applied by noon.

The following windows servers were patched (Windows updates):

ALEXANDRIA

BE1

BE2

BE3

CLTACC1

CLTACC2

CMS

DHCP-1

DM-1 to DM-8

EPO

FE1

FE2

ILLIAD

INTCHK

KEEPER

MAURY2

MBS1

MBS2

MBS3

MEDIA

MERLIN2

PROJECTS

SANJUAN

VASHON

VERONICA

VIAWARP

VMMON

WEBSERVER1 (SP2 applied)

WEBSERVER2 (SP2 applied)

VMWare:

vmhost1

Linux (all available OS and Dell RAID FW/Drives as applicable):

orcas

tahoma

purgatory

styx

hades

gehenna

Additional changes on Sophos PMX servers:

Purgatory:

Turned Perc write cache on (write back), and changed linux readahead:

sudo blockdev –setra 8388608 /dev/sda
Rest:
policy on
adaptive read ahead

Messages Quarantined Before 9:00 AM Unavailable on Web 4/27/2009

Spam messages quarantined prior to 9:00 AM on April 27, 2009, are no longer available on the PureMessage Web site due to the domain name change which took effect at that time. Messages arriving after 9:00 AM will be available as usual.

If you need messages quarantined before this time, please contact the HelpDesk at helpdesk@ups.edu or x8585.

Sophos Upgrade, SSL goes away on CSM/End User Web UI

Today we upgraded sophos puremessage on all servers, and in doing so we had several issues with the end user web ui.  First it switched over to IPv6, to solve that:

1.  Add this to /etc/modprobe.conf

# added 2/5/09 to disable ipv6 from starting -NJW
alias net-pf-10 off
alias ipv6 off

2.  Update “NETWORKING_IPV6=no” in /etc/sysconfig/network

Then in /opt/pmx/etc/manager/httpd2/ssl Update default.conf to this:

# HTTPS Port — see old/ directory for previous/default config -NJW 2/5/2009
Listen *:28080
<VirtualHost *:28080>
SSLCertificateFile etc/manager/httpd2/sophos2007.pem
SSLEngine On
Include etc/manager/httpd2/rpc.conf
</VirtualHost>

Basically just changing to SSL for the port 28080.

Finally restart the server (required for IPv6 kernel?).  If you see yourself needing to update the httpd configs, just restart it as the pmx user each time:

pmx-httpd restart

[Resolved] E-mail Delivery Delay Thursday 2/5/2009 5:30-7:30am – Part 2 of 2

On the morning of Thursday, February 5th from 5:30 to 7:30am, TS will be performing upgrades to improve the effectiveness of our spam filtering system. During the upgrade, delivery of email into and out of the college may be delayed.  E-mail sent may not arrive until after 7:30am.  E-mail sent within the University is not affected by this outage (i.e. e-mail from one @ups.edu account to another).

[Update: 2/5 at 9:30am] Mail delivery was restored by 7:10 am; however, the update for the end user quarantine page took longer than expected. All services were restored by 9:30am.

[Resolved] Spam Quarantine Unavailable Tuesday 2/3/2009 5:30-7:30am – Part 1 of 2

On the morning of Tuesday, February 3rd, from 5:30-7:30am, TS will be performing upgrades to improve reliability of our spam filtering system. During the upgrade, some users will be unable to access their quarantined email messages in the Sophos Pure Message system.  Should you experience problems accessing quarantined messages, please try again after 7:30am.

[Updated 2/3/2009 @ 7:45AM]

The quarantine is mostly available at this time, but a few users may find that certain messages are not available.  We expect this to be resolved within 15-30 minutes.

December 23, 2008: Spam Quarantine Unavailable 5-10am

On December 23rd from 5-10am the Sophos PureMessage spam may be unavailable for some messages/users.  Mail will continue to be delivered as per normal.  Should a user be unable to remove or deliver a message from their quarantine they should try again after 10am.

Due to a campus wide power outage from 5-7am on 12/23, this outage may be postponed depending on completion of that outage.

~~~~~~~~ Update 12/23 8:50am ~~~~~~~~~~~

This upgrade is progressing well, however it is taking a bit longer than expected.  We may still make it up by 10am, but 11am is more realistic.

Please note that mail is being delivered normally and without delay, it’s just half of the spam quarantine is unavailable until 11am.

~~~~~~~ Update at 11:20 ~~~~~~~~~

The quarantine is now back online.  From approximately 10:50-11:15am the quarantine was available off and on.  Please contact the helpdesk if you have problems accessing your quarantine.

Outgoing Email Outage on 11/6 from 5:30-7am (Update- Completed at 6:10am)

On the morning of November 6th at 5:30am messages sent from inside the University to external receipients will be delayed until 7am.  The outage does not affect messages from University users to other University users.

Update:  This outage is compete as of 6:10am on 11/6.  All outgoing mail should now be back up to speed.  Please contact the helpdesk if you see any irregular behavior.

PureMessage Edge Server Maintance 9/26

On Friday 9/26 from 5am – 9:30am a redundant PureMessage server will be down for upgrades.  As this server is redundant end users should not notice any downtime.  Users may be unable to recover certain messages from the quarantine during this time.

Details:

Due to high iowait times support suggested we switch disks in the hades.ups.edu server to RAID10 (currently RAID5).  To achive this we will rebuild the server via our automated installation method and then restore the PureMessage applications and data.  Expected downtime for hades is 4.5 hours (1.5hr backup, 1hr disk rebuid, 1 hr OS load, 1hr restore)

February 1st – PureMessage Problems

Due to a problem with an update to the system that was done almost one year ago, the PureMessage quarantine has not been properly expiring, causing a buildup of SPAM messages in the quarantine and an inflation of the PM database. This caused the system to apparently “hold back” certain messages, generally ones that originated from listservs.

In order to correct this issue, we are currently running processes that will properly expire and reindex the quarantine and the metadata in the database. As this proceeds, the held back messages are delivered. Users will see the appearance of old emails in their inboxes. The number of affected messages seems small. Most people are not seeing any old messages appear, but many are. AS stated above, most of the affected messages appear to be from listservs and email subscription services.