[Resolved] Email Unavailable for L-S Users

Update at 4:25PM

All users should have complete access to their email once again.  Some messages into and out of these users accounts may be delayed as the system catches up.

At this time Students and Technology Services staff members with usernames beginning in the letters L through S may not be able to access their email and calendars.  Technology Services is aware of the problem and is working quickly to resolve the issue.  We expect email to be accessible to all users by 6pm this evening and will update this post as conditions change.

Exchange 2007 Interface Upgrade Postponed

Updated 7/26/09 @ 3:10PM

All changes were backed out and all mail delivery is back to normal by 3:00 PM.

Due to unforeseen issues with the Exchange 2007 interface upgrade today, the change is postponed.

Currently we are reverting back to the previous configuration, which should be complete by 3:00 PM today.  During the change and roll-back periods (8:00 AM to 3:00 PM Sunday, August, 26th) some mail sent from outside of the University may not have been accepted.  Any user sending email that failed will get an error message and will be able to re-send the message after 3:00 PM.

This post will be updated when the roll-back is complete.

6/28/09 Maintenance Window Details

See the post in the “Maintenance Windows” catagory for the public communication.

This month the Sophos PMX servers were patched with the latest Linux patches, and upgraded to the latest PureMessage version.

We also did the following non-recuring tasks:

  • Upgraded Alexandria’s MD3000 Firmware
  • Upgraded AntiVirus to EPO on Webserver1 and Webserver2
  • Cleared stale VSS copies on SanJuan

Finally, all production Windows servers were patched:

  • Alexandria
  • BE1
  • BE2
  • BE3
  • CBORDFSS
  • CLTACC1
  • CLTACC2
  • CMS
  • DHCP-1
  • DM-1 to DM-8
  • EPO
  • FE1
  • FE2
  • ILLIAD
  • INTCHK
  • KEEPER
  • KRONOS
  • LENEL
  • MAURY2
  • MBS1
  • MBS2
  • MBS3
  • MEDIA
  • MERLIN2
  • MICROS
  • MSPROJECT
  • ODYSSEY
  • PROJECTS
  • SANJUAN
  • UPSCA03
  • VASHON
  • VERONICA
  • VIAWARP
  • VMMON
  • WEBSERVER1
  • WEBSERVER2

[Complete] Next Maintenance Window 6/28/09

This month’s outage was completed 1:30PM.  All services other than Cascade Web and Banner were available by noon as expected.

The next production maintenance window is Sunday, June 28th, from 8:00 AM to 4:00 PM.

From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:

  • E-mail
  • Personal and departmental file shares (on Alexandria and Merlin2)
  • Database Applications:  Cascade, FAMIS, Millennium, and Banner

Please note that Cascade Web and Banner will continue to be unavailable until 4:00 PM.

[Resolved] Spam Sent From The E-mail System 6/4/09

A spammer gained control of a user account today when the account owner responded to a phishing message. The spammer sent a large volume of spam. This was noted at 9:30 PM and the user account was locked. TS staff will be checking with several recipient sites such as HotMail and Yahoo to make sure that the college has not been placed on blacklists, but you may experience E-mail delivery problems because of this incident.

5/31/09 Maintenance Details

See the post in the “Maintenance Windows” catagory for the public communication.  This post is to expand on what servers were patched, and basic changes.  All of the following patches were applied by noon.

The following windows servers were patched (Windows updates):

ALEXANDRIA

BE1

BE2

BE3

CLTACC1

CLTACC2

CMS

DHCP-1

DM-1 to DM-8

EPO

FE1

FE2

ILLIAD

INTCHK

KEEPER

MAURY2

MBS1

MBS2

MBS3

MEDIA

MERLIN2

PROJECTS

SANJUAN

VASHON

VERONICA

VIAWARP

VMMON

WEBSERVER1 (SP2 applied)

WEBSERVER2 (SP2 applied)

VMWare:

vmhost1

Linux (all available OS and Dell RAID FW/Drives as applicable):

orcas

tahoma

purgatory

styx

hades

gehenna

Additional changes on Sophos PMX servers:

Purgatory:

Turned Perc write cache on (write back), and changed linux readahead:

sudo blockdev –setra 8388608 /dev/sda
Rest:
policy on
adaptive read ahead

Messages Quarantined Before 9:00 AM Unavailable on Web 4/27/2009

Spam messages quarantined prior to 9:00 AM on April 27, 2009, are no longer available on the PureMessage Web site due to the domain name change which took effect at that time. Messages arriving after 9:00 AM will be available as usual.

If you need messages quarantined before this time, please contact the HelpDesk at helpdesk@ups.edu or x8585.

[Resolved] Incoming E-mail Disrupted

[Update 04/21/2009 03:17 PM] E-mail service has been restored. All mail will be delivered shortly.

Mail inbound to pugetsound.edu has been disrupted. We are working to restore service. No mail has been lost, and all mail will be delivered. We will update you as soon as possible.

[Resolved] CX400 Battery Replacement Tuesday February 17th, 5:30AM

On February 17th from 5:30-6AM batteries in our EMC CX400 Disk Array will be replaced.  This is a fully redundant system and no outage is expected.  However, as with any change, there is always a risk of outage.  If a problem arises with the change, the following services may be impacted:

email
CRM
Database Services (FAMIS, Banner, Millennium, etc.)

Please note that Cascade will not be affected.

Sophos Upgrade, SSL goes away on CSM/End User Web UI

Today we upgraded sophos puremessage on all servers, and in doing so we had several issues with the end user web ui.  First it switched over to IPv6, to solve that:

1.  Add this to /etc/modprobe.conf

# added 2/5/09 to disable ipv6 from starting -NJW
alias net-pf-10 off
alias ipv6 off

2.  Update “NETWORKING_IPV6=no” in /etc/sysconfig/network

Then in /opt/pmx/etc/manager/httpd2/ssl Update default.conf to this:

# HTTPS Port — see old/ directory for previous/default config -NJW 2/5/2009
Listen *:28080
<VirtualHost *:28080>
SSLCertificateFile etc/manager/httpd2/sophos2007.pem
SSLEngine On
Include etc/manager/httpd2/rpc.conf
</VirtualHost>

Basically just changing to SSL for the port 28080.

Finally restart the server (required for IPv6 kernel?).  If you see yourself needing to update the httpd configs, just restart it as the pmx user each time:

pmx-httpd restart