BJP was modified to error appropriately when a runrep command fails. BJP was stopped and restarted at 12:30 pm for the new code to take effect.
Resolved: Mail list services unavailable (listhost.ups.edu)
As of noon today all mail list services are back online. Please contact TechHelp at x8585 if you continue to experience problems with lists.
Early this morning our mail list server had a problem and is not working properly. We are working on resolving the issue, but it is not likely to be resolved before 5pm today. We will update this post as we have more information.
[Resolved] ResNet Internet Disruptions 12/3/09
The ResNet internet connection was down from 12:00 midnight to 3:00 AM this morning. Service has been restored.
wiki.ups.edu moved to new server, userweb.pugetsound.edu
Today between 5pm and 6pm wiki.ups.edu was moved over to userweb.pugetsound.edu. No problems were encountered, but please let us know if you encounter issues.
Nick Webb x3185
Crm restarted 12/1/2009
Both Crm11ip instance and Crm App had to be restarted due to the exceeded processes on the databasee. The problem was caused by the returning mails sent last week.
Installed CpuOct2009 Patch 11/29/2009
Installed Oracle Critical Patch, CpuOct2009, on Cascade, Banner, Famis, Millennium, and Rman databases.
Orcas Reboot 11/23/09 @ 5:30am
Orcas needs a reboot to properly configure VMWare tools, the misconfiguration is causing performance issues on Orcas. All services will be available by 6am.
BJP migrated to Orcas – 11/17/2009
BJP was migrated off of Camano, onto Orcas
CRM bounce-backs weren’t processing
We discovered that the CRM email bounce-backs were not being processed, resulting in a very full mailbox. Nick renamed the inbox, essentially cleaning out the mailbox, thinking that maybe the box was too full to process the messages properly. This appears to have fixed the problem. We will look into whether these are being cleaned out properly when the bounceback is processed (or, being marked for deletion but then never deleted).
We also discovered that messages were being routed to the Junk email mailbox. CRM only processes bounce-backs found in the Inbox. We suspect that our use of Outlook to access the bounces email account has caused this to occur. We will look into using a tool other than Outlook for accessing this IMAP account.
Maintenance Window 11/29/09
All production databases were patched successfully. If you experience any issues with Puget Sound systems, please contact TechHelp at (253) 879-8585.
The next production maintenance window is Sunday, November 29th, from 8:00 AM to noon. From 8:00 AM to noon, all database services may be intermittently or entirely unavailable, including: Cascade, FAMIS, Millennium, and Banner.
[Complete] Maintenance Window 10/25/09
Updated 10/25/2009 @ noon — All production servers were patched successfully. If you experience any issues with Puget Sound systems, please contact TechHelp at (253) 879-8585.
The next production maintenance window is Sunday, October 25th, from 8:00 AM to noon.
From 8:00 AM to Noon, all services may be intermittently or entirely unavailable, including:
- Personal and departmental file shares (on Alexandria and Merlin2)
- Database Applications: Cascade, FAMIS, Millennium, and Banner
Maintenance Detail 10/25/2009
The following systems were patched today between 8am and noon:
- sthelens
- baker
- Tahoma
- userweb
- purgatory
- styx
- hades
- gehenna
- moodle
- quixote
- mysql
- oraweb
- orcas
- vmhost1
- Alexandria
- BE1
- BE2
- BE3
- CBORDFSS (VENDOR DC)
- CLTACC1
- CLTACC2
- CMS
- DHCP-1
- DM-1 to DM-8
- EPO
- FE1
- FE2
- ILLIAD
- INTCHK (VM)
- KEEPER
- KRONOS (VENDOR DC)
- LENEL (VENDOR Security)
- MAURY2
- MBS1
- MBS2
- MBS3
- MEDIA
- MERLIN2
- MICROS (VENDOR DC)
- MSPROJECT
- ODYSSEY (VENDOR DC)
- PROJECTS
- SANJUAN
- UPSCA03
- VASHON
- VERONICA
- VIAWARP (VM)
- VMMON (VM for now)
- WEBSERVER1
- WEBSERVER2
- CISCOWORKS (VM)
[Complete] PreProduction Maintenance Window 10/15/2009
This month, on 10/15/2009 from 7am to 9pm we performed our monthly patch maintenance for PreProduction servers. We patched the following systems:
- VMhost2
- Pilchuck
- Lummi
- Web
- moodle2
- kickstart
- Exdev
- Fidalgo
- Galaxy
- Ketron
- Portal
- VS0
[Resolved] Exchange Email Unavailable for Users EJ-LB
As of 9:00pm 10/13/09 all users have complete access to their email once again. Some messages into and out of these users accounts may be delayed as the system catches up.
Update as of 4:45PM 10/13/09
The outage of email for users EJ-LB continues as we run a database recovery process on the affected volume/users. The process could take up to 18 hours, completing by mid-morning Wednesday (10am).
This post will be updated as more information is available.
Exchange Email is unavailable for users with usernames starting with EJ-LB. Currently there is not a time estimate of when email for these users will be available again, but this post will be updated when the issue is corrected.
Aruba Wireless Controller License and software upgrade
10/10/2009 The Aruba wirless controller was upgraded and additional new licensing applied starting at 4:30am. The controller was rebooted at 5:20am. The process is complete at 5:30am. Monitoring until 6:00am shows all access points are functioning properly.