BJP was migrated off of Camano, onto Orcas
Author Archives: kspiese
CRM bounce-backs weren’t processing
We discovered that the CRM email bounce-backs were not being processed, resulting in a very full mailbox. Nick renamed the inbox, essentially cleaning out the mailbox, thinking that maybe the box was too full to process the messages properly. This appears to have fixed the problem. We will look into whether these are being cleaned out properly when the bounceback is processed (or, being marked for deletion but then never deleted).
We also discovered that messages were being routed to the Junk email mailbox. CRM only processes bounce-backs found in the Inbox. We suspect that our use of Outlook to access the bounces email account has caused this to occur. We will look into using a tool other than Outlook for accessing this IMAP account.
[Resolved] CRM unavailable
CRM, the system for sending campus-wide bulk messages, has experience an unexpected outage.
Technology Services is currently working to resolve the issue.
Service was restored to CRM at 3:59 pm.
Reports server bounced, 4/27/2009
The reports server experienced the same problem as last Monday. Bounced the server to see if it resolves the issue. Paul will continue to research the cause of this problem.
Reports Server bounced
Cascade and FAMIS users reported issues receiving their reports that are run through the reports server. Users received an error message saying that job ###### wouldn’t run. When we looked in the reports server (past jobs), that particular job_id did not show up (it was a future job_id!).
Resolution: bounced reports server
[Completed] Banner Unavailable 2/20 – 2/23/2009
Banner, Budget Administration, and Your Employment information will be unavailable from 5:00 p.m. on Friday, February 20, until 7:00 a.m. on Monday, February 23 due to the Banner migration to new hardware.
[Update 2/22/09 @ 12:00 noon] All services are back on-line. The upgrade went as planned and ahead of schedule.
[Resolved] Server performance problem, 2/20/2009
At 11:30 am, one of the servers began experiencing performance problems. Banner, CRM, FAMIS, and Cascade batch processing are impacted by this.
Update: 2:05 pm, the performance problem on Camano has been resolved. CRM is still down.
Update: 2:45 pm, service has been restored on CRM
[Resolved] Cascade batch processing unavailable, 1/30/2009
Cascade’s batch processing system went down unexpectedly at 11:15 am. We are working to restore this service.
Batch processing service was restored at 11:51 am.
January 11, 2009: Cascade Web unavailable (back online at 12:15 pm)
Cascade Web is experiencing some unexpected complications related to yesterday’s Cascade database upgrade. Technology Services staff are working to resolve the issue as quickly as possible.
Cascade Web was brought back up around 12:15 pm.
Databases unavailable, 1/10/2009
Campus email text:
Saturday, January 10, Technology Services will be performing important updates on our database systems. There will be two groups of outages:
5 AM until 8 PM – Cascade and Cascade Web will be unavailable.
9 AM until Noon – Banner, Famis, Basis, and Millennium will be unavailable.
Email, network files and shares, and other network services will not be impacted.
Details:
Cascade database is switching to new hardware and upgrading to 10g. Administrative toolset is being upgraded to 10g and deployed on new hardware.
Banner, Famis, Basis, and Millennium will be patched to make them compatible with 10g.
November 24, 2008: CRM downtime, 6 am – 8 pm
CRM will be down from 6 am – 8 pm on Monday, November 24. The database instance will be patched then moved from Rainier to Grace.
Cascade Batch Processing outage 11/19
Cascade Batch Processing system was down for approximate 30 minutes, from 3:00 pm to 3:30 pm.
CRM application tier bounced at 10:20 am on 10/1/2008
CRM fulfillment experienced an error during the processing of an email (rollback segment problem). Restarting the email caused a runaway cpu-intensive session even though the email was subsequently cancelled.
Cause of runaway session: user created content using the wrong content type (admission content type instead of campus email), which relied on admission data being in the system.
Resolution: We bounced the application server and killed the runaway session. User recreated email with correct content type.
CRM unavailable – 4/25/2008
CRM was temporarily unavailable for for about 40 minutes.