Firedrills

This firedrills repository contains the scripts for testing automated emails from the chandra MTA team's users to email distribution groups. These script are built as minimally as possible and are typically run via a scheduled cronjob to replicate automated MTA scripts.

Procedure

Pre-Contact

MTA prefers to test the occ_red_alert@ipa in conjunction with replan_alerts@ipa, however we don’t have access to this email group. Contact David Jones, Jason Araujo and/or Chris Eagan with an alert schedule to request that they send out the occ_red_alert@ipa test at a requested time.

Schedule

Typically alerts are tested at the start of the workday on a Friday. You can find example schedules in the crontab of mta@boba-v. The firedrill scripts will automatically send the test emails, they just need to be scheduled into the crontab.

# Crontab for mta@boba-v
# Dec 8, 09:15-09:50am, @ipa, @cfa, occ (JasonA, 09:45), replan_alerts
15 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_cfa_red.pl
20 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_cfa_yellow.pl
25 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_cfa_safemode.pl
30 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_ipa_red.pl
35 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_ipa_yellow.pl
40 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_ipa_safemode.pl
50 9 8 12 * cd /data/mta4/www/Snapshot; ./firedrill_replan_alerts.pl

Inform sot_yellow_alert of Test

To prepare people for the test, inform them by sending an email to the sot_yellow_alert@cfa.harvard.edu google group. This will automatically open an Opsgenie alert as well, but since this alert is meant to be informational, change the priority from P2(High) to P5(informational), mark it as Acked, and Closed so that it’s not kept around as a real alert for everyone. You may also create the notification in Opsgenie directly, btu be sure to mark ALL_USERS as the recipient and

It’s also worthwhile to send a reminder email the day before the test as well as remind people during the 9:00 am standup morning meeting.

Example Email

TO: sot_yellow_alert@cfa.harvard.edu

Subject: [sot_yellow_alert] MTA alert testing Friday 12/08/23, 9:15 am - 9:50 am

A test of MTA alerts will take place on Friday 12/08/23, starting at 9:15AM EST and finishing at 9:50AM EST.

The following alerts will be issued to the following notification type lists @cfa, @ipa, the occ_red_alert@ipa and replan_alerts@cfa lists: @cfa

9:15AM: sot_red_alert@cfa 9:20AM: sot_yellow_alert@cfa 9:25AM: sot_safemode_alert@cfa

@ipa

9:30AM: sot_red_alert@ipa 9:35AM: sot_yellow_alert@ipa 9:40AM: sot_safemode_alert@ipa

OCC/MP

9:45AM: occ_red_alert@ipa 9:50AM: replan_alerts@cfa

All alerts will be prominently marked in the headline and in the description with [TEST TEST TEST] to make clear that the alerts are not real.

Depending on your settings, expect up to 8 email notifications and up to 8 sms notifications: x two -- because of the direct MTA notifications and ALRMNS notifications -- as well as up to 8 voice notifications from ALRMNS.

Please refer to ALRMNS Web on Chandra TWiki for help in getting your ALRMNS account set up. You may choose to tune into all, or the subset of interest to you.

Please contact:

Scott Wolk with questions regarding MTA mailing/sms lists setup. David Jones with questions regarding ALRMNS account setup.

Thank you, Will