More OTel metrics

[robinjam]

https://trello.com/c/25A3gpXh/1358-replace-statsd-instrumentation-across-notify-with-otel

Probably best reviewed with whitespace changes hidden.

Adds the following metrics, which should hopefully replace the existing StatsD ones (the dashboards will be updated in a future PR).

The names are automatically normalised to follow the Prometheus naming convention by the OTel collector, so e.g. provider.request.duration becomes provider_request_duration_seconds. I have deliberately chosen names that do not conflict with the existing metrics that go via the StatsD exporter, so we can have both stacks running in parallel.

---

provider.request.duration {error.type, notification.type, provider.name}

Duration of (HTTP) requests to providers

---

notification.sms.internationals {notification.status, notification.sms.country_code}

Number of international text messages sent

No screenshot for this one, but the behaviour should be identical to the existing notify_international_sms_total metric reported by StatsD.

---

notification.send.duration {error.type, key.type, notification.type, provider.name}

Elapsed time between notification creation and sending to provider

---

notification.deliver.duration {key.type, notification.status, notification.type, provider.name}

Elapsed time between notification creation datetime and delivery datetime reported by provider

[robinjam]

This is ready for review - the "WIP" in the title was a mistake

[risicle]

Might we be able to have international as a label in notification.deliver.duration? I realise it might be a bit of a big ask, because you've put the metric tallying before we've hit the db to find that out...

[risicle]

This made me think quite a bit about notification receipts, duplicate notification receipts and etcetera. I think you've made the right choice, but people using the metrics are going to need to understand that the statuses aren't "exclusive" (i.e. one notification will tally for both pending and e.g. permanent-failure). And using "delivery time" should hopefully nullify most weirdness that results from receipts making it to the front of the queue out-of-order. Assuming we can trust it.

Would it be paranoid to have another metric here in parallel using receipt_dt instead?

[robinjam]

Might we be able to have international as a label in notification.deliver.duration

Good shout, done. I've left it off notification.send.duration because I can't imagine the international-ness being relevant there

people using the metrics are going to need to understand that the statuses aren't "exclusive" (i.e. one notification will tally for both pending and e.g. permanent-failure).

Also a good point, I've updated the descriptions on both notification.sms.internationals and notification.delivery.duration. Weirdly only firetext notifications seem to get a "pending" callback, at least in the dev envs.

Would it be paranoid to have another metric here in parallel using receipt_dt instead?

Hmmmmm... Maybe, not sure what we'd call it though. notification.delivery_callback.duration?

I suppose the name doesn't matter that much as long as the description makes sense.

[risicle]

Weirdly only firetext notifications seem to get a "pending" callback, at least in the dev envs.

I think you're right, looking for "duplicate callback warning" log messages I only saw them from SES and firetext, and the firetext outcomes panel in the "Notify Summary" grafana dashboard has a little tooltip alluding to this.

[risicle]

Cool. Up to you on the parallel metric.