Should ApplicationError have a severity and/or custom time window?

[sleberknight]

This would allow us to have different levels of "error", e.g. WARN vs ERROR and we could also log at the corresponding level, whereas currently we only log at ERROR level.

Maybe it could also be tied into the health check and we set an appropriate level there as well, based on the highest priority unresolved error in the time window.

[sleberknight]

For the severity, we might consider using values such as:

enum ErrorSeverity {  NORMAL, HIGH, HIGHEST }

instead of WARN and ERROR so they can be (somewhat) independent of logging levels. For example, maybe we log NORMAL and HIGH at WARN level and HIGHEST at ERROR level. I'm not entirely sure I like these specific ones, so need to consider others as well.

In addition to a severity, it would sometimes be useful to be able to set a specific time window for how long an error is reported by the RecentErrorsHealthCheck. By default, all errors are reported if they occurred within the global time window (the default is 15 minutes). But for certain types of errors, we might want them to have a longer (or even shorter) time window before they are not reported by the health check anymore. For example, maybe application errors for an outage of a critical service dependency could have an expiration period of 1 hour.

Adding the ability to set a custom time window for an application error might have a negative impact on the APIs of ApplicationErrorDao, ApplicationErrors and ApplicationErrorThrower, which are currently pretty simple. The health check logic might become more complex, since it would now need to take into account custom expiration times. And we'd need a new property in ApplicationError and a corresponding new database column in the application_errors table. Then again, maybe setting an expiration time for each record in application_errors makes the queries and logic simpler, since we would only need to ask for application errors that are not expired.

ApplicationErrors and ApplicationErrorThrower would need to provide some way to set a specific time frame, which might require a bunch more overloaded methods or some other mechanism. Maybe instead of overloads, we provide a builder-style interface that lets you specify a custom time frame first:

// Using ApplicationErrors with the default time frame (works same as it always has)
ApplicationErrors.logAndSaveApplicationError(errorDao, LOG, exception,
        "Could not update Foo with id {}", foo.id());

// Using ApplicationErrors to set a custom time frame
ApplicationErrors.usingTimeWindow(Duration.ofHours(1))
        .logAndSaveApplicationError(errorDao, LOG, exception,
            "Could not update Foo with id {}", foo.id());

// Using ApplicationErrorThrower with the default time frame (works same as it always has)
errorThrower.logAndSaveApplicationError(exception,
        "Could not update Foo with id {}", foo.id());

// Using ApplicationErrorThrower to set a custom time frame
errorThrower.usingTimeWindow(Duration.ofHours(1))
        .logAndSaveApplicationError(exception,
             "Could not update Foo with id {}", foo.id());

This makes the internal code more complicated, but existing code does not need to change unless it wants to use a custom time window.