A lot of checks are marked as overdue, but checks are actually run by icinga

[alamp1360]

Issue with overdue checks.

I am using the latest Icinga version ( v2.16.0-19-g6c73bb4af ).

All the Icinga services are running as docker containers. I am using IcingaDB and a MySQL database.

The issue is that i experience a lot of overdue service and host checks in IcingaWeb.

I tried to analyze the issue with claude code, and the AI tells me that the issue is caused by Icinga not updating the field icinga:nextupdate:service in Redis.
And indeed, when i check the value for one of the overdue services, i can see that it does not get updated.

IcingaCli:

ICINGA2_API_USERNAME=root ICINGA2_API_PASSWORD=pasword icinga2 console --connect 'https://localhost:5665/'
Icinga 2 (version: v2.16.0-19-g6c73bb4af)

<36> => DateTime(get_time()).to_string()
"2026-05-13 15:05:54 +0200"
<37> => get_service("servername", "servicename").icingadb_identifier
"9ead43c6c44fea483ca447ae41250db6ce4d9b96"
<38> => get_service("servername", "servicename").next_check + ": " + DateTime(get_service("servername", "servicename").next_check).to_string()
"1778677714.420000: 2026-05-13 15:08:34 +0200"
<39> => get_service("servername", "servicename").next_update + ": " + DateTime(get_service("servername", "servicename").next_update).to_string()
"1778678014.515168: 2026-05-13 15:13:34 +0200"

In Redis:

docker exec ff5db6698a2c redis-cli ZSCORE icinga:nextupdate:service 9ead43c6c44fea483ca447ae41250db6ce4d9b96 |  jq -r '. as $ts | {raw: $ts, human: ($ts | floor | strflocaltime("%Y-%m-%d %H:%M:%S"))}'
{
  "raw": 1778673826.963446,
  "human": "2026-05-13 14:03:46"
}

As one can see, the service state values for next_check and next_update are recent (Icingaweb does display the same values in the "Source" view of the service), but in redis, the nextupdate SortedSet for that service is very outdated. This field is used for deciding if a service is overdue.

Strange thing is that not all the checks are overdue. Currently there are about 4100 service checks not overdue and about 2500 service checks are overdue.

Claude told me that there seemed to be a change (Commit e88366d) from Icinga version 2.15.3 to more recent versions so that this field is no longer updated "directly" in IcingaDB::NewCheckResultHandler, but by a signal "OnNextCheckChanged". If this signal does not "fire" for some reason, then that field does not get updated in Redis.
Forgive me if this is nonsense, but maybe it helps :)

[yhabteab]

Hi, thanks for reporting!

Claude told me that there seemed to be a change (Commit e88366d) from Icinga version 2.15.3 to more recent versions so that this field is no longer updated "directly" in IcingaDB::NewCheckResultHandler, but by a signal "OnNextCheckChanged". If this signal does not "fire" for some reason, then that field does not get updated in Redis

There was indeed a change in that regard but that's not quite the same commit. As part of #10619, I've changed the implementation of the IcingaDB::NewCheckResultHandler method with commit d364ad9 as follows:

Note that there is no corresponding NextUpdate bit set for the previous rw->SendNextUpdate() call due to an assumption that if the OnNewCheckResult signal is triggered, then we should have already triggered a OnNextCheckChanged previously. I'm not sure why and how that signal doesn't get triggered as can be seen below:

icinga2/lib/icinga/checkable-check.cpp

Lines 374 to 391 in 6c73bb4

	if (!origin) {
	if (cr->GetActive()) {
	UpdateNextCheck();
	} else {
	/* Reschedule the next check for external passive check results. The side effect of
	* this is that for as long as we receive results for a service we
	* won't execute any active checks. */
	double offset;
	double ttl = cr->GetTtl();
	if (ttl > 0)
	offset = ttl;
	else
	offset = GetCheckInterval();
	SetNextCheck(Utility::GetTime() + offset);
	}
	}

Strange thing is that not all the checks are overdue. Currently there are about 4100 service checks not overdue and about 2500 service checks are overdue.

Can you tell any difference between the services that are in overdue and those that are not? Is there any pattern you can deduce something, e.g., are local or remote checks affected by this etc.? Also, if you're using docker maybe you can rebuild the image with the following patch applied and see if that fixes the issue but I've not sure why that signal should get lost somewhere.

diff --git a/lib/icingadb/icingadb-objects.cpp b/lib/icingadb/icingadb-objects.cpp
index c5dc084c0..770c23e52 100644
--- a/lib/icingadb/icingadb-objects.cpp
+++ b/lib/icingadb/icingadb-objects.cpp
@@ -3019,7 +3019,7 @@ void IcingaDB::FlappingChangeHandler(const Checkable::Ptr& checkable, double cha
 void IcingaDB::NewCheckResultHandler(const Checkable::Ptr& checkable)
 {
 	for (auto& rw : ConfigType::GetObjectsByType<IcingaDB>()) {
-		rw->EnqueueConfigObject(checkable, icingadb::task_queue::VolatileState);
+		rw->EnqueueConfigObject(checkable, icingadb::task_queue::VolatileState | icingadb::task_queue::NextUpdate);
 	}
 }

Note that the patch is based on the v2.16.0 release tag.

[capull0]

We can confirm similar behaviour after upgrading to 2.16.0.

We are running two HA Icinga2 clusters (master-master) with IcingaDB and a central PostgreSQL database:

• QA cluster: ~230 hosts / ~3300 services
• Production cluster: ~1000 hosts / ~17000 services

Both clusters are writing performance data via the influxdb-writer feature into a VictoriaMetrics cluster.

Since upgrading to 2.16.0, we started seeing multiple situations per day where nearly all checks became overdue simultaneously.

As a test, we disabled the influxdb-writer feature on both clusters. Since then, both environments have been stable again and no mass-overdue situations occurred anymore.

This looks very similar to the symptoms described here and might also be related to #10842.

At the moment, the correlation between enabling the influxdb writer and the overdue storms is very strong in our environment.

From the log when it starts:

Mai 13 11:04:45 poma-mon-bs01 icingadb[1286207]: high-availability: Continuing being the active instance

Mai 13 11:05:29 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 7 host overdue indicators

Mai 13 11:05:29 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 1743 service overdue indicators

Mai 13 11:05:49 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 187 host overdue indicators

Mai 13 11:05:49 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 4805 service overdue indicators

Mai 13 11:06:05 poma-mon-bs01 icingadb[1286207]: history-sync: Synced 1 state history items

Mai 13 11:06:09 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 235 host overdue indicators

Mai 13 11:06:09 poma-mon-bs01 icingadb[1286207]: overdue-sync: Synced 5510 service overdue indicators

[alamp1360]

I applied the proposed patch and, after struggling with the possible outdated Dockerfile from https://github.com/icinga/docker-icinga2 repo (libprotobuf-dev and protobuf-compiler are missing in build-icinga2 stage and libprotobuf-lite32 is missing in the icinga stage) i was able to build a new image, which reports "v2.16.0-20-g93805d97e" as icinga2 version.
Starting that image the first time, the overdue errors did not appear. I let Icinga run for about 30 minutes.
Then i did a restart, and unfortunately those overdues started to appear again.
So i did another restart, which made all the overdues disappear at first, but then they all re-appeared after some seconds/minutes.

It is a mixture of host- and service-checks, even a "dummy" host-check is overdue. And, strangely enough, services which are in UNKNOWN state because the host is unrechable, are overdue, too. As looking through the checks, it seems all of them are affected. The ones not being overdue at the moment are checks which run only once per hour or once per day. So i guess if i leave it running for a few days or so, all the checks would become overdue.

When i do a "check now" for a service/host, the "execution_start" and "execution_end" attributes do change immediately, so it seems the check is actually being run. But the status keeps being "overdue".

I am not using influx-db writer, but graphite-writer instead. So i cannot confirm if the issues is related to influx-db writer.

Please let me know if i should collect any more data to help you identify and solve the issue.

[alamp1360]

I have collected additional information which might help to identify the issue:
When the issue arises, then there are absolutely no nextupdate updates sent to REDIS.

Strange thing is, that this does not happen always. Sometimes those updates are working, but, after a restart of icinga, the updates stop coming in.
I tried to add debug statements to numerous places into Icinga's icingaDB code. All i can see is that those updates are enqueued via IcingaDB::EnqueueConfigObject, but then they are not further processed by the IcingaDB::PendingItemsThreadProc.

[julianbrost]

@capull0 At first glance, your issue sounds more like it could be related to #10825.

@alamp1360

I applied the proposed patch and, after struggling with the possible outdated Dockerfile from https://github.com/icinga/docker-icinga2 repo (libprotobuf-dev and protobuf-compiler are missing in build-icinga2 stage and libprotobuf-lite32 is missing in the icinga stage) i was able to build a new image, which reports "v2.16.0-20-g93805d97e" as icinga2 version.

We moved it to this repo, so the current version can be found here: https://github.com/Icinga/icinga2/blob/v2.16.0/Containerfile

I am not using influx-db writer, but graphite-writer instead. So i cannot confirm if the issues is related to influx-db writer.

One of the changes in 2.16.0 is that we've switched the individual perfdata writers to a common connection handling implementation. There apparently are some problems with that which we still need to sort out, see #10825.

Though from the problems you've described, I'm not really sure if that's actually the same issue you're describing. Can you please check whether your Graphite still receives the data you're expecting?

Strange thing is, that this does not happen always. Sometimes those updates are working, but, after a restart of icinga, the updates stop coming in. I tried to add debug statements to numerous places into Icinga's icingaDB code. All i can see is that those updates are enqueued via IcingaDB::EnqueueConfigObject, but then they are not further processed by the IcingaDB::PendingItemsThreadProc.

Can you please add checks with the built-in icingadb CheckCommand (if they don't already exist) and check what the icinga2_pending_config_and_state_updates_count and icinga2_pending_config_and_state_updates_backlog metrics report? Try to observe them over a bit of time. You might want to query the information directly from the Icinga 2 REST API given that if the Icinga DB feature has problems, it may not show properly up-to-date information in Icinga Web.

[alamp1360]

I guess we (Claude Code and i) finally found the issue :)
It is related to locking: Both IcingaDB and GraphiteWriter are locking the same object. The issue is, that GraphiteWriter does not give up the Lock as long as the performance data is not 100% written to Graphite. And if there is an issue with graphite, since the routine which actually sends the data to graphite tries to send the data literally forever, the lock is never given up, so IcingaDB never gets the lock...

InfluxDB Writer seems to have the same issue with not giving up the lock.

The proposed fix would be to a) give up the lock before sending the data to graphite and b) to implement a timeout in the performancedata send routine.

Patch for a)

*** lib/perfdata/graphitewriter.cpp_save        2026-05-18 14:53:51.958358000 +0200
--- lib/perfdata/graphitewriter.cpp     2026-05-18 15:09:08.953605000 +0200
***************
*** 224,262 ****

        CheckCommand::Ptr checkCommand = checkable->GetCheckCommand();

!       ObjectLock olock(perfdata);
!       for (const Value& val : perfdata) {
!               PerfdataValue::Ptr pdv;
!
!               if (val.IsObjectType<PerfdataValue>())
!                       pdv = val;
!               else {
!                       try {
!                               pdv = PerfdataValue::Parse(val);
!                       } catch (const std::exception&) {
!                               Log(LogWarning, "GraphiteWriter")
!                                       << "Ignoring invalid perfdata for checkable '"
!                                       << checkable->GetName() << "' and command '"
!                                       << checkCommand->GetName() << "' with value: " << val;
!                               continue;
                        }
-               }

!               String escapedKey = EscapeMetricLabel(pdv->GetLabel());
!               double ts = cr->GetExecutionEnd();

!               SendMetric(checkable, prefix, escapedKey + ".value", pdv->GetValue(), ts);

!               if (GetEnableSendThresholds()) {
!                       if (!pdv->GetCrit().IsEmpty())
!                               SendMetric(checkable, prefix, escapedKey + ".crit", pdv->GetCrit(), ts);
!                       if (!pdv->GetWarn().IsEmpty())
!                               SendMetric(checkable, prefix, escapedKey + ".warn", pdv->GetWarn(), ts);
!                       if (!pdv->GetMin().IsEmpty())
!                               SendMetric(checkable, prefix, escapedKey + ".min", pdv->GetMin(), ts);
!                       if (!pdv->GetMax().IsEmpty())
!                               SendMetric(checkable, prefix, escapedKey + ".max", pdv->GetMax(), ts);
                }
        }
  }

--- 224,286 ----

        CheckCommand::Ptr checkCommand = checkable->GetCheckCommand();

!       struct MetricEntry {
!               String escapedKey;
!               double value;
!               Value crit, warn, min, max;
!       };
!
!       std::vector<MetricEntry> metrics;
!       double ts = cr->GetExecutionEnd();
!
!       {
!               ObjectLock olock(perfdata);
!               for (const Value& val : perfdata) {
!                       PerfdataValue::Ptr pdv;
!
!                       if (val.IsObjectType<PerfdataValue>())
!                               pdv = val;
!                       else {
!                               try {
!                                       pdv = PerfdataValue::Parse(val);
!                               } catch (const std::exception&) {
!                                       Log(LogWarning, "GraphiteWriter")
!                                               << "Ignoring invalid perfdata for checkable '"
!                                               << checkable->GetName() << "' and command '"
!                                               << checkCommand->GetName() << "' with value: " << val;
!                                       continue;
!                               }
                        }

!                       metrics.push_back({
!                               EscapeMetricLabel(pdv->GetLabel()),
!                               pdv->GetValue(),
!                               pdv->GetCrit(),
!                               pdv->GetWarn(),
!                               pdv->GetMin(),
!                               pdv->GetMax()
!                       });
!               }
!       } // Release ObjectLock before Network-I/O

!       try {
!               for (const auto& metric : metrics) {
!                       SendMetric(checkable, prefix, metric.escapedKey + ".value", metric.value, ts);

!                       if (GetEnableSendThresholds()) {
!                               if (!metric.crit.IsEmpty())
!                                       SendMetric(checkable, prefix, metric.escapedKey + ".crit", metric.crit, ts);
!                               if (!metric.warn.IsEmpty())
!                                       SendMetric(checkable, prefix, metric.escapedKey + ".warn", metric.warn, ts);
!                               if (!metric.min.IsEmpty())
!                                       SendMetric(checkable, prefix, metric.escapedKey + ".min", metric.min, ts);
!                               if (!metric.max.IsEmpty())
!                                       SendMetric(checkable, prefix, metric.escapedKey + ".max", metric.max, ts);
!                       }
                }
+       } catch (const std::exception&) {
+               // Send timed out or failed — remaining metrics for this check result are dropped.
+               // Details already logged by PerfdataWriterConnection.
        }
  }

***************

patch for b)

*** lib/perfdata/perfdatawriterconnection.hpp_save      2026-05-18 15:09:43.493587000 +0200
--- lib/perfdata/perfdatawriterconnection.hpp   2026-05-18 15:10:10.048545000 +0200
***************
*** 20,25 ****
--- 20,26 ----
  {
        static constexpr auto InitialRetryWait = 50ms;
        static constexpr auto FinalRetryWait = 32s;
+       static constexpr auto SendTimeout = 60s;

  public:
        DECLARE_PTR_TYPEDEFS(PerfdataWriterConnection);
***************
*** 67,72 ****
--- 68,74 ----

                using RetType = decltype(WriteMessage(std::declval<Buffer>(), std::declval<boost::asio::yield_context>()));
                std::promise<RetType> promise;
+               auto deadline = std::chrono::steady_clock::now() + SendTimeout;

                IoEngine::SpawnCoroutine(m_Strand, [&](boost::asio::yield_context yc) {
                        while (true) {
***************
*** 95,102 ****
                                        m_Stream = MakeStream();
                                        m_Connected = false;

                                        try {
!                                               BackoffWait(yc);
                                        } catch (const std::exception&) {
                                                promise.set_exception(std::make_exception_ptr(Stopped{}));
                                                return;
--- 97,117 ----
                                        m_Stream = MakeStream();
                                        m_Connected = false;

+                                       auto remaining = deadline - std::chrono::steady_clock::now();
+                                       if (remaining <= std::chrono::steady_clock::duration::zero()) {
+                                               Log(LogWarning, m_LogFacility)
+                                                       << "Send timeout exceeded for '" << m_ParentName << "', dropping data point.";
+                                               promise.set_exception(std::current_exception());
+                                               return;
+                                       }
+
                                        try {
!                                               m_ReconnectTimer.expires_after(std::min(m_RetryTimeout,
!                                                       std::chrono::duration_cast<std::chrono::milliseconds>(remaining)));
!                                               if (m_RetryTimeout <= FinalRetryWait / 2) {
!                                                       m_RetryTimeout *= 2;
!                                               }
!                                               m_ReconnectTimer.async_wait(yc);
                                        } catch (const std::exception&) {
                                                promise.set_exception(std::make_exception_ptr(Stopped{}));
                                                return;
***************

I've implemented the patches, built a new icinga image with the changed code and do not see those overdue issues any longer.

The final uncertanty is why graphitewriter is not able to actually send the data under certain circumstances. Graphite is reachable and i can manually send data from the icinga container to graphite container without any issue....

[julianbrost]

That's doesn't seem to be the only issue with the changes to the writers in 2.16.0, and this still needs quite a bit more investigation, so we've decided to revert the changes for 2.16.1 with #10851. This issue will remain open until it's also fixed in the master branch (which is the lower priority at the moment).

[jschmidt-icinga]

Note that the fix in #10857 does not implement a timeout on the connection, because it shouldn't be needed to fix the main issue reported here (checks going overdue). The work queue of Graphite will likely still rise until the root cause of #10825 is found and fixed. I'll look into if adding a timeout makes sense in the fix for that issue.