Skip to content

fix: prevent false-positive gap detection on zero-consumption periods#3554

Open
mgazza wants to merge 1 commit intomainfrom
fix/false-positive-gap-detection
Open

fix: prevent false-positive gap detection on zero-consumption periods#3554
mgazza wants to merge 1 commit intomainfrom
fix/false-positive-gap-detection

Conversation

@mgazza
Copy link
Collaborator

@mgazza mgazza commented Mar 11, 2026

Summary

  • Fixes false-positive gap detection in previous_days_modal_filter() that injected phantom load (~6 kWh/night) during legitimate zero-consumption overnight periods
  • Tracks sensor data point provenance during minute_data() processing via a new data_point_minutes set
  • In the gap detector, checks whether the sensor was actively reporting during each gap. If online (≥1 data point/hour), skips filling. If offline, fills as before.
  • Fully backward-compatible: existing callers unaffected, no attribute = old behavior

Root Cause

The gap detector checks data[m] == data[m+5] to find missing data. After clean_incrementing_reverse(), zero-consumption periods have equal consecutive cumulative values. The detector falsely flags these as "gaps" and fills them with average_day / 1440 kWh/minute of synthetic consumption.

Test 10 proves this: 8 false gaps per 8 days, 3360 minutes (29.2% of data) incorrectly filled.

Approach: Data Point Provenance

Rather than changing the gap detection heuristic (which serves a legitimate purpose for sensor-offline periods), we record which minutes had actual sensor readings during minute_data(). The gap filter then distinguishes:

  • Sensor online, zero consumption → skip filling (correct)
  • Sensor offline, no data → fill with average (correct, existing behavior)

Files Changed

File Change
utils.py Add data_point_minutes param to minute_data()
fetch.py Init set, pass through minute_data_load() + download_ge_data(), filter in previous_days_modal_filter()
tests/test_previous_days_modal.py Tests 6-8: sensor active → skip, sensor offline → fill, backward compat
tests/test_fill_load_from_power.py Test 11: e2e verification all 8 false-positive gaps filtered out

Test plan

  • All existing tests pass (previous_days_modal, fill_load_from_power, minute_data, minute_data_load, get_now_cumulative)
  • Test 6: Overnight zeros with sensor data points → gaps skipped (12 kWh stays 12 kWh)
  • Test 7: Gap with no sensor data points → gap correctly filled (12 → 18 kWh)
  • Test 8: No load_data_point_minutes attribute → backward compat (gaps filled as before)
  • Test 11: End-to-end proof all 8 false-positive overnight gaps filtered out

Supersedes #3546

🤖 Generated with Claude Code

The gap detector in previous_days_modal_filter() checks if consecutive
values are equal (data[m] == data[m+5]) to find missing data. After
clean_incrementing_reverse(), zero-consumption overnight periods have
equal consecutive values, triggering false gap detection and injecting
phantom load (~6 kWh/night for a 24 kWh/day average).

Track sensor data point provenance during minute_data() processing via
a new data_point_minutes set parameter. In the gap detector, check
whether the sensor was actively reporting during each gap period. If
the sensor was online (≥1 data point/hour), skip filling. If offline,
fill as before.

Supersedes #3546 which attempted to fix the symptom via interpolation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant