fix: prevent false-positive gap detection on zero-consumption periods#3554
Open
fix: prevent false-positive gap detection on zero-consumption periods#3554
Conversation
The gap detector in previous_days_modal_filter() checks if consecutive values are equal (data[m] == data[m+5]) to find missing data. After clean_incrementing_reverse(), zero-consumption overnight periods have equal consecutive values, triggering false gap detection and injecting phantom load (~6 kWh/night for a 24 kWh/day average). Track sensor data point provenance during minute_data() processing via a new data_point_minutes set parameter. In the gap detector, check whether the sensor was actively reporting during each gap period. If the sensor was online (≥1 data point/hour), skip filling. If offline, fill as before. Supersedes #3546 which attempted to fix the symptom via interpolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
previous_days_modal_filter()that injected phantom load (~6 kWh/night) during legitimate zero-consumption overnight periodsminute_data()processing via a newdata_point_minutessetRoot Cause
The gap detector checks
data[m] == data[m+5]to find missing data. Afterclean_incrementing_reverse(), zero-consumption periods have equal consecutive cumulative values. The detector falsely flags these as "gaps" and fills them withaverage_day / 1440kWh/minute of synthetic consumption.Test 10 proves this: 8 false gaps per 8 days, 3360 minutes (29.2% of data) incorrectly filled.
Approach: Data Point Provenance
Rather than changing the gap detection heuristic (which serves a legitimate purpose for sensor-offline periods), we record which minutes had actual sensor readings during
minute_data(). The gap filter then distinguishes:Files Changed
utils.pydata_point_minutesparam tominute_data()fetch.pyminute_data_load()+download_ge_data(), filter inprevious_days_modal_filter()tests/test_previous_days_modal.pytests/test_fill_load_from_power.pyTest plan
load_data_point_minutesattribute → backward compat (gaps filled as before)Supersedes #3546
🤖 Generated with Claude Code