Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces centralized global configuration (Settings) and QC whitelisting (WhiteSet/SensorWhiteSet), refactors plotting + xarray conversions to consume these settings, and replaces several assert/sys.exit paths with toolkit exceptions while bumping the package to 1.0.0a1.
Changes:
- Add
Settingssingleton + plotting defaults and migrate label definitions/plot styling to settings-driven lookups. - Add QC whitelist support and thread it through multiple QC checks; add new datetime utilities for consistent timezone handling.
- Update docs, CI/workflows, and pickle-solution test artifacts to align with the new APIs/version.
Reviewed changes
Copilot reviewed 109 out of 457 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/pkled_solutions/test_importing_solutions/testdemodata/test_subset_by_stations/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_importing_solutions/testdemodata/test_import_demo_data/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_importing_solutions/testdemodata/test_get_station/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_importing_solutions/testdemodata/test_get_info/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_weighted_diurnal_debias_modeldata_gapfill/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_weighted_diurnal_debias_gf_on_station/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_raw_modeldata_gapfill/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_partially_filled_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_interpolation_on_station/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_interpolation_on_dataset/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_interpolation_chaining_on_dataset/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_import_data/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/station_with_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/station_no_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/station_filled/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/sensordata_with_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/sensordata_no_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/sensordata_filled/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/dataset_with_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/dataset_no_gaps/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_gap_status_df/dataset_filled/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_diurnal_debias_modeldata_gapfill/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_debias_modeldata_gapfill/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gf_solutions/testdatawithgaps/test_chaining_gapfill_methods/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_landcover_frac_extraction/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_import_demo_metadata/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_altitude_extraction/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_LCZ_extraction/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_ERA5_extraction_on_metadata_only/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_gee_solutions/testdemodataset/test_ERA5_extraction/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_subsetting_time/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_import_data/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_if_analysis_can_be_created_from_station/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_filtering/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_basic_analysis_method_calls/datatype.json | Add datatype metadata for pickled solution artifact |
| tests/pkled_solutions/test_analysis_solutions/testdemodata/test_aggregate_df_method/datatype.json | Add datatype metadata for pickled solution artifact |
| src/metobs_toolkit/xrconversions.py | Switch to Settings-driven label/timezone serialization + type-check-only imports |
| src/metobs_toolkit/timestampmatcher.py | Replace asserts with toolkit exceptions + move timedelta helper import |
| src/metobs_toolkit/template_build_prompt.py | Update Obstype API usage + replace sys.exit with MetObsTemplateError |
| src/metobs_toolkit/template.py | Replace sys.exit/assert with MetObsTemplateError + decorator import |
| src/metobs_toolkit/site.py | Safer extradata default + rename args + Settings-driven decorator import |
| src/metobs_toolkit/settings_collection/version.py | Bump package version to 1.0.0a1 |
| src/metobs_toolkit/settings_collection/settings.py | Add Settings singleton with nested get/set/reset and plotting/label helpers |
| src/metobs_toolkit/settings_collection/plotting_defaults.py | Centralize plotting defaults for Settings |
| src/metobs_toolkit/settings_collection/label_defenitions.py | Restructure label definitions incl. plot metadata + rename buddy-check label |
| src/metobs_toolkit/settings_collection/init.py | Export Settings and version |
| src/metobs_toolkit/sensordata.py | Standardize timezone handling, Settings label lookups, and richer outlier details |
| src/metobs_toolkit/qc_collection/window_variation_check.py | Add whitelist support + datetime helper usage |
| src/metobs_toolkit/qc_collection/whitelist.py | Add WhiteSet/SensorWhiteSet implementation |
| src/metobs_toolkit/qc_collection/step_check.py | Add whitelist support |
| src/metobs_toolkit/qc_collection/repetitions_check.py | Add whitelist support + datetime helper usage |
| src/metobs_toolkit/qc_collection/persistence_check.py | Add whitelist support + datetime helper usage |
| src/metobs_toolkit/qc_collection/grossvalue_check.py | Add whitelist support |
| src/metobs_toolkit/qc_collection/distancematrix_func.py | Decorator import refactor |
| src/metobs_toolkit/qc_collection/common_functions.py | Decorator import refactor |
| src/metobs_toolkit/plot_collection/timeseries_plotting.py | Label/plot styling now driven by Settings + stricter internal validation |
| src/metobs_toolkit/plot_collection/qc_info_pies.py | Pie plotting uses Settings-driven styling + create_axes helper |
| src/metobs_toolkit/plot_collection/pd_basic_plotting.py | Decorator import refactor |
| src/metobs_toolkit/plot_collection/general_functions.py | Settings-driven colormap + legend API generalized |
| src/metobs_toolkit/plot_collection/folium_spatial.py | Replace sys.exit with exception |
| src/metobs_toolkit/plot_collection/default_style.py | Remove legacy plotting defaults module |
| src/metobs_toolkit/plot_collection/cycle_plotting.py | Settings-driven plotting defaults + type-check-only imports |
| src/metobs_toolkit/plot_collection/init.py | Stop exporting removed default_plot_settings |
| src/metobs_toolkit/obstypes.py | Rename constructor arg to name + typing cleanup |
| src/metobs_toolkit/modeltimeseries.py | Standardize timestamp timezone handling + Settings-driven plot defaults |
| src/metobs_toolkit/io_collection/metadataparser.py | Decorator import refactor + explicit None returns |
| src/metobs_toolkit/io_collection/filewriters.py | Add fmt_output_filepath helper + Path typing |
| src/metobs_toolkit/io_collection/filereaders.py | Decorator import refactor |
| src/metobs_toolkit/io_collection/dataparser.py | Decorator import refactor |
| src/metobs_toolkit/gf_collection/gf_common_methods.py | Decorator import refactor |
| src/metobs_toolkit/gf_collection/diurnal_debias_gapfill.py | Decorator import refactor |
| src/metobs_toolkit/gf_collection/debias_gapfill.py | Decorator import refactor |
| src/metobs_toolkit/geometry_functions.py | Decorator import refactor |
| src/metobs_toolkit/geedatasetmanagers.py | Use fmt_output_filepath + datetime arg normalization + signature updates |
| src/metobs_toolkit/gee_api.py | Decorator import refactor + improved debug logging |
| src/metobs_toolkit/gap.py | Settings-driven labels + typing improvements |
| src/metobs_toolkit/backend_collection/uniqueness.py | Decorator import refactor |
| src/metobs_toolkit/backend_collection/printing_collection.py | Pull print formatting defaults from Settings |
| src/metobs_toolkit/backend_collection/loggingmodule.py | Use Settings defaults + fmt_output_filepath; move decorator elsewhere |
| src/metobs_toolkit/backend_collection/filter_modeldatadf.py | Rename arg to obstype + decorator import refactor |
| src/metobs_toolkit/backend_collection/errorclasses.py | Add MetObsInternalError + decorator import refactor |
| src/metobs_toolkit/backend_collection/df_helpers.py | Remove to_timedelta (moved) + decorator import refactor |
| src/metobs_toolkit/backend_collection/decorators.py | Add shared decorators module (log_entry) |
| src/metobs_toolkit/backend_collection/datetime_collection.py | Add timestamp + timezone conversion utilities + to_timedelta |
| src/metobs_toolkit/backend_collection/datetime_agg_collection.py | Decorator import refactor |
| src/metobs_toolkit/backend_collection/dataframe_constructors.py | Settings-driven label lookup + decorator import refactor |
| src/metobs_toolkit/backend_collection/argumentcheckers.py | Use to_timedelta helper + warn on tz-naive datetime args |
| src/metobs_toolkit/analysis.py | Rename args to obstype + use Settings-driven plot defaults |
| src/metobs_toolkit/init.py | Export WhiteSet and Settings at top-level |
| pytest.ini | Configure pytest-mpl baseline path |
| pyproject.toml | Version/classifier updates + pyarrow constraint + poetry packages config |
| docs/topics/template_doc.ipynb | Update rendered outputs for new repr/order |
| docs/topics/metadata_only.ipynb | Update rendered outputs for metadata fields |
| docs/topics/log_example.ipynb | Update logging example + API change for add_FileHandler |
| docs/topics/index.rst | Add new topics (whitesets/settings) + title casing |
| docs/topics/changing_settings.ipynb | Add new Settings usage tutorial notebook |
| docs/reference/whitelist.rst | Add WhiteSet API docs |
| docs/reference/settings.rst | Add Settings API docs |
| docs/reference/index.rst | Link WhiteSet and Settings in API index |
| docs/reference/dataset.rst | Update QC method name reference (safetynets) |
| docs/examples/obstype_example.ipynb | Update API usage (Obstype name=...) and rendered outputs |
| deployment/test_gee_auth.py | Minor output cleanup |
| deployment/develop_pipeline.sh | Adjust install/black/pytest invocation paths |
| .github/workflows/main_workflow.yml | Add new notebooks + include new tests in matrix |
| .github/copilot-instructions.md | Sync documentation/examples with renamed API + decorator module |
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+114
to
+115
| @classmethod | ||
| def get(cls, key: str, default: Any = None) -> Any: |
Comment on lines
+141
to
+152
| # Ensure initialized | ||
| cls() | ||
|
|
||
| # Handle dot notation for nested keys | ||
| keys = key.split(".") | ||
| value = cls._config | ||
|
|
||
| try: | ||
| for k in keys: | ||
| value = value[k] | ||
| logger.debug(f"Settings: Retrieved: {key} --> {value}") | ||
| return value |
| plotdf.loc[plotdf["label"].isin(all_outlier_labels), "value"] = np.nan | ||
| plotdf.loc[plotdf["label"].isin(all_outlier_labels()), "value"] = np.nan | ||
|
|
||
| linekwargs = Settings.get("plotting_settings.time_series.linekwargs", {}) |
Comment on lines
+339
to
+340
| # update linekwargs | ||
| linekwargs.update({"linestyle": linestyle, "color": colormap[staname]}) |
| default_kwargs = {"figsize": default_style["figsize"]} | ||
| default_kwargs.update(figkwargs) # overwrite default | ||
| ax = plotting.create_axes(**default_kwargs) | ||
| allfigkwargs = Settings.get("plotting_settings.cycle_plot.figkwargs", {}) |
Comment on lines
+82
to
+83
| # Class variable for internal timezone storage | ||
| _target_tz: str = Settings.get("store_tz") |
Comment on lines
13
to
18
| @@ -12,6 +14,7 @@ def gross_value_check( | |||
| records: pd.Series, | |||
| lower_threshold: Union[int, float], | |||
| upper_threshold: Union[int, float], | |||
| sensorwhiteset: SensorWhiteSet, | |||
| ) -> pd.DatetimeIndex: | |||
| Threshold below which records are flagged as outliers. | ||
| upper_threshold : int or float | ||
| Threshold above which records are flagged as outliers. | ||
| sensorwhiteset : SensorWhiteSet |
Comment on lines
177
to
187
| @name.setter | ||
| @log_entry | ||
| def name(self, value: str): | ||
| def name(self, value: str) -> str: | ||
| """Set the name of the observation type.""" | ||
| self._name = str(value) | ||
|
|
||
| @description.setter | ||
| @log_entry | ||
| def description(self, value: str): | ||
| def description(self, value: str) -> str: | ||
| """Set the description of the observation type.""" | ||
| self._description = str(value) |
| ) -> xr.Dataset: | ||
| """ | ||
| Convert a model time series object to an xarray Dataset. | ||
| Convert a Modelimeseries object to an xarray.Dataset. |
* Update ModelTimeSeries repr and change modeldata return type to list * Gap filling fix (#589) * fix bug and add tests baselines (#571) * fix bug and add tests baselines * fix location of baseline * black edits * handle comma as decimal symbol when importing data (#576) * Add convert_to_numeric_series function and integrate into dataset and sensordata modules * black edits * fix review * Error make plot model name (#579) * added modeldata_name variable to make_plot() of stations class. * add modeldata_kwargs to make_plot in order to select a specific modeldata series * add plot test * black edits * new parameter modeltype adds functionality to plot different type of modeldata compared to obstype, if nothing is specified modeltype=obstype * trigger * resolve empty dicts * tests added * tests adapted (no fake argument anymore) * remove check for dataset if modeltype exists * arg renaming + argsorder change * black edits --------- Co-authored-by: Thomas Vergauwen <thomas.vergauwen@meteo.be> * Fix buddy check bug (#583) * add reference of the iteration to the detailed msg * fix duplicates by joining messages * add a test to trigger this edgecase * black edits * Update tests/test_qc.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * grammer errors in comments --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * develop alpha version bump * first attempt max_gap_duration_to_fill implementation * changed > to >= * rechanged >= to > * print add * print * gap is filled or not filled , not partly * small error fix * errorfix * implementation of intuitive gap overview on all levels called 'singular_gaps() ( datset, station, sensor) + deal with edgecase of partially filled gaps by including a new option in def fillstatus _partially_successful_label = _partially_successful_label = "partially successful gapfill" currently this will cause the gap to be refilled totally (also were the gap filling was succesfull) when a new gapfilling is called. we can extend this later to only filling the 'gap in the gap' * version set to prerelease * Refactor gap handling: remove singular_gaps property and add gap_status_overview_df method for improved gap status reporting * make the gapfill status flagger more human readable * Refactor gap handling: replace singular_gaps property with gap_status_overview_df method for improved gap status reporting * Refactor gap handling: remove singular_gaps property and implement gap_status_overview_df method for enhanced gap reporting; update max_gap_duration_to_fill default to 12 hours * Update max_gap_duration_to_fill default to 12 hours and improve gap size validation checks * minor code style refactoring * black edits * fix identation bug * code refactoring of gaps overview in sensordata * gap_overview_dfs in seperate module * add the new method to the API docs * update and add tests * Update the example * rename function to gap_overview_df * fix bug * black edits * serialize timedelta and timestamp attr in xr conv * fix unicode error when exporting to netcdf * fix tests * by default overwrite the partially GF * update tests * black edits * black edits --------- Co-authored-by: Thomas Vergauwen <82087298+vergauwenthomas@users.noreply.github.com> Co-authored-by: Thomas Vergauwen <thomas.vergauwen@meteo.be> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix filter to modeldata bug (#592) * develop version bump * fix the filtering to modeldata dataframe * use helper func for extracting modeltimeseries * test different plotting methods of modeldata on different levels * add log_entry * black edits * fix the xr engine problem when saving as netcdf * simple pd plot functionality (#595) * functionality of pd_plot * move the sensordata pd plot to seperate module * fix warnings * implementation for modeldata pd_plot * add the pd_plot methods to the api doc * use id for sensordata label in pd_plot * add tests * prerelease version bump * black edits * code review fixes * fix docstring identation * black edits * check for duplicates in loghandlers before adding * Distance matric uses BallTree with haversine (for scaling) * debug version bump * Add min_value and max_value parameters to gap-filling methods to prevent unphysical data (#598) * Initial plan * Add min_value and max_value parameters to core gap-fill functions Co-authored-by: ADRIE-A3 <109740197+ADRIE-A3@users.noreply.github.com> * Add tests for min_value and max_value gap-fill parameters Co-authored-by: ADRIE-A3 <109740197+ADRIE-A3@users.noreply.github.com> * Add documentation and examples for gap-fill value constraints Co-authored-by: ADRIE-A3 <109740197+ADRIE-A3@users.noreply.github.com> * Add implementation summary for gap-fill value constraints Co-authored-by: ADRIE-A3 <109740197+ADRIE-A3@users.noreply.github.com> * remove unwanted files * Remove min_value and max_value parameters from Gap and SensorData classes * pass minvalue maxvalue args for GF methods * Add min_value and max_value parameters to Dataset gap-filling methods * remove unwanted files * minor version bump * add test * black edits * Gee credits local pipeline (#599) * black edits * bugfix that data._credentials does not exist in ee v1.6.12. Now with a test catch condition on ee.initialize * add gee credential testing and checking for the credential file in the (local) testing pipeline --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ADRIE-A3 <109740197+ADRIE-A3@users.noreply.github.com> Co-authored-by: Thomas Vergauwen <thomas.vergauwen@meteo.be> Co-authored-by: Thomas Vergauwen <82087298+vergauwenthomas@users.noreply.github.com> * Geemap fix (#601) * add comment * open logs in default text browser * close figures after comparison to fix the failing tests * black edit * Update pyproject.toml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add clipping in docstring schematic description * Update src/metobs_toolkit/sensordata.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_gf.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_plotting.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fixes for codereview * black edit * version bump * trigger WF * remove all references to max_consec_fill * Priority groups (#602) * Add catch_white_records function and update repetitions_check to exclude white records from outlier detection * implement white records for repetitions check on all levels. * whitelist for persistence * Update white_records parameter type to Union[pd.Index, None] in Dataset and Station classes * white records for gross value check * add white_records to step and window variation check * implement whitelist argument for buddy checks * pass white records for repetitions check * update to use the sensorwhiteset argument * update to using the sensorwhiteset argument * implement the sensorwhiteset argument * refactor: replace white_records with whiteset in QC methods and related functions * cleanup the module * add tests * black formatting * add WhiteSet to api docs * add important box * fix for timezone bug * add topic and refer in the qc example to the topic on WhiteSets * comment protector * black edits * remove default value for sensorwhiteset in persistence_check, repetitions_check, and step_check functions * refactor SensorWhiteSet initialization to use None as default for white_timestamps * remove default value for sensorwhiteset in window_variation_check function * Update tests/test_qc.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor save_whitelist_records function to improve documentation and remove unused code * add logging decorator to get_info method in WhiteSet class * remove unused import of SensorWhiteSet from whitelist module * add a string formatting for in the xr attributes * black edits * refer to the topic section * sync docstrings * fix futurwarning * dev version bump --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Buddy check with multiple safetynets (#603) * Add catch_white_records function and update repetitions_check to exclude white records from outlier detection * implement white records for repetitions check on all levels. * whitelist for persistence * Update white_records parameter type to Union[pd.Index, None] in Dataset and Station classes * white records for gross value check * add white_records to step and window variation check * implement whitelist argument for buddy checks * pass white records for repetitions check * update to use the sensorwhiteset argument * update to using the sensorwhiteset argument * implement the sensorwhiteset argument * refactor: replace white_records with whiteset in QC methods and related functions * cleanup the module * add tests * black formatting * add WhiteSet to api docs * add important box * fix for timezone bug * add topic and refer in the qc example to the topic on WhiteSets * comment protector * black edits * remove default value for sensorwhiteset in persistence_check, repetitions_check, and step_check functions * refactor SensorWhiteSet initialization to use None as default for white_timestamps * remove default value for sensorwhiteset in window_variation_check function * Update tests/test_qc.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor save_whitelist_records function to improve documentation and remove unused code * add logging decorator to get_info method in WhiteSet class * remove unused import of SensorWhiteSet from whitelist module * add a string formatting for in the xr attributes * black edits * refer to the topic section * sync docstrings * fix futurwarning * dev version bump * adding a 'details' column to the outliersdf * implement the buddy with safetynets (generalisation) * black edits * check the safety net argument * exclude 'details' form comparison * fix tests * update the examples * update solutions * fix: correct formatting of log messages and documentation comments * minor version bump * Update src/metobs_toolkit/qc_collection/buddy_check.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: deprecate buddy_check_with_LCZ_safety_net method in Dataset class * rename safety_nets to safetynets * black edits * drop explicit attributs in whitelist * fix test issue * fix link to whitelist topic page * drop duplicated attributes in docstring * fix bool and list attributes of QC * fix tests * black edits --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Argument name and format consistency (#606) * rename all occurences of target_obstype to obstype * revert function name * rename trgobstype to obstype * rename target_obstype to obstype * rename obstype name to name * fix tests * rename arguments to more easy to read version of gee_*_dataset * implement a formatter for output files * filepath arugment consistency * add initialize_gee argument where missing and consitent use of update arguments * pass modelname and modelvariable arugments to dataset GF methods * minor version bump * test examples * simpler naming for gee_manager * filepath handling for logging to file * black edits * store and check version compatibility * fix circular import with lazy import * fix tests * fix test * black edits * running tests * code review fixes * introduce a seperate module for datetime transformations * fix and add log tests * Imported classes for typehinting are now imported via TYPE_CHECKING * remove info file * add debug logging for CRS conversion failure in _validate_metadf * refactor error handling to raise custom exceptions instead of using sys.exit * refactor error handling to raise KeyError for unmapped labels instead of using sys.exit * refactor: update timezone handling to use TYPE_CHECKING imports and improve extradata initialization * refactor: improve outlier message formatting and enhance template file validation * refactor: update SensorData type hints to remove type ignore comments * refactor: format error message for empty timeseries in GEEDynamicDatasetManager * refactor: clean up imports and improve timezone error handling in SensorData * docs: add version compatibility section and deprecation warning guidance for method signature changes * refactor: update return value for known observation type check in Analysis class * refactor: standardize max_gap_duration_to_fill initialization in Dataset class * refactor: standardize return values to None for gap handling methods in Gap class * code refactoring * add future import * Configclass (#607) * refactor: update type hints for modeltimeseries, sensordata, station, dataset, and QC attributes * implementation of the Settings class (test have still a fail) * bugfix: keep tz when adding it to mulitindex * add plot_as and marker for the step outliers * black edits * spell checking * cleanup * hardcode utc fixes * code review * run notebook * fix whitelist test * black edits * drop old solutions * tests are passing * minor version bump * version and dev status bump * drop info file * running notebooks in current version * figure default settings in Settings class (#608) * colorby plotkwargs * fix typo * move log_entry to new location * use Settings for plot settings and fix move of log_entry * move log_entry * add plot defaults in settings * refactor plotting functions to use updated Settings structure for kwargs * typhinting fix * figures in new figsize * change default figsize * figure results with new default styles * black edits * rerun and fix notebooks * bad location of baseline * fix typehint warning * remove unused import of functools in loggingmodule.py * remove unused imports from cycle_plotting.py * better description on the use of the Settings class * increase the default size of scatters in plots + add plottest with outliers * better description of Settings * use underscore for non-publick methods in Settings * black edits * testfix * include pyarrow 22.x.x * duplicate timestamp in sperate module * invalid check in seperate module * regular checks now use the QCresult class per sensor * Refactor the buddy check over multiple modules and catch logs using a wrapped Station class * Future proof test pipeline (#610) * new futur proof structure of saving and comparing solutions in testpipeline * new solution structure applied to the test_qc * use dicts as solution implemented * new solutions format for gf test * put plotting test in gf tests + use test dependencies * update testgee solutions * update geetest solutions * startpoint solutions for plot tests * fix solutions * clear old solution * update test importing to new framework * update test logging * fix test to work with new solutionsclass * update obstype test and solution * fix analysis test * fix test * fix figure location * cleaner code refactoring * make buddy check deterministic * black edits * run test from repo root * minor version bump * avoid cached package use when running testpipeline * rerun with tight layout * fix location of baseline * minor version bump * fix qc_stats * labels defined in settings * bugfix when empyt outliers * fix test * fix issue on get_qc_stats with empty outliersdf * bugfix for unmet condition error * colorbugfix * bugfix when no whitelist is used * drop unused method * drop unused variable * add get_qc in test, for sanity checking * add test with buddy and mf=true * fix the processing of safetynet details + creation of a final deatail string * rename min_sample_spread and use the z_robust bool arg + robust z score * implement the MP version of buddy check * fix the number of cpus by settings for MP qc checks * min_std deprecation warning * update some tests * fix and add a buddy test * buddy check test solutions * without storing values in qcresult * rename update * qc_overview_df for sensordata * qc_overview_df for station and dataset level * choose to use basic or detailed labels for outliersdf * bugfix when saftynet is triggerd in previous iteration, but not in current * put the qc detail df constructors in a seperate module * qc overview construction in seperate module * from buddywrapstation to buddywrapsensor * drop the MP executor locals from the qcresult so make it serializable again * replace index for details by using the flags index * add some logs * minor version bump * log full wrapstation on error (debugging) * Potential bug fixes * improve memory and load distribution for multiprocessing * minor version bump * add a todo * drop debug lines * bugfix, return checksettings * minor version bump * hardcode the checksettings, there was a memory leak and unwanted data with locals(). * fix memory leak issue and to mutch info in qcchecksettings with locals() * remove complex joining functions since parallelization is per station * debug version bump * add the max N buddies functionallity+ minor version bump * details bugfix * DEBUG, return wrapsensors for analysis * debug version bump * bump because of invalid version format * add only_if_previous_had_no_buddies functionallity for buddy saftynets * implement the minimum buddy radius functionallity * fix the ee bug * Sync the dependencies with master * put qc checks in seperate folder * docstrings * get rid of unused force argument * rename all references to buddywrappedstation to buddywrapedsensor * add some small qc tests on edge cases and the outliersdetailsdf * fix minor code review remarks * drop unused ci pipeline * fix issue of remapping timestamps of outliers and duplicated records * simple code format fixes * update docstrings * updating outliers now can take the same check into account (qc chaining) * seperate (buddy check tests) and extra test + a new dataset for buddy checks * add distancematrix to as method to Dataset * update docstring * fix example * fix and extend the qc example * bugfix in the QCresult to string formatting * fix filling example * version bump * fix examples * black edits * include details in the solution * remove warning in docstring * fix the raised warning * cleanup docstring * cleanup docstrings * cleanup docstrings * replace pd.concat to save_concat * Bugfix in syncronization of timeseries in buddy check * minor code review fixes * add sanity checks (in __main__ protectors) to testing framework (CI/CD, pytest and develop-pipeline) * convert sanity checks to series of assert checks * update docstrings * explicit add scikit-learn (for balltree method) * black edits * excluse the metobs package when pip audit in CI/CD pipeline * fix black vulnerability (CVE-2026-32274) --------- Co-authored-by: Leon Adriaensen <109740197+ADRIE-A3@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.