-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathChangeLog
More file actions
154 lines (135 loc) · 7.6 KB
/
ChangeLog
File metadata and controls
154 lines (135 loc) · 7.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
----------------------------------------------------------------------------
DTLMod (0.5) not released yet (target: May 2026)
Improvements:
- Memory efficiency: progressive eviction of metadata transaction entries
- Once all subscribers have consumed a transaction, its entries are evicted
from the in-memory Metadata::transaction_infos_ map
- When metadata export is enabled and publishers and subscribers coexist
(file streaming), evicted entries are progressively flushed to
per-variable temporary files; the final metadata file is assembled at
pub_close() from those files and any remaining in-memory entries,
preserving the existing format and transaction count
- When the stream is opened by subscribers only after all publishers have
closed (sequential scenario), memory-only eviction is performed (the
metadata file has already been written by pub_close())
- Memory footprint of the File engine now grows as O(N_pub) instead of
O(N_pub × N_transactions) for long-running concurrent streaming workloads
-
API Changes:
- New Engine method:
- Engine::cancel_transaction() cancel the currently ongoing transaction
performed by this engine. This method has to be called by an external
actor not involved in the transaction. It will make the transaction raise
a DTLMod::TransactionCancelledException that must be caught if you want
the publishers and subscribers to survive to the cancellation of this
transaction.
- New Stream helper methods:
- Stream::get_engine_type() and Stream::get_transport_method() respectively
return the enum value of the engine type and transport method for a
Stream.
- As a consequence, in the Python bindings, the Stream.engine_type and
Stream.transport_method readonly properties that convert the engine type
and transport method to a string have been renamed to
Stream.engine_type_str and Stream.transport_method_str while the existing
properties now return the enum values.
- Stream::get_reduction_method(name) to retrieve a reduction method
associated to the stream.
- Behavior modification:
- DTL::add_stream can now take an Engine Type and a Transport Method as
parameters (both defaulting to 'Undefined' if not specified).
----------------------------------------------------------------------------
DTLMod (0.4) February 16, 2026
Major improvements:
- Data Reduction framework
- New abstract ReductionMethod interface for extensible data reduction strategies
- Decimation method: spatial subsampling with per-dimension stride, optional
interpolation (linear, quadratic, cubic), configurable computational cost,
and support for both publisher-side and subscriber-side application
- Compression method: size reduction with preserved shape, supporting three
compressor profiles (fixed ratio, SZ-inspired, and ZFP-inspired models),
separate compression and decompression costs, and per-transaction ratio
variability
- Publisher-side reduction state is propagated to subscribers through
inquire_variable, enabling detection and prevention of conflicting
double reductions
- Reduction operations work with both File and Staging engines
- New documentation pages for the reduction feature (Reduction, Decimation,
and Compression) and updated Main Concepts page
- Improved test coverage
- Comprehensive C++ and Python tests for both reduction methods
- Coverage of error handling, parameter validation, re-parameterization,
and publisher-subscriber workflows for each reduction method
- New test for subscriber-first arrival pattern in File engine
- Code quality and CI improvements
- Improved coverage reporting with SonarQube and CodeFactor integration
- Added subscriber synchronization barrier in File engine
- Removed unnecessary defensive guards and race condition checks that
cannot occur under SimGrid's maestro orchestration
API Changes:
- New Stream method:
- Stream::define_reduction_method(name) creates a named reduction method
("decimation" or "compression") for the stream
- New Variable methods and properties:
- Variable::set_reduction_operation(method, parameters) applies a reduction
with key-value parameters to a variable
- Variable::is_reduced(), is_reduced_by_publisher(), is_reduced_by_subscriber()
query the reduction state
- Variable::get_reduction_method() retrieves the applied ReductionMethod
- New ReductionMethod class with query methods:
- get_reduced_variable_global_size(), get_reduced_variable_local_size()
- get_reduced_variable_shape()
- get_flop_amount_to_reduce_variable(), get_flop_amount_to_decompress_variable()
- Full Python bindings for all reduction operations, including nine new
exception types for parameter validation errors
- Engine::put() now automatically simulates reduction cost and transports the
reduced data size when the variable is reduced
- Engine::get() now automatically simulates decompression cost after receiving
compressed data
----------------------------------------------------------------------------
DTLMod (0.3) January 19, 2026
Major improvements:
- Enhanced code quality with comprehensive refactoring
- Improved modern C++17 usage throughout the codebase
- Reduced code complexity and technical debt
- Memory safety and robustness improvements
- Fixed critical memory safety issue in DTL connection management
- Fixed race condition in Engine creation
- Plugged memory leaks
- Major refactoring with improved memory management, encapsulation, and move-only semantics
- Improved CI/CD infrastructure
- Added Valgrind and sanitizer checks (AddressSanitizer, UndefinedBehaviorSanitizer)
- Enhanced test coverage and validation
- Comprehensive proofread of the documentation
API Changes:
- DTL::get_stream_by_name_or_null() renamed to DTL::get_stream_by_name()
Returns std::optional<std::shared_ptr<Stream>> instead of raw pointer
Python: dtl.get_stream_by_name() now returns None if stream not found
- Metadata management transferred from Engine to Stream
Stream::get_metadata_file_name() replaces Engine::get_metadata_file_name()
Python: stream.metadata_file_name replaces engine.metadata_file_name
- Stream method chaining improvements
set_engine_type() and set_transport_method() now return Stream& instead of Stream*
Enables cleaner fluent interface: stream.set_engine_type(...).set_transport_method(...)
- DTL::create() now accepts std::string_view and has empty default parameter
- String parameters changed to std::string_view for better performance:
- DTL::add_stream(), DTL::get_stream_by_name()
- Stream and Variable constructors and methods
- get_engine_type_str() and get_transport_method_str() now return std::optional<const char*>
----------------------------------------------------------------------------
DTLMod (0.2) November 11, 2025
- New comprehensive documentation
- Full python bindings
- Improved code coverage
- Reduced technical debt
- Fixed deadlocks in File engine
- Update to SimGrid 4.1 and FSMod 0.4
- API:
- Make metadata export configurable at stream level.
- Have a simpler version of Engine::put() that automatically uses the
local size of the Variable.
- Stream::remove_variable now throws an UnknownVariableException rather
than returning a bool
----------------------------------------------------------------------------
DTLMod (0.1) July 4, 2025
First Release
----------------------------------------------------------------------------