Daskify stats#6
Conversation
Dask abstract lazy cubedata
Lazy pp loading
* Swap out Biggus concatenate for Dask concatenate * Re-enable passing concatenate tests
* fixed typo lazy_array to lazy_data; give fill_value attribute to all lazy data
* Changed most occurrences of biggus.Array instance checks to either dask.array.Array instance checks or lazy data checks * updated headers
* header dates corrected whitespace removed
dask based merge
| elif isinstance(other, iris.cube.Cube): | ||
| try: | ||
| da.broadcast_to(cube.lazy_data(), other.lazy_data()) | ||
| da.broadcast_to(cube.lazy_data(), other.lazy_data().shape) |
There was a problem hiding this comment.
This should probably use other.shape rather than other.lazy_data().shape
| other = iris.util.as_compatible_shape(other, cube).lazy_data() | ||
| else: | ||
| other = other.lazy_data() | ||
| if cube.has_lazy_data() and not in_place: |
There was a problem hiding this comment.
I dislike having to check for the existence of lazy / non lazy data in multiple places, but in the current implementation .data is used in at least two different places. This if check makes sure that the 'other' data stored in the unary_func is in the same format as as the cube data. I'd much prefer to check once for laziness then explicitly fork the logic.
| @@ -120,29 +120,25 @@ def _assert_is_cube(cube): | |||
| def _assert_compatible(cube, other): | |||
There was a problem hiding this comment.
Does this function even do anything? It's used in one place, right after a try/except block around 'da.broadcast_to'. So it seems like any broadcast errors would already have been thrown.
There was a problem hiding this comment.
Although, seeing as da.broadcast_to is only used to validate that the cubes are compatible da.core.broadcast_shapes is potentially a more efficient method as it seems to do less work.
|
Some of the test failures are now only caused because they expect the resultant cube to have realized data (e.g. in the results cml). It would be simple to just swap the test results to expect the state to be deferred, but these tests are still helpful as they show that we aren't accidentally converting an already realized cube back into a lazy one (by creating a new cube from the source cubes .lazy_data()). |
|
(iris._lazy_data._MAX_CHUNK_SIZE reduced to 100) |
djkirkham
force-pushed
the
daskify_stats
branch
from
1115177 to
c0631f9
Compare
5 participants
This is a working implementation of lazy arithmetic (addition at least). The tests still fail for various reasons