`abstractarray` test sometimes fails with max copying

[udesou]

At first glance, It does not seem related to any previous issue. Note that I cannot reproduce this consistently.

Test Failed at /home/eduardo/julia/test/abstractarray.jl:1257
  Expression: pointer(Av, i) == pointer(Sv, i) == pointer(Ss, i)
   Evaluated: Ptr{Int64}(0x0000020110f62f28) == Ptr{Int64}(0x0000020103b57cf8) == Ptr{Int64}(0x0000020103b57cf8)

edit: may be related to #249, see https://github.com/mmtk/julia/blob/1a472bbe2d4ced580586a2f60f0d9a70071ccc90/test/abstractarray.jl#L1236

[qinsoon]

I think this is simply an FFI issue. The idea of this test is that you can take multiple views into the same array memory. The test uses a lot of pointer() to get the underlying array memory from different views, and assert that they are pointing to the same memory.

However, in an assertion like pointer(Av, i) == pointer(Sv, i) == pointer(Ss, i), GCs can happen between each pointer call, and make the pointer returned as invalid. For example, in the fail from the original post, we see

  Expression: pointer(Av, i) == pointer(Sv, i) == pointer(Ss, i)
   Evaluated: Ptr{Int64}(0x0000020110f62f28) == Ptr{Int64}(0x0000020103b57cf8) == Ptr{Int64}(0x0000020103b57cf8)

The value pointer(Av, i) was computed, and then a GC happened after pointer(Av, i), and moved the underlying memory. Thus the following two calls to pointer() gets the new memory address, and were not the same as the first call.

If we add GC.gc() between the pointer calls, we can even see all the 3 pointer values are different.

        pav = pointer(Av, i)
        GC.gc()
        psv = pointer(Sv, i)
        GC.gc()
        pss = pointer(Ss, i)
        @test pav == psv == pss

Simple 3d strided views and permutes: Test Failed at /home/yilin/Code/julia_workspace/julia/test_abstractarray.jl:51
  Expression: pav == psv == pss
   Evaluated: Ptr{Int64}(0x0000020100f7c178) == Ptr{Int64}(0x0000020100bf6178) == Ptr{Int64}(0x00000201007f8178)

Other than pointer assertions, there are other assertions about the values. If we remove the pointer assertions, the value assertions actually pass.

[qinsoon]

We discussed this with Julia folks, and confirmed that the objects should be pinned before taking a pointer from them. The remaining question is that why those pointers are not pinned implicitly by conservative stack scanning. The pointers should appear in the stack or in the registers, and should be pinned implicitly -- but they don't get pinned. I am still curious about this, and investigating it.

[qinsoon]

The interesting thing is that Ptr{T} is a heap object that holds the pointer. So the Ptr{T} object appears on the stack, and gets pinned. But the object it refers to does not appear on the stack, so it can be moved by the GC.

A minimal example

function my_test(x::Array)
    p = pointer(a)
    ccall(:jl_breakpoint, Cvoid, (Any,), p)
    GC.gc()
    p2 = pointer(a)
    ccall(:jl_breakpoint, Cvoid, (Any,), p2)
    println(p)
    println(p2)
end

We can see p and p2 get printed out as below. The pointer values are the actual addresses of the array (which gets moved by the GC).

Ptr{Int64}(0x00000200ffc50e10)
Ptr{Int64}(0x00000200ffc2c020)

But in the two breakpoints above, we see

Thread 1 hit Breakpoint 1, jl_breakpoint (v=0x200ffc2e400) at /home/yilin/Code/julia_workspace/julia/src/rtutils.c:1586
Thread 1 hit Breakpoint 1, jl_breakpoint (v=0x20100373fd0) at /home/yilin/Code/julia_workspace/julia/src/rtutils.c:1586

0x200ffc2e400 and 0x20100373fd0 are the pointer objects, the pointers to the array are stored in them.

[qinsoon]

pointer() for arrays return the buffer address. If we pin an array object, we will still see pointer() returns different addresses across GCs, as the underlying buffers can still get moved by the GC.

Arrays may not be the only case. Theoratically, it would be any type whose pointer() returns a different address other than the address of the object itself.

[udesou]

Fixed by mmtk/julia#97.