keep python nested list/array structure

[kaarrot]

I was wondering is there any way to retain structure of python nested lists.

In Python - a nested list/array is defined:
e = [[1,2,3],[9,8,[3,2,1]]]

The standard PyCall import output in Julia looks like this:
2x3 Array{Any,2}:
1 2 3
9 8 Any[3,2,1]

Often conversion to multidimensional arrays is very convenient but sometimes I need a bit more explicit control, especially when I want to keep things in order. I'm looking basically for a 1:1 representation:

[1,2,3],[9,8,Any[3,2,1]]

Is there any way I can instruct Julia/PyCall not to do the conversion into multidimensional arrays?
So far I can prevent it by sticking to pyobject type, but then I'm not sure what is the right PyCall's conversion function.

@pyimport mymodule
c = [] # which initilalzes 1-dim Array
PyCall.py2array(c,mymodule.pymember("e"))

Thanks

[stevengj]

Good question. Because multidimensional AbstractArrays in Julia are converted to Python lists-of-lists, I implemented the inverse transformation for going from Python to Julia. But maybe this was a bad default, and it certainly isn't always the right choice.

Once the PyBuffer support is implemented, so that contiguous arrays are decoupled from NumPy (#70), there is an even stronger impetus to leave lists-of-lists alone.

For now, it doesn't seem like there is an easy way to change this behavior, short of editing the PyCall source code: if you edit the pyarray_dims function (in conversions.jl) to return len after len = ccall((@pysym :PySequence_Size), Int, (PyPtr,), o), it should do what you want.

Maybe I should make this the default and only convert lists-of-lists to multidimensional arrays if you explicitly call py2array or something.

[kaarrot]

Thanks for a quick reply, I'll give it a go.
Maybe I should make this the default and only convert lists-of-lists to multidimensional arrays if you explicitly call py2array or something.
I would opt for "pythonic" representation as the default one to avoid confusion for PyCall newcomers as myself.

[kaarrot]

Ok, so modifying the function did the trick. I had to return (1,) though instead of len.

function pyarray_dims(o::PyObject, forcelist=true)
    if !(forcelist || pyisinstance(o, @pysym :PyList_Type))
        return () # too many non-List types can pretend to be sequences
    end
    len = ccall((@pysym :PySequence_Size), Int, (PyPtr,), o)
    return (1,)
end

Do you think if makes sense to have a global variable/constant to override this behaviour? Or is it better just to leave it as is for now until PyBuffer will be supported?

[stevengj]

I think a global would have too many potential side effects.

[stevengj]

Shouldn't it be return (len,)? Otherwise it thinks all arrays have length 1.

[kaarrot]

Ah, my bad yes you are right, otherwise I get just a PyObject.
Btw. Is there any way to restart PyCall after I make a change in the source? reload("PyCall") looks like is not working in this case and I need to restart Julia every time.
Also how would you reload python module loaded via @pyimport? Is there anything like @pyreload?
Thanks,

[mmiller-max]

Just run into this with my current project, is it likely that the fix will be released for this?

[PallHaraldsson]

@kubaroth, I was looking for PyReload.jl and found this issue, after looking at that other issue, and two comments that might be helpful for you: #775 (comment)