GH-49644: [Python] Support converting list of multi-dimensional arrays to FixedShapeTensor

[aboderinsamuel]

Rationale for this change

Constructing a fixed-shape-tensor array from a list of individual ndarrays only
worked when each element was 1-D; ≥2-D elements failed with
ArrowInvalid: Can only convert 1-dimensional array values. The only workaround
was stacking the list into a single ndarray and using
FixedShapeTensorArray.from_numpy_ndarray.

What changes are included in this PR?

The C++ list converter PyListConverter::AppendNdarray now accepts
multi-dimensional ndarray elements for fixed-size lists (the storage of a
fixed-shape tensor) by flattening them in C order. The fixed-size-list builder
still validates that the flattened length matches the list width, so wrong sizes
error cleanly. Variable-sized lists remain restricted to 1-D values to avoid
ambiguity. As a side benefit, plain fixed_size_list also accepts
multi-dimensional ndarray elements now.

Are these changes tested?

Yes:

• test_tensor_array_from_list_of_ndarrays — construction from 2-D and 3-D
  ndarrays, null handling, storage parity with from_numpy_ndarray, and the
  size-mismatch error, across int8/int64/float32.
• test_fixed_size_list_from_multidim_ndarray — plain fixed_size_list from
  multi-dim arrays, plus a check that variable-sized lists still reject 2-D.

Are there any user-facing changes?

Yes — pa.array([multi-dim ndarrays], type=fixed_shape_tensor(...)) (and the
same for fixed_size_list) now works instead of raising. Existing 1-D behavior
and variable-sized-list behavior are unchanged.

Scoped to construction only; the reverse to_numpy shape-preservation also
raised in the issue is intentionally left as a separate follow-up.

• GitHub Issue: [Python] Support converting list of multi-dimensional arrays to FixedShapeTensor #49644

[github-actions]

⚠️ GitHub issue #49644 has been automatically assigned in GitHub to PR creator.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for constructing fixed-shape tensors and fixed-size lists from lists of multi-dimensional NumPy ndarrays by flattening values in C order (GH-49644).

Changes:

• Add tests covering tensor arrays built from lists of ndarrays (including nulls and shape mismatch).
• Add tests ensuring fixed-size lists accept multi-dimensional ndarray elements (and reject invalid cases).
• Update ndarray-to-list conversion to allow flattening for FIXED_SIZE_LIST while keeping variable-sized lists restricted to 1D.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
python/pyarrow/tests/test_extension_type.py	Adds coverage for building FixedShapeTensorArray from a list of ndarrays.
python/pyarrow/tests/test_array.py	Adds a regression test for fixed-size list conversion from multi-dimensional ndarrays.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Implements multi-dimensional ndarray flattening for fixed-size lists during conversion.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[AlenkaF]

Thank you @aboderinsamuel!
Can you have a look at the linter error here: https://github.com/apache/arrow/actions/runs/27719407632/job/82001639136?pr=50203#step:5:98

Also, am curious how you decided to use PyArray_Ravel? Are there any other alternatives?

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[aboderinsamuel]

Hi @AlenkaF The macOS-intel failure is an intermittent dependency-install flake. Homebrew lock on aws-sdk-cpp, before the build/tests even run. It's hitting other PRs too (e.g. #50146), and #50195 is actively moving the macOS AWS-SDK dependency, so it looks like known infra rather than my change. Happy to keep re-running until it catches a clean runner. Everything else is green. @rok 🙂

[AlenkaF]

Repeating a question :)

[AlenkaF]

Why did you decided to use PyArray_Ravel? Are there any other alternatives?

[aboderinsamuel]

PyArray_Ravel(ndarray, NPY_CORDER) returns a 1-D, C-order view, which lets me reuse the existing 1-D append paths unchanged (the typed fast paths and the PySequence fallback). It's zero-copy when the input is already C-contiguous and only copies for non-contiguous/Fortran-order input, where a copy is unavoidable to get C-order data anyway, which also matches the fixed-shape-tensor's row-major storage.

Alternatives: PyArray_Flatten(NPY_CORDER) gives the same result but always copies; a manual NpyIter/buffer loop could avoid even the view allocation but would mean duplicating the typed-append fast paths. Ravel was the smallest change with the best reuse/performance balance.

[rok]

PyArray_Ravel over PyArray_Flatten seems ok. If I read the context correctly we're using a builder to append these arrays. When builder finishes it creates a new buffer with a copy of concatenated arrays. So PyArray_Ravel should be ok if the list of arrays is not deleted before appending is complete, which I expect won't happen. @AlenkaF am I reading this right?

[AlenkaF]

Could you have a look at PyArray_CheckFromAny with flag NPY_ARRAY_C_CONTIGUOUS? If I understand correctly it reshapes if necessary or gives access to the data/view directly. Then we can use PyArray_DATA as in other places in the codebase.

I am not sure which would fit better but am interested in what you think.

[rok]

Also to note here - PyArray_Ravel seems to default to C order. So we should document somewhere that no matter the input arrays order we will return C order. (@AlenkaF please doublecheck me here :) )

[AlenkaF]

Great, I think having more control can be helpful. Thanks for looking into it!
Also, some other flags might be needed, see: https://numpy.org/doc/stable/reference/c-api/array.html#basic-array-flags

[aboderinsamuel]

Thank you 🙏, i'm currently taking a look at the reference

[aboderinsamuel]

Thanks for the pointer to the flags doc, @AlenkaF, really useful read 🙂

Done: switched to PyArray_CheckFromAny with NPY_ARRAY_C_CONTIGUOUS | NPY_ARRAY_ALIGNED, then view it as 1-D and read in C order. The ALIGNED flag was the piece I'd have missed on my own, so thanks for steering me there! For byte order I kept the explicit ISBYTESWAPPED check (moved up top per @rok) instead of NPY_ARRAY_NOTSWAPPED, and left out WRITEABLE since we only read here.

Really appreciate you and @rok taking the time on this, learning a ton from it.

[aboderinsamuel]

Also, the comment now notes the array is always flattened in C order regardless of the input's layout. @rok

[rok]

@AlenkaF I think this is ok now. Do you agree?

[github-actions]

⚠️ GitHub issue #49644 has been automatically assigned in GitHub to PR creator.

[tadeja]

Hi all. I'd add two points for further review of current PR changes (thanks @aboderinsamuel !)
Example of wrong shape and example of permuted tensor currently produce results but might need to return errors instead:

import numpy as np
import pytest
import pyarrow as pa

np_dtype = np.dtype("int8")
tensor_type = pa.fixed_shape_tensor(pa.from_numpy_dtype(np_dtype), (2, 3))
pa.array([np.arange(6, dtype=np_dtype).reshape(3, 2)], type=tensor_type)
# As (3, 2) has the right number of elements (6), but the wrong shape for a (2, 3) tensor,
# shouldn't we get an error?

Currently returns:

<pyarrow.lib.FixedShapeTensorArray object at 0x12524ec20>
[
  [
    0,
    1,
    2,
    3,
    4,
    5
  ]
]

np_dtype2 = np.dtype("float32")
elements = [
  np.arange(6, dtype=np_dtype2).reshape(2, 3),
  np.arange(6, 12, dtype=np_dtype2).reshape(2, 3),
]
permuted_type = pa.fixed_shape_tensor(pa.from_numpy_dtype(np_dtype2), (2, 3),
                                    permutation=[1, 0])
pa.array(elements, type=permuted_type)
# For permuted tensor types built from multi-dim ndarrays, doesn't this store the wrong layout?

Currently returns:

<pyarrow.lib.FixedShapeTensorArray object at 0x104597ac0>
[
  [
    0,
    1,
    2,
    3,
    4,
    5
  ],
  [
    6,
    7,
    8,
    9,
    10,
    11
  ]
]

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[rok]

Some comments about comments. I think the one important question left is the PyArray_Ravel question.

[aboderinsamuel]

@rok I'd keep PyArray_Ravel here. It's a zero-copy reshape of the already-C-contiguous array, and it gives the 1-D array that the dtype-mismatch fallback (value_converter_->Extend, just below) needs to iterate element-wise. Reading PyArray_DATA directly works for the typed fast path, but the fallback would still need a 1-D handle either way, so the ravel keeps a single shared path with no extra copy. Happy to switch if you'd prefer, but I'd lean to keeping it as-is.

[rok]

@aboderinsamuel - did you mean to push changes?

[aboderinsamuel]

yes, @rok. in a few minutes

[rok]

This looks good and I think we're ready to merge. Will just wait for @AlenkaF to chime in on the PyArray_Ravel comment.

[rok]

Maybe we could add an example of using the new constructor here? e.g.:

Suggested change

	Create an extension array from a list of multi-dimensional NumPy arrays.
	Each element is flattened in row-major (C) order, its shape must match
	the tensor shape.
	>>> import numpy as np
	>>> pa.array([np.array([[1, 2], [3, 4]], dtype=np.int32),
	... np.array([[10, 20], [30, 40]], dtype=np.int32)],
	... type=tensor_type)
	<pyarrow.lib.FixedShapeTensorArray object at ...>
	[
	[
	1,
	2,
	3,
	4
	],
	[
	10,
	20,
	30,
	40
	]
	]
	"""