Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments.

H/T: @kou, the documentation and #5685 for helping me get this off the ground.

Incorporate GCS file system into python and other bug fixes.

Bugs/Other changes:

• Add GCS bindings mostly based on AWS bindings in Python and associated unit tests
• Tell was incorrect, it double counted when the stream was constructed with an offset.
• Missed setting the define in config.cmake which means FileSystemFromUri was never tested and didn't compile this is now fixed
• Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well.
• Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support).
• Fix bug introduced in https://github.com/apache/arrow/commit/a5e45cecb24229433b825dac64e0ffd10d400e8c which caused failures when a deletion occurred on a bucket (not an object in the bucket).
• Ensure output streams are closed on destruction (this is consistent with S3)

This PR moves src/arrow/python directory into pyarrow and arranges PyArrow to build it. The build on the Python side is made in two steps:

1. _run_cmake_pyarrow_cpp() where the C++ part of the pyarrow is build first (the part that was moved in the refactoring)
2. _run_cmake() where pyarrow is built as before

No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with:

>>> pushd python/build/dist/temp 
>>> ctest

This PR switches our build system to require C++17 instead of C++11.

Because the conda packaging jobs are out of sync with the conda-forge files, the Windows conda packaging jobs are broken with this change. The related task (sync conda packaging files with conda-forge) is tracked in ARROW-17635.

This also updates pinned vcpkg to use xsimd 8.1.0.

This also implements auto python-wheel-windows-vs2017 image update mechanism. We have a problem of "docker build" on Windows. "docker build" doesn't reuse pulled image as caches. "docker build" always rebuilds an image. This implements manual reuse mechanism like the following:

if ! docker pull; then
   docker build # build only when built images don't exist
fi
docker run

But this doesn't work when ci/docker/python-wheel-windows-vs2017.dockerfile is updated but pinned vcpkg revision isn't changed. In the case, "docker build" isn't run because "docker pull" is succeeded.

To work this mechanism, this introduces "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION". We must bump it manually when we update ci/docker/python-wheel-windows-vs2017.dockerfile. "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" is used in tag name. So "docker pull" is failed with new "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" and then "docker build" is used.

adds python3.8 wheels

as far as I can tell python3.8 isn't available for Conda yet (https://github.com/conda-forge/python-feedstock/pull/274), so that's will have to be added later

Corresponds to status of feedstock as of https://github.com/conda-forge/arrow-cpp-feedstock/pull/848, minus obvious & intentional divergences in the setup here (with the exception of unpinning xsimd, which was pinned as of 9.0.0, but isn't anymore).

This PR uses "pkg-config --static ... arrow" to collect build flags. "pkg-config --static ... arrow" reports suitable build flags that depend on build options and used libraries for Apache Arrow C++. This works with the system AWS SDK C++.

PR for Scalar UDF integration

This is the first phase of UDF integration to Arrow. This version only includes ScalarFunctions. In future of PRs, Vector UDF (using Arrow VectorFunction), UDTF (user-defined table function) and Aggregation UDFs will be integrated. This PR includes the following;

• [x] UDF Python Scalar Function registration and usage
• [x] UDF Python Scalar Function Examples
• [x] UDF Python Scalar Function test cases
• [x] UDF C++ Example extended from Compute Function Example
• [x] Added aggregation example (optional to this PR: if required can remove and push in a different PR)

macOS development target is changed to 10.13 from 10.11. See also the discussion on mailing list: https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w

This PR adds support for google-cloud-cpp to the Conda files. Probably the most difficult change to grok is the change to compile with C++17 when using Conda:

• Conda defaults all its builds to C++17, this bug goes into some detail as to why.
• Arrow defaults to C++11 if no CMAKE_CXX_STANDARD argument is provided.
• Abseil's ABI changes when used from C++11 vs. C++17, see https://github.com/abseil/abseil-cpp/issues/696
• Therefore, one must compile with C++17 to use Abseil in Conda.
• And because google-cloud-cpp has a direct dependency on Abseil, exposed through the headers, one must use C++17 to use google-cloud-cpp too.

This PR fixes the issue of handling NaNs in the Parquet predicate push-down. While computing the valid bounds for a column, if the max or min of the column is null, the range should ignore that.

This patch trying to fix https://github.com/apache/arrow/issues/15052 . The problem is mentioned here: https://github.com/apache/arrow/issues/15052#issuecomment-1367486164

When read 1 value, DeltaBitPackDecoder will not call InitBlock, causing it always read last_value_.

Seems the problem is introduced in https://github.com/apache/arrow/pull/10627 and https://github.com/amol-/arrow/commit/d982bedcf5e03d44c01949b192da54a8c1e525d8

I will add some test tonight

• Closes: #15052

Describe the enhancement requested

We updated version information for documents manually in 10.0.1 release: #14887

We should automate it to reduce release cost. See also https://github.com/apache/arrow/pull/14887#issuecomment-1347332418 for implementation idea.

Component(s)

R, Release

Describe the bug, including details regarding any error messages, version, and platform.

see https://github.com/apache/arrow/issues/14829#issuecomment-1367506270

Component(s)

Continuous Integration, R