Skip to content

GH-41990: [C++] Fix AzureFileSystem compilation on Windows #48971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nateprewitt wants to merge 8 commits into
base: main
Choose a base branch
from
+35 −4
Open

GH-41990: [C++] Fix AzureFileSystem compilation on Windows #48971

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cac716c
Fix MSVC build for Azure
nateprewitt Jan 22, 2026
50d2511
Add build tooling for Azure SDK
nateprewitt Jan 23, 2026
1f9b988
Update cpp/src/arrow/filesystem/azurefs.cc
nateprewitt Jan 26, 2026
2ba5b3d
Update cpp/cmake_modules/ThirdpartyToolchain.cmake
nateprewitt Jan 26, 2026
a81ac55
Use OVERRIDE_FIND_PACKAGE to address Azure SDK WIL requirement
nateprewitt Jan 26, 2026
3f5cea2
Alphabetize variables and add them to DEPENDENCIES
nateprewitt Jan 26, 2026
7754f6d
Remove unneeded comments and use = default for destructor
nateprewitt Jan 26, 2026
c50396c
Change kMaxBlockSizeBytes to Long Long for cross platform compatibility
nateprewitt Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/cpp_windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,14 @@ jobs:
runs-on: ${{ inputs.os }}
timeout-minutes: 60
env:
ARROW_AZURE: ON
ARROW_BOOST_USE_SHARED: OFF
ARROW_BUILD_BENCHMARKS: ON
ARROW_BUILD_SHARED: ON
ARROW_BUILD_STATIC: OFF
ARROW_BUILD_TESTS: ON
ARROW_DATASET: ON
ARROW_FILESYSTEM: ON
ARROW_FLIGHT: OFF
ARROW_HDFS: ON
ARROW_HOME: /usr
Expand Down
22 changes: 22 additions & 0 deletions cpp/cmake_modules/ThirdpartyToolchain.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,13 @@ else()
)
endif()

if(DEFINED ENV{ARROW_WIL_URL})
set(ARROW_WIL_URL "$ENV{ARROW_WIL_URL}")
else()
set_urls(ARROW_WIL_URL
"https://github.com/microsoft/wil/archive/${ARROW_WIL_BUILD_VERSION}.tar.gz")
endif()

if(DEFINED ENV{ARROW_BOOST_URL})
set(BOOST_SOURCE_URL "$ENV{ARROW_BOOST_URL}")
else()
Expand Down Expand Up @@ -4054,6 +4061,21 @@ endif()

function(build_azure_sdk)
message(STATUS "Building Azure SDK for C++ from source")

# On Windows, Azure SDK's WinHTTP transport requires WIL (Windows Implementation Libraries).
# Fetch WIL before Azure SDK so the WIL::WIL target is available.
if(WIN32)
message(STATUS "Fetching WIL (Windows Implementation Libraries) for Azure SDK")
fetchcontent_declare(wil
${FC_DECLARE_COMMON_OPTIONS} OVERRIDE_FIND_PACKAGE
URL ${ARROW_WIL_URL}
URL_HASH "SHA256=${ARROW_WIL_BUILD_SHA256_CHECKSUM}")
prepare_fetchcontent()
set(WIL_BUILD_PACKAGING OFF)
set(WIL_BUILD_TESTS OFF)
fetchcontent_makeavailable(wil)
endif()

fetchcontent_declare(azure_sdk
${FC_DECLARE_COMMON_OPTIONS}
URL ${ARROW_AZURE_SDK_URL}
Expand Down
8 changes: 5 additions & 3 deletions cpp/src/arrow/filesystem/azurefs.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -970,7 +970,7 @@ Status StageBlock(Blobs::BlockBlobClient* block_blob_client, const std::string&
/// Writes will be buffered up to this size (in bytes) before actually uploading them.
static constexpr int64_t kBlockUploadSizeBytes = 10 * 1024 * 1024;
/// The maximum size of a block in Azure Blob (as per docs).
static constexpr int64_t kMaxBlockSizeBytes = 4UL * 1024 * 1024 * 1024;
static constexpr int64_t kMaxBlockSizeBytes = 4LL * 1024 * 1024 * 1024;
Copy link
Contributor

@h-vetinari h-vetinari Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normally I wouldn't consider long long "good for portability", but since the result here is implicitly cast to int64_t, that should be fine 👍


/// This output stream, similar to other arrow OutputStreams, is not thread-safe.
class ObjectAppendStream final : public io::OutputStream {
Expand Down Expand Up @@ -1362,7 +1362,7 @@ Result<HNSSupport> CheckIfHierarchicalNamespaceIsEnabled(
// without hierarchical namespace enabled.
directory_client.GetAccessControlList();
return HNSSupport::kEnabled;
} catch (std::out_of_range& exception) {
} catch (const std::out_of_range&) {
// Azurite issue detected.
DCHECK(IsDfsEmulator(options));
return HNSSupport::kDisabled;
Expand Down Expand Up @@ -2499,7 +2499,7 @@ class AzureFileSystem::Impl {
try {
auto delete_result = deferred_response.GetResponse();
success = delete_result.Value.Deleted;
} catch (const Core::RequestFailedException& exception) {
} catch (const Core::RequestFailedException&) {
success = false;
}
if (!success) {
Expand Down Expand Up @@ -3218,6 +3218,8 @@ class AzureFileSystem::Impl {
std::atomic<LeaseGuard::SteadyClock::time_point> LeaseGuard::latest_known_expiry_time_ =
SteadyClock::time_point{SteadyClock::duration::zero()};

AzureFileSystem::~AzureFileSystem() = default;

AzureFileSystem::AzureFileSystem(std::unique_ptr<Impl>&& impl)
: FileSystem(impl->io_context()), impl_(std::move(impl)) {
default_async_is_sync_ = false;
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/filesystem/azurefs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ class ARROW_EXPORT AzureFileSystem : public FileSystem {
void ForceCachedHierarchicalNamespaceSupport(int hns_support);

public:
~AzureFileSystem() override = default;
~AzureFileSystem() override;

static Result<std::shared_ptr<AzureFileSystem>> Make(
const AzureOptions& options, const io::IOContext& = io::default_io_context());
Expand Down
5 changes: 5 additions & 0 deletions cpp/thirdparty/versions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,9 @@ ARROW_THRIFT_BUILD_VERSION=0.22.0
ARROW_THRIFT_BUILD_SHA256_CHECKSUM=794a0e455787960d9f27ab92c38e34da27e8deeda7a5db0e59dc64a00df8a1e5
ARROW_UTF8PROC_BUILD_VERSION=v2.10.0
ARROW_UTF8PROC_BUILD_SHA256_CHECKSUM=6f4f1b639daa6dca9f80bc5db1233e9cbaa31a67790887106160b33ef743f136
# WIL (Windows Implementation Libraries) is required by Azure SDK on Windows for WinHTTP transport
ARROW_WIL_BUILD_VERSION=v1.0.250325.1
ARROW_WIL_BUILD_SHA256_CHECKSUM=c9e667d5f86ded43d17b5669d243e95ca7b437e3a167c170805ffd4aa8a9a786
ARROW_XSIMD_BUILD_VERSION=14.0.0
ARROW_XSIMD_BUILD_SHA256_CHECKSUM=17de0236954955c10c09d6938d4c5f3a3b92d31be5dadd1d5d09fc1b15490dce
ARROW_ZLIB_BUILD_VERSION=1.3.1
Expand Down Expand Up @@ -142,6 +145,7 @@ DEPENDENCIES=(
"ARROW_AWS_CRT_CPP_URL aws-crt-cpp-${ARROW_AWS_CRT_CPP_BUILD_VERSION}.tar.gz https://github.com/awslabs/aws-crt-cpp/archive/${ARROW_AWS_CRT_CPP_BUILD_VERSION}.tar.gz"
"ARROW_AWS_LC_URL aws-lc-${ARROW_AWS_LC_BUILD_VERSION}.tar.gz https://github.com/awslabs/aws-lc/archive/${ARROW_AWS_LC_BUILD_VERSION}.tar.gz"
"ARROW_AWSSDK_URL aws-sdk-cpp-${ARROW_AWSSDK_BUILD_VERSION}.tar.gz https://github.com/aws/aws-sdk-cpp/archive/${ARROW_AWSSDK_BUILD_VERSION}.tar.gz"
"ARROW_AZURE_SDK_URL azure-sdk-for-cpp-${ARROW_AZURE_SDK_BUILD_VERSION}.tar.gz https://github.com/Azure/azure-sdk-for-cpp/archive/${ARROW_AZURE_SDK_BUILD_VERSION}.tar.gz"
"ARROW_BOOST_URL boost-${ARROW_BOOST_BUILD_VERSION}-cmake.tar.gz https://github.com/boostorg/boost/releases/download/boost-${ARROW_BOOST_BUILD_VERSION}/boost-${ARROW_BOOST_BUILD_VERSION}-cmake.tar.gz"
"ARROW_BROTLI_URL brotli-${ARROW_BROTLI_BUILD_VERSION}.tar.gz https://github.com/google/brotli/archive/${ARROW_BROTLI_BUILD_VERSION}.tar.gz"
"ARROW_BZIP2_URL bzip2-${ARROW_BZIP2_BUILD_VERSION}.tar.gz https://sourceware.org/pub/bzip2/bzip2-${ARROW_BZIP2_BUILD_VERSION}.tar.gz"
Expand All @@ -168,6 +172,7 @@ DEPENDENCIES=(
"ARROW_SUBSTRAIT_URL substrait-${ARROW_SUBSTRAIT_BUILD_VERSION}.tar.gz https://github.com/substrait-io/substrait/archive/${ARROW_SUBSTRAIT_BUILD_VERSION}.tar.gz"
"ARROW_THRIFT_URL thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz https://www.apache.org/dyn/closer.lua/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz?action=download"
"ARROW_UTF8PROC_URL utf8proc-${ARROW_UTF8PROC_BUILD_VERSION}.tar.gz https://github.com/JuliaStrings/utf8proc/archive/${ARROW_UTF8PROC_BUILD_VERSION}.tar.gz"
"ARROW_WIL_URL wil-${ARROW_WIL_BUILD_VERSION}.tar.gz https://github.com/microsoft/wil/archive/refs/tags/${ARROW_WIL_BUILD_VERSION}.tar.gz"
"ARROW_XSIMD_URL xsimd-${ARROW_XSIMD_BUILD_VERSION}.tar.gz https://github.com/xtensor-stack/xsimd/archive/${ARROW_XSIMD_BUILD_VERSION}.tar.gz"
"ARROW_ZLIB_URL zlib-${ARROW_ZLIB_BUILD_VERSION}.tar.gz https://zlib.net/fossils/zlib-${ARROW_ZLIB_BUILD_VERSION}.tar.gz"
"ARROW_ZSTD_URL zstd-${ARROW_ZSTD_BUILD_VERSION}.tar.gz https://github.com/facebook/zstd/releases/download/v${ARROW_ZSTD_BUILD_VERSION}/zstd-${ARROW_ZSTD_BUILD_VERSION}.tar.gz"
Expand Down
Loading