GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer

[Reranko05]

What changes are included?

This PR introduces a reusable JsonWriter wrapper around simdjson's JSON builder API and migrates the integration JSON writer to use it instead of RapidJSON.

Specifically, this PR:

• Adds a reusable JsonWriter abstraction in arrow/json.
• Migrates the integration JSON writer implementation to JsonWriter.
• Updates the integration tests to use JsonWriter.
• Adds unit tests for JsonWriter.

This is part of the incremental migration from RapidJSON to simdjson.

Performance

I compared the serialization performance of IntegrationJsonWriter::WriteRecordBatch() + Finish() using a temporary benchmark with a 1M-row RecordBatch. The benchmark was run 10 times on both main and this branch.

Branch	Average Time
main	2157.4 ms
This PR	2153.6 ms

No measurable performance regression was observed.

Are these changes tested?

Yes.

I added unit tests for JsonWriter and verified that both JSON and integration tests pass locally:

• arrow-json-test
• arrow-json-integration-test

Are there any user-facing changes?

No.

• GitHub Issue: [C++] Introduce JsonWriter and migrate integration JSON writer #50567

[github-actions]

⚠️ GitHub issue #50567 has been automatically assigned in GitHub to PR creator.

[Reranko05]

@kou could you review this PR when you have a chance?

[kou]

Could you check performance difference?

[Reranko05]

@kou, I compared the serialization performance of IntegrationJsonWriter::WriteRecordBatch() + Finish() using a temporary benchmark with a 1M-row RecordBatch. I ran the benchmark 10 times on both main and this branch. After excluding one obvious outlier from each run, the averages were:

main: 2157.4 ms
this branch: 2153.6 ms

I didn't observe any measurable performance regression.

[Reranko05]

@kou I investigated the failing workflows and they appear to be unrelated to this PR. Could you please verify?

[rok]

Thanks for working on this @Reranko05! Exciting to see this work!

We should probably add meson support for json_writer, see (cpp/src/arrow/meson.build, cpp/src/arrow/json/meson.build, cpp/src/arrow/integration/meson.build).

[rok]

Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?

[Copilot]

Pull request overview

This PR adds a reusable arrow::json::JsonWriter wrapper around simdjson’s builder API and migrates the C++ integration JSON writer (and its tests) away from RapidJSON’s writer, as part of the incremental RapidJSON → simdjson migration.

Changes:

• Added arrow::json::JsonWriter (header/impl) plus unit tests.
• Migrated integration JSON serialization code paths to emit JSON using JsonWriter.
• Updated CMake/test linkage and CI environment to ensure simdjson is available/linked where needed.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cpp/src/arrow/json/json_writer.h	Introduces the JsonWriter public API.
cpp/src/arrow/json/json_writer.cc	Implements JsonWriter on top of simdjson builder primitives.
cpp/src/arrow/json/json_writer_test.cc	Adds unit coverage for basic writer operations.
cpp/src/arrow/json/CMakeLists.txt	Adds the new unit test and links simdjson for the test target.
cpp/src/arrow/integration/json_internal.h	Updates writer function signatures to accept arrow::json::JsonWriter*.
cpp/src/arrow/integration/json_internal.cc	Migrates integration JSON emission to JsonWriter (writer-side).
cpp/src/arrow/integration/json_integration.cc	Switches IntegrationJsonWriter implementation to JsonWriter.
cpp/src/arrow/integration/json_integration_test.cc	Updates integration tests to use JsonWriter for schema/array JSON generation.
cpp/src/arrow/integration/CMakeLists.txt	Links simdjson where integration tests/executables now depend on it.
cpp/src/arrow/CMakeLists.txt	Adds json/json_writer.cc to Arrow JSON sources and links simdjson for integration targets.
ci/docker/ubuntu-24.04-cpp.dockerfile	Sets simdjson_SOURCE=BUNDLED to ensure simdjson availability in CI images.

[Reranko05]

Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?

I wasn't able to find any existing Conbench C++ microbenchmarks covering JSON parsing or writing. I did run a local benchmark comparing IntegrationJsonWriter::WriteRecordBatch() + Finish() on this branch against main and didn't observe any measurable regression. If I missed an existing Conbench benchmark, please let me know and I will run this against it.

[Reranko05]

@kou It was just a temporary benchmark. I used the following to measure IntegrationJsonWriter::WriteRecordBatch() + Finish() on a 1M-row RecordBatch:

auto schema = ::arrow::schema({
    field("f1", boolean()),
    field("f2", int8()),
    field("f3", int32()),
    field("f4", uint64()),
    field("f5", utf8())
});

std::vector<std::shared_ptr<Array>> arrays;
const int num_rows = 1000000;
MakeBatchArrays(schema, num_rows, &arrays);

auto batch = RecordBatch::Make(schema, num_rows, arrays);

ASSERT_OK_AND_ASSIGN(auto writer, IntegrationJsonWriter::Open(schema));

auto start = std::chrono::steady_clock::now();

ASSERT_OK(writer->WriteRecordBatch(*batch));
ASSERT_OK_AND_ASSIGN(auto json_data, writer->Finish());

auto end = std::chrono::steady_clock::now();

std::cout << "Serialization time: "
          << std::chrono::duration_cast<std::chrono::milliseconds>(
                 end - start)
                 .count()
          << " ms\n";

I ran it 10 times on both main and this branch, discarded one obvious outlier from each set, and compared the average serialization time.

[Reranko05]

We can require simdjson 4.0 or later. If system simdjson is old, we can fallback to bundled simdjson.

@rok Does this approach work for you as well?

[rok]

We can require simdjson 4.0 or later. If system simdjson is old, we can fallback to bundled simdjson.

@rok Does this approach work for you as well?

Yes, this sounds good!

[kou]

I used the following to measure IntegrationJsonWriter::WriteRecordBatch() + Finish() on a 1M-row RecordBatch:

Thanks. I also tried it.

simdjson: 69 ms
RapidJSON: 65 ms

Is RapidJSON fast enough? Is simdjson not fast?

Can we try naive std::ostringstream implementation something like the following?

namespace arrow::json {
class ARROW_EXPORT JsonWriter {
  // ...
 private:
  // ...
  std::ostringstream output_;
  // ...
};
}

namespace arrow::json {
void JsonWriter::StartObject() {
  MaybeComma();
  output_ << "{";
  needs_comma_ = false;
}
// ...

If this performance is similar to simdjson/RapidJSON, they may not be so fast for writing.
If this is slower than simdjson/RapidJSON, both of them are fast.

[Reranko05]

@kou I have a question regarding the benchmark results. Earlier, when I benchmarked IntegrationJsonWriter::WriteRecordBatch() + Finish() on both the RapidJSON and simdjson implementations using a 1M-row RecordBatch, I measured around 2.1 s for both (in a debug build). After replacing the implementation with a naive std::ostringstream, I also measured around the same order of magnitude. You mentioned seeing about 65 ms for RapidJSON and 69 ms for simdjson. Were those measurements taken in a different setup?

[kou]

I used -DCMAKE_BUILD_TYPE=Release via --preset=ninja-release-maximal.

I think that we need to use a release build to measure. I expect that simdjson uses SIMD for writing. It'll require a release build.

[Reranko05]

@kou I reran the benchmark in a Release build using the same temporary benchmark (IntegrationJsonWriter::WriteRecordBatch() + Finish() on a 1M-row RecordBatch).

Results:

• main (RapidJSON): 324 ms
• This PR (simdjson): 490 ms
• This PR with a naive std::ostringstream implementation: 1323 ms

So, in this benchmark, the std::ostringstream implementation is about 2.7× slower than the current implementation and about 4× slower than the RapidJSON implementation. My absolute timings are still higher than the ~65–69 ms you reported, so I suspect my benchmark setup differs from yours.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Reranko05]

Marking this as ready for review. The remaining CI failures appear to be unrelated to this change (dependency download, environment, or infrastructure issues).