fix: preserve 'NA' string values in CSV parsing instead of converting to None

[devin-ai-integration]

Summary

ResponseToFileExtractor._read_with_chunks uses pd.read_csv() which by default interprets strings like 'NA', 'N/A', 'NaN', 'null' as NaN values (keep_default_na=True). This causes legitimate field values — e.g. 'NA' meaning "North America" in a Salesforce picklist — to be silently converted to None/NULL in synced data.

Fix: pass keep_default_na=False so pandas treats all cell values as literal strings (since we already set dtype=object). Empty CSV fields (which pandas would previously auto-tag as NaN) are now explicitly converted to None via .replace({"": None}) to maintain backward-compatible behavior.

- pd.read_csv(data, chunksize=chunk_size, iterator=True, dialect="unix", dtype=object)
+ pd.read_csv(data, chunksize=chunk_size, iterator=True, dialect="unix", dtype=object, keep_default_na=False)

- chunk.replace({nan: None})
+ chunk.replace({nan: None, "": None})

Link to Devin session: https://app.devin.ai/sessions/59e21b90c6c34df7bcae4bddc1947b44

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1782155104-fix-na-string-preservation#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1782155104-fix-na-string-preservation

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 121 tests  +2   4 110 ✅ +2   8m 14s ⏱️ -2s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit edc81bf. ± Comparison against base commit 977f051.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 124 tests  +2   4 112 ✅ +2   12m 0s ⏱️ +24s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit edc81bf. ± Comparison against base commit 977f051.

♻️ This comment has been updated with latest results.

[Anatolii Yatsuk (tolik0)]

Reviewed the fix — it is correct and well-targeted. One suggestion on the tests:

The two added tests only exercise "NA", but keep_default_na=True was silently corrupting all 19 of pandas' default NA tokens — N/A, NULL, None, nan, null, #N/A, <NA>, etc. Those other tokens are arguably the under-reported part of this bug (this customer only happened to notice "NA").

I verified the fix against pandas 2.2.3 and it preserves all of them:

Cell value	Before fix	After fix
NA, N/A, NULL, None, nan	→ None (corrupted)	preserved as string ✓
empty unquoted (,)	None	None ✓
empty quoted ("")	None	None ✓
whitespace (" ")	" "	" " ✓

Suggestion: extend test_na_string_values_are_preserved to include a couple of the other tokens (e.g. N/A, NULL, nan) so the test documents the full intent and guards against a regression on the broader set, not just "NA". Cheap insurance, non-blocking.

The {nan: None} entry in the .replace(...) is now effectively dead (with keep_default_na=False + dtype=object pandas never emits NaN), but it is harmless defensive code — fine to leave as-is.

[devin-ai-integration]

Anatolii Yatsuk (@tolik0) ☑️ Resolved in 6e4989f. Extended test_na_string_values_are_preserved to cover 8 of pandas' default NA tokens: NA, N/A, NULL, None, nan, null, #N/A, <NA>.

Agreed on the {nan: None} in .replace() being effectively dead code now — leaving it as harmless defensive code.

---

Devin session

[Anatolii Yatsuk (tolik0)]

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/28020507330

[Copilot]

Pull request overview

This PR updates the declarative ResponseToFileExtractor CSV parsing behavior to avoid Pandas’ default conversion of common “NA-like” string tokens (e.g., "NA", "N/A", "null") into NaN/None, which can silently corrupt legitimate source values (e.g., Salesforce picklists).

Changes:

• Pass keep_default_na=False to pd.read_csv(...) so NA-like string tokens are preserved as literal strings.
• Explicitly map empty-string cells ("" / empty fields) to None to preserve prior behavior for missing values.
• Add unit tests covering both preservation of NA-like tokens and conversion of empty fields to None.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
airbyte_cdk/sources/declarative/extractors/response_to_file_extractor.py	Disables Pandas default NA token coercion and normalizes empty strings to None during chunk conversion.
unit_tests/sources/declarative/extractors/test_response_to_file_extractor.py	Adds regression tests ensuring NA-like strings are preserved and empty fields still become None.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.