fix(low-code): make Spec.generate_spec idempotent and non-mutating - #1103
fix(low-code): make Spec.generate_spec idempotent and non-mutating#1103Daryna Ishchenko (darynaishchenko) wants to merge 2 commits into
Conversation
Previously generate_spec() converted advanced_auth enum fields
(auth_flow_type, scopes_join_strategy) to strings by assigning the
converted values back onto the typed model. A second call in the same
process then raised AttributeError ('.value' on a str), and any other
reader of advanced_auth saw a string where an enum is expected.
Now the model is serialized to a dict first and enum values are
normalized only in that throwaway copy, so repeated calls are
idempotent and the typed model is never mutated. Adds a regression
test that calls generate_spec() twice and asserts the model keeps its
enum.
Split out of #1066 per review feedback, where this fix was bundled
with an unrelated feature.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@daryna/spec-generate-spec-idempotency#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch daryna/spec-generate-spec-idempotencyPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
Reverts the spec.py enum-handling change and its test to the main version; the fix now lands separately via #1103, as requested in review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthrough
ChangesSpecification generation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
unit_tests/sources/declarative/spec/test_spec.py (1)
165-180: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCover the nested enum branch too.
airbyte_cdk/sources/declarative/spec/spec.pynormalizesscopes_join_strategyat Lines 65-66, but this test only sets and checksauth_flow_typeat Lines 170 and 180. A regression in the nested branch would still pass. Could we add an OAuth input with a non-defaultscopes_join_strategy, then verify repeated output and preservation of the original nested enum, wdyt?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@unit_tests/sources/declarative/spec/test_spec.py` around lines 165 - 180, The test_generate_spec_is_idempotent_and_does_not_mutate_the_model test should also exercise the nested OAuth scopes_join_strategy normalization path. Configure advanced_auth with a non-default scopes_join_strategy, then assert repeated generate_spec outputs remain equal and the original nested scopes_join_strategy enum is unchanged.airbyte_cdk/sources/declarative/spec/spec.py (1)
58-68: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid using the deprecated Pydantic v2 dump API.
The project declares Pydantic
^2.7and only ignoresExperimentalClassWarningin tests, so thisdict()call can emit deprecation noise. Usemodel_dump(mode="python")here, wdyt?Proposed change
- advanced_auth = self.advanced_auth.dict() + advanced_auth = self.advanced_auth.model_dump(mode="python")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@airbyte_cdk/sources/declarative/spec/spec.py` around lines 58 - 68, Replace the deprecated dict() call in the advanced_auth serialization flow with Pydantic’s model_dump(mode="python"), preserving the existing Enum normalization and subsequent oauth configuration handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@airbyte_cdk/sources/declarative/spec/spec.py`:
- Around line 58-68: Replace the deprecated dict() call in the advanced_auth
serialization flow with Pydantic’s model_dump(mode="python"), preserving the
existing Enum normalization and subsequent oauth configuration handling.
In `@unit_tests/sources/declarative/spec/test_spec.py`:
- Around line 165-180: The
test_generate_spec_is_idempotent_and_does_not_mutate_the_model test should also
exercise the nested OAuth scopes_join_strategy normalization path. Configure
advanced_auth with a non-default scopes_join_strategy, then assert repeated
generate_spec outputs remain equal and the original nested scopes_join_strategy
enum is unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5efc3719-ad5c-43b6-b52f-96d9bf8ee265
📒 Files selected for processing (2)
airbyte_cdk/sources/declarative/spec/spec.pyunit_tests/sources/declarative/spec/test_spec.py
PyTest Results (Full)4 190 tests 4 178 ✅ 12m 36s ⏱️ Results for commit cdcbb8d. ♻️ This comment has been updated with latest results. |
Anatolii Yatsuk (tolik0)
left a comment
Anatolii Yatsuk (tolik0)
left a comment
There was a problem hiding this comment.
Nice, well-scoped fix — thanks for splitting it out of #1066.
I verified this locally against the PR head:
- The bug reproduces on
main: a secondgenerate_spec()raisesAttributeError: 'str' object has no attribute 'value'. - On this branch, two calls compare equal, the model keeps its enum, and the nested
scopes_join_strategypath also round-trips correctly ('comma'in the output,ScopesJoinStrategy.commastill on the model). pytest unit_tests/sources/declarative/spec/test_spec.py→ 12 passed;mypy,ruff check, andruff format --checkall clean.
The approach is sound. Pydantic v1 .dict() preserves Enum members (no use_enum_values on these models) and builds fresh nested dicts, so the isinstance(..., Enum) guards fire and the in-place mutation of oauth_input correctly propagates into the parent dict. Nothing else in the CDK reads advanced_auth.auth_flow_type, so no caller depended on the old mutation side effect. Dropping the two # type: ignore comments is a nice bonus.
Two things worth addressing before merge (inline), plus one optional simplification.
… semantics Address review feedback on #1103: - Replace the hand-written walk over the two known enum fields with a single JSON round-trip, so enums are normalized at any depth. declarative_component_schema.py is code-generated and already defines 12 Enum classes; the explicit form would silently pass an Enum through to ConnectorSpecificationSerializer the next time an enum field is added under the advanced_auth subtree. - Rename the local to advanced_auth_dict to make clear it is the serialized copy. - Parametrize the idempotency test over the top-level, nested and no-auth_flow_type shapes, and assert the emitted scopes_join_strategy is the plain string the protocol declares (it is typed Optional[str], so an un-normalized enum would otherwise slip past both the serializer and the first == second check). - Add a test pinning the auth_flow_type=None passthrough, which previously raised AttributeError for a manifest carrying only a predicate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
CodeRabbit (@coderabbitai) addressing your two nitpicks — one applied, one doesn't hold:
# airbyte_cdk/sources/declarative/models/declarative_component_schema.py:9
from pydantic.v1 import BaseModel, Extra, Field
So the proposed diff would be an |
|
Tip For best results, initiate chat on the files or code changes.
The You are correct about 🐇 ✏️ Learnings added
You are interacting with an AI system. |
2 participants
Summary
Spec.generate_spec()converted theadvanced_authenum fields (auth_flow_type, nestedscopes_join_strategy) to their string values by assigning the converted values back onto the typed model. That in-place mutation meant:generate_spec()call in the same process raisedAttributeError(.valueon an already-convertedstr) — hit by any flow that generates the spec more than once (Connector Builder, tests);self.advanced_authafterwards saw a string where the type system promises an enum.This change serializes the model to a dict first and normalizes enum values only in that throwaway copy, so repeated calls are idempotent and the typed model is never mutated. Normalization happens via a single JSON round-trip (
json.loads(self.advanced_auth.json())) rather than a hand-written walk over the two known enum fields —declarative_component_schema.pyis code-generated and already defines 12Enumclasses, so an explicit walk would silently pass anEnumthrough toConnectorSpecificationSerializerthe next time an enum field is added anywhere under theadvanced_authsubtree.Behavior change:
advanced_authwithout anauth_flow_typeAuthFlowis a pydantic model, soif self.advanced_auth:is always truthy. A manifest that setsadvanced_authwith only apredicate_key(noauth_flow_type) previously hard-failed withAttributeError: 'NoneType' object has no attribute 'value'; it now emitsAdvancedAuth(auth_flow_type=None, ...).Passing
Nonethrough is the more correct behavior — bothAuthFlow.auth_flow_type(declarative_component_schema.py:1618) and the protocolAdvancedAuth.auth_flow_type(airbyte_cdk/models/airbyte_protocol.py:119) areOptional. This PR pins that semantic with a dedicated test rather than leaving it implicit.Context
Split out of #1066, where this fix was bundled with the (unrelated)
RateLimitedMultipleTokenAuthenticatorfeature; review feedback there asked for it to be its own PR. The fix is being removed from #1066.Testing
unit_tests/sources/declarative/spec/test_spec.py— 15 passed. The idempotency test is parametrized over threeadvanced_authshapes so both normalization branches and theNonecase are covered:top_level_auth_flow_type_enumnested_scopes_join_strategy_enum— asserts the emittedscopes_join_strategyis the plain string the protocol declares. That field is typedOptional[str], so an un-normalized enum would otherwise slip past both the serializer and thefirst == secondcheck.no_auth_flow_typeVerified the new assertions actually bite: reverting
generate_specto a plain.dict()with no normalization fails 4 tests, including both new parametrized cases.mypy,ruff check, andruff format --checkare clean on both touched files.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Tests