[Gemini] Add compatibility classes between apitools client and modern bq client - #39998
[Gemini] Add compatibility classes between apitools client and modern bq client#39998jrmccluskey wants to merge 4 commits into
Conversation
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @damccorm This should be good to take a run at, had to track down a docs precommit problem. Going to put improving the readability of the output of that test suite on my to-do list. |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
damccorm
left a comment
damccorm
left a comment
There was a problem hiding this comment.
Left a couple of preliminary comments to make sure I'm understanding the approach broadly, thanks
| "TableReference"): | ||
| TableReference = apitools_bigquery.TableReference | ||
| DatasetReference = getattr( | ||
| apitools_bigquery, "DatasetReference", None) or _DatasetReferenceCompat |
There was a problem hiding this comment.
If apitools is imported, will we ever need this fallback? Same thing applies elsewhere
There was a problem hiding this comment.
Yeah that assignment block was just redundant, e.g. the apitools library is at present always available in Beam as a core dependency. Cleaned that up.
There was a problem hiding this comment.
Well, correction in that the apitools library is a GCP extras dependency. Need some extra massaging to make the import clean for unit tests in an environment where apitools itself isn't installed.
There was a problem hiding this comment.
Fixing the root cause of the confusion here separately in #40082, will also probably see if this antipattern exists elsewhere in the code base
| self.f = f or [] | ||
|
|
||
|
|
||
| if apitools_bigquery is not None and hasattr(apitools_bigquery, |
There was a problem hiding this comment.
Who do we expect to need this fallback? IIUC, it would just be people who depend on the internal generated BQ client, but we don't expect "normal" users to use it. Is that right? If so, I think we should force users to opt in to the fallback somehow - this will both alert them that the deprecated module will eventually go away and that they're in a potentially unsafe mode.
There was a problem hiding this comment.
These classes should only really be needed if the old client and types are being used, yes. The intent is to throw a warning on import of the generated client and types (this is shown in the master PR and will be done in a follow-up) but if we wanted to make a really explicit opt-in here we could require some sort of arg?
There was a problem hiding this comment.
The intent is to throw a warning on import of the generated client and types (this is shown in the master PR and will be done in a follow-up) but if we wanted to make a really explicit opt-in here we could require some sort of arg?
Yeah, I think we should do this (and still warn if they set it). This could probably just be an environment variable the user adds directly before the import since actually plumbing through an arg seems hard.
My thought here, though, is basically:
- This module is clearly internal. I think we're within our "rights" to make a breaking change
- This module will go away, and the sooner folks start to move off of it the better.
- At the same time, it is helpful to provide a smooth upgrade path so that folks can address this independently of the beam version bump
- Warnings are easy to miss/ignore, while a forced action with an exception at least makes it clear that (a) this is potentially unsafe, and (b) this will go away.
As much as possible, I want to avoid users depending on this without knowing (since I don't think it is guaranteed to be a perfect 1:1 replacement)
There was a problem hiding this comment.
Implicit here is that I have a hard time trusting this shim. I expect it works in the happy path cases, but I doubt we're replicating behavior perfectly.
FWIW, the rest of the PR LGTM
2 participants
Creates compatibility shims between generated BigQuery classes and the modern google-cloud-bigquery client to be used in migrating support over to the modern client.
Master PR - #39889
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.