Fix corrupted regex patterns in secret key detection#63
Conversation
Several secret detection patterns in keys_extractor() were corrupted since
the initial commit - curly braces {32} rendered as f32g, escaped dots \.
rendered as n, and escaped dollars \$ rendered as n$. This caused Google
YouTube OAuth, Amazon MWS, and PayPal/Braintree token patterns to never
match any real secrets.
Also fixed: "PayPal" label was actually describing the Amazon MWS token
format (amzn.mws.*), and "Amazon MWS" had the PayPal/Braintree
access_token$production$ format. Labels are now correct. Fixed "Slack
Webook" typo to "Slack Webhook". Converted affected patterns to raw
strings to prevent future escape corruption.
|
|
There was a problem hiding this comment.
Pull request overview
This PR updates keys_extractor() secret-detection regex patterns that were previously corrupted (e.g., mangled quantifiers and escaped characters), aiming to restore detection for several token formats (Google OAuth-related values, Amazon MWS tokens, and PayPal/Braintree access tokens) and correct a Slack webhook label typo.
Changes:
- Replaced corrupted Google OAuth/YouTube OAuth regex patterns with corrected patterns, using raw string literals for safer escaping.
- Fixed Amazon MWS and PayPal/Braintree token regex patterns and adjusted labels to match the actual formats.
- Corrected the
"Slack Webook"label typo to"Slack Webhook".
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "Google OAuth Secret": r"[0-9a-zA-Z\-_]{24}", | ||
| "Google OAuth Auth Code": r"4/[0-9A-Za-z\-_]+", | ||
| "Google OAuth Refresh Token": r"1/[0-9A-Za-z\-_]{43}|1/[0-9A-Za-z\-_]{64}", | ||
| "Google OAuth Access Token": r"ya29\.[0-9A-Za-z\-_]+", | ||
| "Google API Key": r"AIza[0-9A-Za-z\-_]{35}", |
There was a problem hiding this comment.
Acknowledged — these patterns were completely non-functional before (corrupted quantifiers like f32g instead of {32}). This PR makes them syntactically correct. The remove_url_from_keys() stripping issue is pre-existing and orthogonal — will open a follow-up for that.
| "Google YouTube OAuth ID Gmail, GCloud": r"[0-9]+-[0-9A-Za-z_]{32}\.apps\.googleusercontent\.com", | ||
| "Amazon MWS": r"amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", | ||
| "PayPal Braintree": r"access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}", |
There was a problem hiding this comment.
Same as above — corrected the regex syntax. The dot/underscore stripping by the sanitizer is a separate pre-existing issue. Also converted this to a raw string for consistency in the follow-up commit.
| "AWS": "(?:.*awsSecretKey|.*aws_secret|.*api-key|.*aws_account_secret).*" | ||
| "(?=.*[A-Z])(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])", | ||
| "Slack Webook": "T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}", | ||
| "Slack Webhook": "T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}", |
There was a problem hiding this comment.
Converted to raw string in the follow-up commit for consistency. The slash stripping is part of the same sanitizer issue — will address in a separate PR.
|
Re: Copilot's review about Valid point. The sanitizer does strip characters that some of these patterns depend on (dots in However, this is a pre-existing issue that's separate from the regex corruption. Before this fix, these patterns used mangled quantifiers like The sanitizer stripping issue existed before and should be addressed separately — either by running Happy to open a follow-up issue for that if maintainers agree on the approach. Also converting the Slack Webhook pattern to a raw string for consistency as suggested. |
Several regex patterns in
keys_extractor()have been non-functional since the initial commit. The curly brace quantifiers got mangled during copy-paste (likely from a rendered source like a PDF or webpage) —{32}becamef32g, escaped dots\.becamen., and escaped dollar signs\$becamen$.This means Google YouTube OAuth IDs, Amazon MWS tokens, and PayPal/Braintree access tokens were never being detected by the key extractor, regardless of how many scans were run.
What was broken:
[0-9]+-[0-9A-Za-z_]f32gn.appsn.googleusercontentn.com— matches nothingaccess_tokenn$productionn$[0-9a-z]f16gn$[0-9a-f]f32g— matches nothingamznn.mwsn.[0-9a-f]f8g-[0-9a-f]f4g-...— matches nothingWhat it should be:
[0-9]+-[0-9A-Za-z_]{32}\.apps\.googleusercontent\.comamzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}Also fixed the label swap — "PayPal" was labeled on what is actually the Amazon MWS pattern (amzn.mws.UUID) and vice versa. Corrected "PayPal" to "PayPal Braintree" and "Amazon MWS" to match the actual token format. Fixed "Slack Webook" typo.
All patterns are now raw strings to prevent future escape issues.
Tested against known token formats — all six previously-broken patterns now correctly match real secrets.