Skip to content

Use unsigned values for the final DES left shifts - #9197

Open
dand-oss wants to merge 1 commit into
FirebirdSQL:masterfrom
dand-oss:fix/des-unsigned-shift
Open

dand-oss wants to merge 1 commit into
FirebirdSQL:masterfrom
dand-oss:fix/des-unsigned-shift

Conversation

@dand-oss

@dand-oss dand-oss commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Convert L1 and R1 to ULONG with static_cast before the final left shifts in des_cipher. This preserves the output bits while avoiding shifts of negative signed values in pre-C++20 builds.

The negative-shift UBSan diagnostic is reproducible in C++17. Master now uses C++20, so this PR does not claim the same operation is undefined under master's current language mode; the change also provides a small backportable fix for the Firebird 5 line.

Validation on Linux x86-64:

  • The patched production enc.cpp passes GCC 16.2 C++20 syntax compilation with upstream's generated configuration headers.
  • A single-threaded harness using the enc.cpp algorithm, with only its Firebird mutex wrapper removed and its integral typedefs supplied, reproduces left shift of negative value -1608722304 on the unpatched code under Clang 21 / C++17 / UBSan.
  • Patched C++17 + UBSan and patched C++20 both match system libcrypt for 24,576 password/salt combinations (six passwords, all 4,096 two-character salts). Unpatched C++20 also matches the same reference.
  • git diff --check passes.

A complete patched server build has not been completed locally; the algorithm harness does not test Firebird's locking or initialization lifecycle.

Cast L1 and R1 to ULONG before left shift operations to avoid
undefined behavior when these signed values are negative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant