Use the Windows API to demangle MSVC symbols on Windows

[ZivDero]

This PR adds support to use the Windows API to demangle MSVC symbols on Windows.
Shouldn't affect other platforms.
Resolves #252

[encounter]

Thanks for the PR! While a cool idea in theory, I’d much prefer to try to improve the built-in demangler as much as possible. That way, all platforms benefit (including web / decomp.me).

[ZivDero]

Thanks for the PR! While a cool idea in theory, I’d much prefer to try to improve the built-in demangler as much as possible. That way, all platforms benefit (including web / decomp.me).

While I agree in principle, since MSVC mangling isn't documented, that may prove more challenging.

Here are the test diff for this PR:

────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────
  143   143 │         },
  144   144 │         Symbol {
  145   145 │             name: "?Dot@Vector@@QEAAMPEAU1@@Z",
  146   146 │             demangled_name: Some(
  147       │-                "public: float __cdecl Vector::Dot(struct Vector *)",
        147 │+                "public: float __cdecl Vector::Dot(struct Vector * __ptr64) __ptr64",
  148   148 │             ),
  149   149 │             address: 0,
  150   150 │             size: 87,
  151   151 │             kind: Function,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  158   158 │         },
  159   159 │         Symbol {
  160   160 │             name: "?DistSq@Vector@@QEAAMPEAU1@@Z",
  161   161 │             demangled_name: Some(
  162       │-                "public: float __cdecl Vector::DistSq(struct Vector *)",
        162 │+                "public: float __cdecl Vector::DistSq(struct Vector * __ptr64) __ptr64",
  163   163 │             ),
  164   164 │             address: 0,
  165   165 │             size: 141,
  166   166 │             kind: Function,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  173   173 │         },
  174   174 │         Symbol {
  175   175 │             name: "?Sub@Vector@@QEAAXPEAU1@0@Z",
  176   176 │             demangled_name: Some(
  177       │-                "public: void __cdecl Vector::Sub(struct Vector *, struct Vector *)",
        177 │+                "public: void __cdecl Vector::Sub(struct Vector * __ptr64,struct Vector * __ptr64) __ptr64",
  178   178 │             ),
  179   179 │             address: 0,
  180   180 │             size: 105,
  181   181 │             kind: Function,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  188   188 │         },
  189   189 │         Symbol {
  190   190 │             name: "?Vector_MagSquared@@YAMPEAUVector@@@Z",
  191   191 │             demangled_name: Some(
  192       │-                "float __cdecl Vector_MagSquared(struct Vector *)",
        192 │+                "float __cdecl Vector_MagSquared(struct Vector * __ptr64)",
  193   193 │             ),
  194   194 │             address: 0,
  195   195 │             size: 82,
  196   196 │             kind: Function,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  203   203 │         },
  204   204 │         Symbol {
  205   205 │             name: "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z",
  206   206 │             demangled_name: Some(
  207       │-                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector *, struct Vector *, struct Vector *, float)",
        207 │+                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector * __ptr64,struct Vector * __ptr64,struct Vector * __ptr64,float)",
  208   208 │             ),
  209   209 │             address: 0,
  210   210 │             size: 429,
  211   211 │             kind: Function,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  574   574 │         },
  575   575 │         Symbol {
  576   576 │             name: "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcName$1",
  577   577 │             demangled_name: Some(
  578       │-                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector *, struct Vector *, struct Vector *, float)",
        578 │+                "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcName$1",
  579   579 │             ),
  580   580 │             address: 16,
  581   581 │             size: 12,
  582   582 │             kind: Object,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  589   589 │         },
  590   590 │         Symbol {
  591   591 │             name: "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcName$2",
  592   592 │             demangled_name: Some(
  593       │-                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector *, struct Vector *, struct Vector *, float)",
        593 │+                "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcName$2",
  594   594 │             ),
  595   595 │             address: 28,
  596   596 │             size: 20,
  597   597 │             kind: Object,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  604   604 │         },
  605   605 │         Symbol {
  606   606 │             name: "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcVarDesc",
  607   607 │             demangled_name: Some(
  608       │-                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector *, struct Vector *, struct Vector *, float)",
        608 │+                "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcVarDesc",
  609   609 │             ),
  610   610 │             address: 48,
  611   611 │             size: 192,
  612   612 │             kind: Object,
┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
  619   619 │         },
  620   620 │         Symbol {
  621   621 │             name: "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcFrameData",
  622   622 │             demangled_name: Some(
  623       │-                "bool __cdecl Tools_CapsuleTestMagSq(struct Vector *, struct Vector *, struct Vector *, float)",
        623 │+                "?Tools_CapsuleTestMagSq@@YA_NPEAUVector@@00M@Z$rtcFrameData",
  624   624 │             ),
  625   625 │             address: 240,
  626   626 │             size: 16,
  627   627 │             kind: Object,
────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────

Most of these are alright IMO, though the fact that Microsoft's official API can't unmangle the Tools_CapsuleTestMagSq ones is interesting.

What are your thoughts on this?

[encounter]

The fact that there's already instances where it regresses is unfortunate! And the __ptr64 feels like noise to me.

I just tried out the undname-rs library with

❯ cargo run --example undecorate_name '?_Release@?$_com_ptr_t@V?$_com_IIID@UIPiggyback@@$1?IID_IPiggyback@@3U_GUID@@B@@@@AAEXXZ'
private: void __thiscall _com_ptr_t<class _com_IIID<struct IPiggyback, &struct _GUID const IID_IPiggyback>>::_Release(void)

and it seems to work with the COM symbols, which is great! But it chokes on the same symbols that UnDecorateSymbolName does. Those look like they're static variables at a glance?

[ZivDero]

The fact that there's already instances where it regresses is unfortunate! And the __ptr64 feels like noise to me.

I just tried out the undname-rs library with

❯ cargo run --example undecorate_name '?_Release@?$_com_ptr_t@V?$_com_IIID@UIPiggyback@@$1?IID_IPiggyback@@3U_GUID@@B@@@@AAEXXZ'
private: void __thiscall _com_ptr_t<class _com_IIID<struct IPiggyback, &struct _GUID const IID_IPiggyback>>::_Release(void)

and it seems to work with the COM symbols, which is great! But it chokes on the same symbols that UnDecorateSymbolName does. Those look like they're static variables at a glance?

I know for a fact that IDA Pro also fails to unmangle function static variables. Perhaps it's also using the Windows API. Scratch that, it does not. What we could do is leave msvc-demangler in place, but then if we're on Windows pass the result to the Windows function, so that it can unmangle anything that the the lib couldn't. That way we don't generate extra diffs and also get the widest coverage.

[encounter]

Sure, having the fallback seems good, but it looks like msvc-demangler is actually ignoring the static variable name, and instead just printing the demangled function itself, which seems like a bug too.

[ZivDero]

Sure, having the fallback seems good, but it looks like msvc-demangler is actually ignoring the static variable name, and instead just printing the demangled function itself, which seems like a bug too.

Right, that's a bug then...
http://demangler.com/ also does the same and gives the function name instead.

Your call, I suppose, whether we want to keep this not-quite-right behavior in.
Checked undname.exe that comes with Visual Studio - also does not demangle the data symbol. Notably, this only applies to in-function data, globals are de-mangled fine.

[HaydnTrigg]

Just weighting in after having done quite a few PC and Xbox 360 titles exported now and dealt a lot function declaration reconstruction for source generation and IDA. The best mangle/demangler is the one inside of the Clang code. Please don't ever make the built-in Windows demangler the default.

I have tested the differences of the Clang and MSVC demanglers on many different PC (32 and 64bit) and Xbox 360 titles and found that I can't trust the built-in Windows mangle/demangler for anything reasonably complicated. It is hot garbage and breaks on many of the internal compiler mangled symbols and struggles a lot with strange cases like defining types/lambdas inside of templated functions.

https://github.com/llvm/llvm-project/blob/main/clang/lib/AST/MicrosoftMangle.cpp