Unrecognised COFF object format when using cl.exe /GL

[jayvdb]

A bit of the history at gimli-rs/object#154 and motivation at indygreg/PyOxidizer#183

The default Python extension C flags on Windows with cl.exe includes /GL (Whole Program Optimization). The docs say

.obj files produced with /GL will not be available to such linker utilities as EDITBIN and DUMPBIN.

When that is used, the .obj format is very strange. Omit the option, and the usual .obj format is used and goblin and object process it wonderfully.

As I can hack the code involved, omitting that flag is possible, so this doesnt block us, but it would be nice to have the optimisations, and have one less hack. Anyways, I can proceed, so thanks very much for a workable library.

[m4b]

So iiuc we just need to add the magic number to our magic numbers ? Or is something else involved ?

[philipc]

No, this is the equivalent of clang's -flto, which causes the compiler to output bitcode instead of an object file. So it's a different file format (maybe it contains an embedded COFF file though). I'm not aware of any documentation on it, and based on that /GL link it is likely to depend on the compiler version. So in my opinion this is likely to be out of scope for this crate.

[jayvdb]

Given no other tool supports this format, it is reasonable that goblin doesnt either.
What would be a good solution is to detect this format based on what little info we can find via docs or a bit of experimentation, and emitting an error that describes the probable cause and how to avoid it, or support it as a container format where the innards are not penetrable (yet).

fwiw, I tried parsing the stream starting at offset 6, which is where the COFF signature starts, and it did seem to get a little way through it, with error Scroll(BadInput { size: 7, msg: "invalid utf8" }), which seemed to indicate it was getting into reading a section and failing on the sym names table. That suggests the COFF metadata was parsable. It would be useful to at least extract the arch of the .obj.

[m4b]

@jayvdb sure if you want to submit a PR making error messaging better or somehow returning more info feel free! If no I’m going to close this issue though since I don’t think there is much else we can do.

Let me know !

[jayvdb]

I think making error messaging better is something I can do, so I would like to try it.

[nico-abram]

I would be surprised if the /GL .obj files were alike regular COFF files at all, since according to this Raymond Chen blog post :

In the Visual Studio system, one of the ways of activating a large set of non-classical behaviors is to enable Whole program optimization, also known as Link-time code generation. When this is enabled, the division of labor between the compiler and linker shifts. The compiler still takes source code, but instead of generating machine code, it merely generates a parse tree (perhaps partially-digested). The linker then takes all these parse trees, combines them together (according to classical rules), and generates machine code.