diff --git a/docs/for-contributors/Generator/generator-mods.md b/docs/for-contributors/Generator/generator-mods.md new file mode 100644 index 0000000000..fdac8c17ec --- /dev/null +++ b/docs/for-contributors/Generator/generator-mods.md @@ -0,0 +1,683 @@ +# Generator Mods + +Silk's SilkTouch bindings generator is designed to be a linear pipeline where a set of mods sequentially transform C# +source code represented by Roslyn syntax nodes. This approach used by Silk 3 is in contrast to the approach in Silk 2, +where a monolithic generator output code represented by bespoke data structures. Silk 3 focuses on breaking down each +transformation step into its own mod to aid in maintainability and understanding of the codebase. + +This document explains how mods are implemented and the mods provided by the generator. + +## Implementation + +### IMod Interface + +SilkTouch mods implement the `IMod` interface, which contains the `InitializeAsync` and `ExecuteAsync` methods. + +The primary work of a mod is done within the `ExecuteAsync` method. This method takes in a +`Microsoft.CodeAnalysis.Project` containing the C# source code representing the current state of the bindings. This is +the primary input *and* output of each mod. The output of each mod is passed directly into the next mod for further +transformation. + +`InitializeAsync` is rarely used and is used to initialize data before any transformations have begun so that other mods +can access that data. This is especially so because most communication between mods should be done through the C# source +code representing the generated bindings instead. + +### Mod Configuration + +Mods are configured through the generator config JSON file. Silk's config file is named `generator.json` and is located +at the root of the Silk.NET repository. This config file can be used as reference for your own config files. In +addition, most mods have a configuration class located in their source code with additional documentation. + +## Available Mods + +This section provides a high level explanation of what each mod does. The list is sorted in alphabetical order. + +In particular, config options and specific implementation details are omitted here as the source code documentation +provides the information in a more clear format with less chance of being outdated. However, high level design decisions +will be documented here. + +Standardized sections: + +- **Mod categories** - Assigns a category to the mod and is purely for documentation purposes. Alphabetically sorted. + This allows for easy searching for related mods. The category is also used to provide recommendations and information + relating to those categories. + +- **Name affix categories** - Lists the name affix categories that the mod adds. Alphabetically sorted. Please refer + to the [Name Processing](name-processing.md) documentation to understand what these are and how to make use of them. + That said, advice for how to handle each category is also provided alongside information about what the name affix + represents. + +- **Usage recommendations** - Provides information such as situations the mod is useful for, how to configure it, and + where to place it in the mod order. This information can include examples of how Silk's own bindings use it or whether + the mod is mainly designed for Silk internal use. + +General recommendations: + +For the most part, mods should be configured to run in an order similar to the mod orders used by Silk's existing +bindings. The same goes for the configuration, but more care needs to be done regarding whether the configuration is +specific to that set of bindings. + +### AddApiProfiles + +Mod categories: Metadata + +This mod adds `[SupportedApiProfile]` attributes throughout the generated bindings for the purpose of providing API +analyzers the ability to understand when a specific API can be used. + +This mod is WIP: + +- Ideally, the mod internally uses `[NativeName]` attributes to associate data with the API exposed by the bindings. + Currently, the managed C# names are used, meaning that name prettification and other name modifications can lead to + inaccurate `[SupportedApiProfile]` attributes. + +Usage recommendations: + +This mod should be positioned late in the mod order, after all APIs have been added to the generated bindings. +If the attribute is missing on a certain API and a later mod adds that API, investigate whether this mod can be moved +to be after that mod. + +### AddIncludes + +Mod categories: Creation + +This mod interacts with `ClangScraper` by providing standard include directories and other user-specified include paths +to `ClangScraper`. + +Usage recommendations: + +This should be positioned at the start of the mod order. + +### AddOpaqueStructs + +Mod categories: Creation + +This mod adds an empty struct for each name specified in its mod configuration. + +Usage recommendations: + +(TODO: To be added) + +### AddVTables + +Mod categories: Transformation + +This mod transforms `[DllImport]` and `[Transformed]` methods to use Silk-style virtual tables. These vtables allow for +different styles of accessing native APIs, such as through an instance of an API object or through static methods. + +Usage recommendations: + +(TODO: To be added) + +### BakeSourceSets + +Mod categories: Transformation + +This mod merges multiple sets of source code into one set of source code. + +Usage recommendations: + +(TODO: To be added) + +### ChangeNamespace + +Mod categories: Transformation + +This mod moves types from one namespace to another. + +Usage recommendations: + +(TODO: To be added) + +### ChangeNativeClass + +Mod categories: Transformation + +This mod moves members from one type to another. + +Usage recommendations: + +(TODO: To be added) + +### ClangScraper + +Mod categories: Creation + +This is a critical mod used to generate the initial set of raw C# bindings for C APIs. The use of this mod is equivalent +to using `ClangSharpPInvokeGenerator`. The C# source code generated by this mod is typically the starting point for all +of the transformations done by the rest of the SilkTouch mods. + +Note that this mod is platform specific and will have different outputs depending on the platform. This is because +`ClangScraper` makes use of system headers. Any platform specific differences in the API for which bindings are being +generated will also affect the output. + +(TODO: Document platform specific differences and how they are handled by Silk) + +Usage recommendations: + +This mod is configured differently from the rest of the mods. The JSON mod configuration follows the same pattern as the +rest of the mods, but the bulk of the configuration comes in the form of `.rsp` files. These `.rsp` files represent the +command line arguments passed to `ClangSharpPInvokeGenerator`. More information about how to use this mod and how to +configure the generator as a whole is found in the [Using the Generator](using-the-generator.md) documentation. + +This mod should be positioned at the start of the mod order, after `AddIncludes`. + +### ExtractHandles + +Mod categories: Creation + +This mod adds empty structs for missing types that are identified to be used as handle types. To be a handle type, the +type must only be ever referenced through a pointer. After the empty struct representing the handle type is extracted, +`TransformHandles` can then be used to transform the pointer to be wrapped within the handle struct. + +Side note: This mod is similar to `AddOpaqueStructs` in that it adds empty structs, but `ExtractHandles` has a much more +automated approach since it deals specifically with handle types that are referenced using pointers. + +This code has been manually trimmed for the sake of example and comes from the state of the Vulkan bindings before +`ExtractHandles` executes. In this case, `VkInstance_T` will be identified as a missing handle type and an empty struct +will be added for it. On the other hand, `VkInstanceCreateInfo` and `VkAllocationCallbacks` will not be affected since +they already exist. Similarly, `VkResult` is not affected because it is not referenced through a pointer. This example +has a matching test case in the SilkTouch unit tests. +```cs +public struct VkAllocationCallbacks; +public struct VkInstanceCreateInfo; + +public class Vk +{ + public static extern VkResult vkCreateInstance( + VkInstanceCreateInfo* pCreateInfo, + VkAllocationCallbacks* pAllocator, + VkInstance_T** pInstance + ); +} +``` + +The result of running `ExtractHandles` on the above code will lead to the creation of a new type: +```cs +public unsafe partial struct VkInstance_T +{ +} +``` + +Usage recommendations: + +This mod should be used if a set of bindings contains types referenced only through pointers and those pointers are +missing from the final set of generated bindings. + +Furthermore, this mod should be used alongside `TransformHandles` so that the handles are transformed into a more +user-friendly version. `ExtractHandles` should be positioned before `TransformHandles` and any other mods that might use +its results in the mod order. + +### ExtractNestedTyping + +Mod categories: Creation + +This mod handles a few responsibilities, the primary of which is to extract nested types out of their parent types and +into the containing namespace as non-nested types. The second is replacing function pointers with structs and delegate +types representing those function pointers. The third is moving C-style enum constants into their respective enums. + +Arguably, the non-primary responsibilities of this mod should be split out into separate mods, but has not yet been done +so due to lack of time invested into doing so. There is also the slight performance hit of splitting out the operations +out since it will lead to three separate passes over the codebase, but this cost is negligible for most reasonably sized +native APIs. + +Examples for how `ExtractNestedTyping` works can be found in the `ExtractNestedTypingTests` test cases. + +Name affix categories: + +- `FunctionPointerDelegateType` - This is a suffix that always has the value of `Delegate`. This is used for the + delegate representation of a function pointer type to distinguish the delegate type from the struct type for extracted + function pointers. + +- `FunctionPointerParent` - This is a prefix used by the delegate representation of a function pointer type. This is + used to ensure that the delegate type always uses the current name of its struct counterpart as part of its own name. + +- `NestedStructParent` - This is a prefix that references the name of the type that the extracted type was previously + nested in. This is used to ensure that the extracted type always uses the current name of its original "parent" type + as part of its own name. + +These affixes are usually left unconfigured in `PrettifyNames`. + +Usage recommendations: + +This mod must be used before `PrettifyNames` when using `PrettifyNames` and there are nested types in the generated +bindings. This is because `PrettifyNames` does not handle nesting when renaming identifiers. Other mods may have similar +restrictions. This restriction is generally because nesting increases complexity, and as such, mods are written with the +assumption that nested types are extracted beforehand. + +### IdentifySharedPrefixes + +Mod categories: Metadata, Naming + +This mod is designed to handle C-style namespace prefixes where all types, functions, and constants in a library share +a common prefix. This includes casing convention differences. For example, constants often use screaming case while +type and function names use camel case or pascal case. This can be seen in Vulkan, where functions are prefixed with +`vk`, such as in `vkCreateInstance` and `vkCmdBindPipeline`, and constants are prefixed with `VK_`, such as in +`VK_MAX_MEMORY_HEAPS` and `VK_TRUE`. + +Furthermore, identification of shared prefixes is done per scope and only in cases where C-style namespace prefixes +might be used. For example, a struct type typically does not use namespace prefixes because the struct itself acts as +a way to disambiguate names contained inside of it. However, C-style enum values are often defined as global constants +using macros such as `#define SDL_BLENDMODE_BLEND_PREMULTIPLIED 0x00000010u`. After these constants are moved to their +corresponding enum types by `ExtractNestedTyping`, `IdentifySharedPrefixes` then handles the identification of the +prefix shared by the enum type's members. In the case of `SDL_BlendMode`, all of the members of `SDL_BlendMode` share +`SDL_BLENDMODE_` as their common prefix. + +Note: Despite `VK_` and `SDL_BLENDMODE_` being the "true" shared prefix, `IdentifySharedPrefixes` annotates the +identifier with `VK` and `SDL_BLENDMODE` as the shared prefix, without the trailing underscore. While the inclusion of +the underscore can be debated and can subtly affect the bindings output by the generator in edge cases, this is the +current behavior of `IdentifySharedPrefixes`. + +Implementation-wise, this mod's functionality was notably originally part of `PrettifyNames`. In the original form, +`PrettifyNames` handled both the identification of and removal of shared prefixes. This has now been split out to +simplify `PrettifyNames` and to provide better control over how shared prefixes are processed. + +Examples for how `IdentifySharedPrefixes` works can be found in the `IdentifySharedPrefixesTests` test cases. + +The [Name Processing](name-processing.md) documentation also covers `IdentifySharedPrefixes` to a limited extent, +notably relating to how existing name affixes are treated in the +[IdentifySharedPrefixes and Name Affixes](name-processing.md#identifysharedprefixes-and-name-affixes) section. + +Name affix categories: + +- `SharedPrefix` - This is a prefix that represents the shared prefix that is identified for a group of names. Silk's + bindings typically remove this prefix because these prefixes are typically used to prevent naming collisions in C + libraries. C# has its own namespacing functionality, thus making this prefix irrelevant. This prefix can be kept or + prettified if preferred over removing the prefix. The prefix can also be configured as a discriminator, which removes + the prefix by default, but allows the prefix to be used in case of name conflicts. + +Example name affix configurations for `PrettifyNames`: + +``` +"SharedPrefix": { + "Remove": true +} +``` + +``` +"SharedPrefix": { + "IsDiscriminator": true +} +``` + +Usage recommendations: + +This mod should be used when the transformation of C-style namespace prefixes or similar naming patterns is desired. +The most common case of this is the removal of such prefixes by using `IdentifySharedPrefixes` before `PrettifyNames`. +`PrettifyNames` can then be configured like above to remove or use shared prefixes as discriminators. + +This mod also interacts with `ExtractNestedTyping`, which moves constants that are identified as likely being part of +an enum to the corresponding enum. As such, this `IdentifySharedPrefixes` should be positioned after +`ExtractNestedTyping` in the mod order. + +### InterceptNativeFunctions + +Mod categories: Transformation + +This mod intercepts native functions and allows the generator user to provide a manual implementation in a non-generated +partial file. + +This is done by identifying native functions by name and by their `[DllImport]` attribute. If the native function's name +is one of the functions to intercept, the original method is replaced with two new method: + +1. A `private` version of the original that is suffixed with `-Internal`. +2. A `public` version with no method body, but using the `partial` keyword. + +This second `partial` method is what allows generator users to provide their own implementation. Similar to overriding +`virtual` methods, the generator user is free to do anything within this implementation, but common use cases involve +wrapping the method before calling the original `-Internal` suffixed version of the method. + +Examples for how `InterceptNativeFunctions` works can be found in the `InterceptNativeFunctionsTests` test cases. + +Name affix categories: + +- `InterceptedFunction` - This is a suffix that always has the value of `Internal`. This is used for the original + version of the private, intercepted native function to distinguish it from the new, public version. + +Usage recommendations: + +Use this when there is a strong reason to directly replace the original native function rather than create a custom +overload or utility method. + +For example, this is used in the Vulkan bindings to capture the created `Instance` and `Device` objects to be used in +native function loading. Specifically, `vkCreateInstance` and `vkCreateDevice` are intercepted so that the created +objects can be passed to `vkGetInstanceProcAddr` and `vkGetDeviceProcAddr` respectively. In this case, the reason for +intercepting these native functions is so that function pointers can be automatically loaded without user intervention. + +### MarkNativeNames + +Mod categories: Metadata, Naming + +This mod naively adds `[NativeName]` attributes to most identifiers in the generated bindings and is designed to be +placed immediately after `ClangScraper`. Syntax nodes that are not output by `ClangScraper` are intentionally not +processed. + +The name stored by the `[NativeName]` attribute matches the C# identifier at the time the mod runs. This assumes that +the name used by the C# source code matches the name used by the native source code, which is *usually* the case when +this mod is placed immediately after `ClangScraper`. However, there are cases where the names output by `ClangScraper` +do not correspond to native names. These cases are usually because there is no native name available, such as for +inline array types (named in the format `_name_e__FixedBuffer`) or backing fields used by bitfield structs (named +`_bitfield`). + +Usage recommendations: + +As mentioned, this mod is best used immediately after `ClangScraper` runs. This should mark all identifiers output +by `ClangScraper` itself. + +However, do not assume that this mod is sufficient to mark all identifiers present in the final set of generated +bindings. When other mods introduce new identifiers, those mods will need to add `[NativeName]` attributes themselves, +or have another mod do it for them, in the case those identifiers represent a native API. + +Because the name stored as the value for `[NativeName]` comes from the native API, it can be used as a stable identifier +for an API. This is in contrast to the current C# identifier being used, as that identifier is often transformed by mods +such as `[PrettifyNames]`. + +### MixKhronosData + +Mod categories: Creation, Metadata, Naming, Transformation + +This is a monolithic mod that handles behavior specific to Khronos-style APIs such as OpenGL, OpenAL, Vulkan, and more. + +Because this mod is intended more for internal use rather than public use, the documentation here will focus on +decisions made during the development of the mod and other internal details rather than the exact usage of the mod. +For information relating to how the mod should be used, please use Silk's `generator.json` configuration, source code, +and `MixKhronosDataTests` as a reference. + +(TODO: Document major decisions relating to MixKhronosData. This is difficult because of the mod's long development +history. This should be done over time as further changes are made to the mod.) + +To combat the monolithic nature of the mod, the mod is split into multiple phases. This refers to both the +`InitializeAsync` and `ExecuteAsync` phases, as well as the use of multiple rewriters. The mod also implements +multiple interfaces that integrate `MixKhronosData` into the behavior of other mods. + +`InitializeAsync` is where `MixKhronosData` initializes its data by reading the Khronos-style XML specification file +containing data relating to the API that the generator is generating bindings for. A list of such specifications is +provided below. These XML specs roughly follow the same format, but have subtle or major differences depending on the +history of that API. For example, OpenGL is similar to OpenAL, but differs greatly from Vulkan and OpenXR who are +themselves similar. As such, it is best to have all of the specification files open for reference when working on +parsing. It may also be helpful to have the corresponding header files open. + +`ExecuteAsync` is where `MixKhronosData` does multiple sequential transformation steps on the source code representing +the generated bindings. These steps are split into different rewriter phases in a way that focuses on balancing +performance with maintainability. Performance-wise, these rewriters should be combined as much as possible. This is +because repeated loops over the project source code has an associated time cost. However, for sake of maintainability, +it is much easier to understand how transformations are done when they are separated out into different phases. +Fortunately, as long as the transformations do not use the symbol representation (eg: `ISymbol`, `SemanticModel`) of +the source code, the transformation is fairly lightweight and only adds a few seconds to the total execution time of +the mod. + +Khronos-style XML specifications: + +OpenAL: https://raw.githubusercontent.com/kcat/openal-soft/refs/heads/master/registry/xml/al.xml +OpenCL: https://raw.githubusercontent.com/KhronosGroup/OpenCL-Docs/refs/heads/main/xml/cl.xml +OpenGL Windows: https://raw.githubusercontent.com/KhronosGroup/OpenGL-Registry/main/xml/wgl.xml +OpenGL X11: https://github.com/KhronosGroup/OpenGL-Registry/blob/main/xml/glx.xml +OpenGL: https://raw.githubusercontent.com/KhronosGroup/OpenGL-Registry/refs/heads/main/xml/gl.xml +OpenXR: https://raw.githubusercontent.com/KhronosGroup/OpenXR-SDK-Source/main/specification/registry/xr.xml +Vulkan: https://raw.githubusercontent.com/KhronosGroup/Vulkan-Docs/refs/heads/main/xml/vk.xml + +Be aware that these link to the latest version. Silk's repo may be using an older version of these XML files. + +Name affix categories: + +- `KhronosFunctionDataType` - `IdentifyFunctionDataTypes` must be set to true in the `MixKhronosData` configuration for + this affix category to be identified. This is a suffix relevant to OpenGL-like APIs where functions like `glColor3` + have variants such as `glColor3i`, `glColor3f`, and `glColor3b`. These suffixes indicate the data type that the + function expects. In this case, integer, float, and byte, respectively. Silk's bindings configure these as + discriminators so that they can be removed when removing them does not lead to method overload conflicts. + +- `KhronosHandleType` - This is a suffix used on handle structs resulting from the typedefs used by Khronos in their + headers. For example, Vulkan uses the following macro to define handle types: + `#define VK_DEFINE_HANDLE(object) typedef struct object##_T* object;`, used as `VK_DEFINE_HANDLE(VkInstance)`. + Although Vulkan uses the handle type as `VkInstance`, `ClangScraper` outputs the type as `VkInstance_T` due to the + typedef. As such, `MixKhronosData` identifies this suffix so that `PrettifyNames` can be configured to remove this + suffix later. + +- `KhronosImpliedVendor` - `IdentifyEnumMemberImpliedVendors` must be set to true in the `MixKhronosData` configuration + for this affix category to be identified. This is a suffix used on enum members instead of `KhronosVendor` when an + enum member has the same vendor suffix as **the containing enum type. This suffix exists in native code because the + enum member is usually defined as a standalone, global constant without any other context whether the enum member is + part of an extension. In C#, this is not a problem because the enum type itself conveys that information. For example, + in Vulkan, `VkPresentModeKHR` in Vulkan is a `KHR` suffixed enum type that contains `VK_PRESENT_MODE_IMMEDIATE_KHR` as + a member. In C#, this `KHR` suffix on the member is redundant. As such, Silk's bindings are configured to remove this + suffix. + +- `KhronosNamespaceEnum` - This is a prefix added to the "namespace" enum of OpenGL-like APIs such as `GLEnum`, + `ALEnum`, and `ALCEnum`. In this case, the value of the prefix would be `GL`, `AL`, and `ALC`, respectively. This is + so that the casing of the prefix is preserved by `PrettifyNames`. As such, Silk's bindings leaves the affix category + unconfigured in the `PrettifyNames` configuration so that `PrettifyNames` uses the default behavior of preserving the + affix. + +- `KhronosNonExclusiveVendor` - `IdentifyEnumTypeNonExclusiveVendors` must be set to true in the `MixKhronosData` + configuration for this affix category to be identified. This is a suffix used on enum types instead of `KhronosVendor` + when the enum type's vendor suffix does not match the vendor suffixes used by the enum members contained within that + type. For example, `BufferUsageARB` has the `ARB` vendor suffix, but contains non-suffixed members such as + `GL_STREAM_DRAW`. Similarly, `GetMultisamplePNameNV` contains `ProgrammableSampleLocationARB`, which is also a + mismatch. This affix category is only intended to be used for OpenGL-like APIs where enum member promotion was not + fully defined, leading to inconsistent vendor suffixing where a non-promoted enum type contains a promoted enum + member. Modern APIs like Vulkan do not have this issue. In modern APIs, there can be "mismatches", but those are cases + where promoted enum types contain non-promoted enum members, which is allowed. As such, Silk's bindings enables + `IdentifyEnumTypeNonExclusiveVendors` and configures `KhronosNonExclusiveVendor` affixes to be removed only for + OpenGL-like APIs. Furthermore, `IdentifyEnumTypeNonExclusiveVendors` also interacts with + `IdentifyEnumMemberImpliedVendors`. Specifically, if an enum type is identified to have a non-exclusive vendor, that + vendor will not be used to identify implied vendors, as it is assumed that the non-exclusive vendor will be removed. + Also note that the behavior of `IdentifyEnumTypeNonExclusiveVendors` can be considered "too aggressive" since it + triggers off of *any* mismatch. For example, if a vendor suffixed enum type contains something generic such as + `GL_NONE`, the enum type vendor suffix will still be identified as a `KhronosNonExclusiveVendor`. This behavior was + ported from the now removed `NameTrimmer`-based implementation and is kept for simplicity and consistency with the old + implementation. + +- `KhronosNonVendor` - This is a suffix added to any identifier that is identified to contain a suffix listed in the + `NonVendorSuffixes` list of the `MixKhronosData` configuration. This is used in cases when a suffix might block the + identification of other suffixes. For example, OpenAL has names such as `alAuxiliaryEffectSlotfDirect` where the + `Direct` suffix is after the `KhronosFunctionDataType` suffix, thus blocking the `KhronosFunctionDataType` suffix + from being identified. In this case, adding `Direct` as a non-vendor suffix fixes the issue. Because this is a + "helper" affix, Silk's bindings leave this affix category unconfigured in the `PrettifyNames` configuration. + +- `KhronosVendor` - This is a suffix added to any identifier that is identified to contain a Khronos vendor suffix such + as `KHR`, `EXT`, or `NV`. The list of vendor suffixes used during identification is retrieved from the provided XML + specification. Silk's bindings are configured to move `KhronosVendor` suffixes to the end of the name. This is to + match Khronos's own naming convention. This primarily affects cases where Silk's generator added additional suffixes + to the end of the name, such as with `HandleType` suffixes like in `DebugUtilsMessengerHandleEXT` in Vulkan. + +Usage recommendations: + +This mod should only be used when generating bindings for Khronos-style APIs. While the mod does not strictly require +the XML specification file, `MixKhronosData` has not been tested for use without the XML specification. + +One key consideration when using `MixKhronosData` is identifying the conventions used by the API for which bindings are +being generated for. Older Khronos APIs are more similar to OpenGL while newer Khronos APIs are more similar to Vulkan. +These conventions determine which settings should be used in the `MixKhronosData` configuration. For exact configuration +details, please refer to the configurations used by Silk for the existing Khronos-style bindings as well as the source +code. + +Note: Although `MixKhronosData` has not been used by Silk for generating bindings for APIs that do not have XML +specifications, we will likely experiment with using this mod for bindings that do not have XML specifications such as +SPIRV-Reflect in the near future. Whether we end up using this mod for these types of bindings depends on how much +benefit `MixKhronosData` provides for those bindings. + +### PrettifyNames + +Mod categories: Naming, Transformation + +This is the mod central to name processing. `PrettifyNames` focuses on the bulk prettification of names and the +processing of name affixes declared by other mods. + +Name processing as a whole and information about the high level implementation details of `PrettifyNames` is available +in the [Name Processing](name-processing.md) documentation. + +Usage recommendations: + +This mod should be positioned late in the mod order to ensure that all mods that introduce new identifiers and +`[NameAffix]` attributes have run. + +The placement of this mod can also affect mods that rely on the C# identifiers of types and their members rather than +the names specified by that identifier's `[NativeName]` attribute. One notable mod of this type is `TransformEnums`. + +When configuring how name affixes are processed in the `PrettifyNames` configuration, note that the name affix category +configuration is designed to be verbose on purpose. If a category is left unconfigured, it will simply use the default +configuration. Mods cannot provide default affix category configurations. This is to ensure that what you see in the +`PrettifyNames` configuration directly corresponds to the output. + +Furthermore, when defining values for the `Order` and `DiscriminatorPriority` properties, prefer to use consecutive +values since there is no need to reserve space for intermediate values when new name affix categories are introduced. +This is because it is easy enough to update the other entries to use a higher/lower value for the set of bindings being +configured. New name affixes are also very unlikely to be introduced after the set of bindings have been created. + +### StripAttributes + +Mod categories: Metadata + +This mod removes attributes that are listed in the `Remove` list of its config. + +Usage recommendations: + +This mod is intended to be used as a way to clean up intermediate metadata attributes and other attributes usually +useful during bindings generation or debugging, but not particularly useful to the end user of the generated bindings. + +These are attributes removed in Silk's own bindings: + +- `NameAffix` - Metadata attribute used to store name affix information. Introduced by various mods. + +- `NativeTypeName` - Metadata attribute used to store native type information. Introduced by `ClangScraper`. + +- `Transformed` - Metadata attribute used to denote that an API is a transformed variant of another API. Introduced by + various mods. + +Tip: When debugging the name processing pipeline, disabling the `StripAttributes` mod (or just removing the +`[NameAffix]` attribute from the list of attributes to be removed) can be helpful. Disabling the stripping of other +attributes can also be helpful for this or other purposes. + +### TransformEnums + +Mod categories: Transformation + +This mod focuses on the transformation of enum types. + +Usage recommendations: + +This mod can be used to reduce the platform specific differences in the generated bindings. For example, enums use +signed backing types by default on Windows while enums default to unsigned backing types on Unix. This is controlled by +the `CoerceBackingTypes` configuration option. + +This mod can also be used to transform `[Flags]` enums in various ways. A `None = 0` member can be added for `[Flags]` +enums that do not have an equivalent. Member values can also be rewritten to use hexadecimal for `[Flags]` enums and +decimal values for normal enums. + +Finally, the last feature is that member can be according to a filter. This filter can filter the type name and the +member name by regex. The filter can also filter by member value. One common use case is for removing the "max value" +enum members used by native libraries to ensure that enum backing types are a specific width. For example, in Vulkan, +the value used is `0x7FFFFFFF`, equivalent to the maximum value of a 32-bit signed integer. + +Note: Currently, `TransformEnums` requires the `[Flags]` attribute on enum types to be added by another mod. +`TransformEnums` does not yet have the functionality to identify enums on its own. This is a relatively high priority +task. However, also note that identification will be done using a heuristic and may not be perfectly accurate. If other +metadata is available for identifying `[Flags]` enums, consider using that instead for better results. + +### TransformFunctions + +(TODO: This section preemptively contains information for changes made by +https://github.com/dotnet/Silk.NET/pull/2574. Remove this todo once the PR is merged.) + +Mod categories: Transformation + +This mod focuses on the transformation of methods, such as by changing parameters types and adding new overloads. + +(TODO: To be expanded) + +Notably, the transformations include transforming methods to use the Silk DSL types (`Ptr`, `Ref`, `MaybeBool`, etc). + +Name affix categories: + +- `RawFunction` - This is a suffix added when method overloads conflict with each other. Specifically, if a transformed + version of a method differs from the original method only by return type, the *original* has the `-Raw` suffix added + along with the corresponding `[NameAffix]` attribute. + +Usage recommendations: + +(TODO: To be expanded) + +The `BoolTypes` property in the configuration should match the configuration used in `TransformProperties`. + +### TransformHandles + +Mod categories: Transformation + +This mod focuses on the transformation of opaque structs into more developer-friendly handle types. + +This is done by finding references to empty structs. If all references to the empty struct are done through pointers, +that struct will be treated as a handle type and transformed. + +Empty structs identified as handle types will be transformed in two ways: + +1. The struct itself will be transformed to wrap the underlying pointer and have methods/operators added for ease of + use. + +2. All references to that struct will have their pointer dimension reduced. For example, `VkBuffer**` becomes + `VkBuffer*` This is because the mentioned struct transformation means that the innermost pointer dimension is now + stored inside the struct. + +This mod currently only processes handle types that wrap pointer types. Integer types are not yet supported. + +Name affix categories: + +- `HandleType` - This is a suffix added to handle types transformed by `TransformedHandles`. Note that + `TransformHandles` only adds the attribute and does not rename the actual handle type. This is so that the rename is + deferred until `PrettifyNames` where all renames are done in bulk for performance reasons. This pattern is explained + in the [Name Processing - Deferring Renames](name-processing.md#deferring-renames) documentation. + +Usage recommendations: + +This mod should be placed after `ExtractHandles` and `AddOpaqueStructs`. This is because `TransformHandles` relies on +the previously mentioned mods to add the empty struct types. `TransformHandles` itself does not add new structs, it only +transforms existing ones that it identifies as a handle type. + +### TransformProperties + +(TODO: This section preemptively contains information for changes made by +https://github.com/dotnet/Silk.NET/pull/2574. Remove this todo once the PR is merged.) + +Mod categories: Transformation + +This mod focuses on the transformation of fields and properties. Despite the name, fields are also handled because they +often need to be transformed alongside properties or have very similar transformations that it makes sense to colocate +these transformations in the same mod. + +This mod currently handles the following transformations: + +1. Transform string constant properties to use the `Utf8String` type. For example, + `static ReadOnlySpan Thing => "thing"u8;` becomes + `static Utf8String Thing => "thing"u8;`. + +2. Transform fields and properties identified to be boolean-like to use the `MaybeBool` type. This is similar to the + transformation done by `TransformFunctions`. + +Usage recommendations: + +(TODO: To be expanded) + +The `BoolTypes` property in the configuration should match the configuration used in `TransformFunctions`. + +## Mod Categories + +### Creation + +These mods focus on the creation of new APIs that strictly do not exist in any form in the current state of the +bindings. + +Generally, these mods should be early in the mod order so that other mods have the chance to modify their outputs. + +### Metadata + +These mods deal with metadata, either by annotating the generated bindings or by providing metadata to other mods. + +### Naming + +These mods deal with the naming of type and member identifiers within the generated bindings. + +### Transformation + +These mods focus on the transformation of existing APIs. While these mods can create new APIs, these new APIs are based +on APIs that already exist in the generated bindings. + +Generally, these mods should be placed after any mods (such as ones in the Creation category) that introduce any APIs +that might get transformed by these Transformation mods. diff --git a/docs/for-contributors/Generator/name-processing.md b/docs/for-contributors/Generator/name-processing.md new file mode 100644 index 0000000000..c5150d7bcb --- /dev/null +++ b/docs/for-contributors/Generator/name-processing.md @@ -0,0 +1,419 @@ +# Name Processing and Prettification + +A primary goal of Silk.NET is to provide a first-class .NET experience for the bindings that it provides. + +One such way that Silk.NET achieves this is by transforming native identifiers into identifiers that follow the +Microsoft Framework Design guidelines. This is the process referred to as "prettification". +Of these guidelines, most notable are the guidelines relating to capitalization. + +Naming Guidelines: https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/naming-guidelines + +Capitalization Conventions: https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/capitalization-conventions + +## High-Level Overview + +This section explains how names flow through the SilkTouch generator pipeline. +- For more information about the pipeline itself, please see the [Generator Mods](generator-mods.md) documentation. + +`vkCreateSwapchainKHR` from Vulkan is used here as an example. + +1. Names enter the pipeline from native sources (eg: C header files). + - Eg: `vkCreateSwapchainKHR` as input from Vulkan during the `ClangScraper` mod. + +2. Mods add metadata to each name as C# attributes. + - `[NativeName("vkCreateSwapchainKHR")]` from `MarkNativeNames` + - `[NameAffix("Suffix", "KhronosVendor", "KHR")]` from `MixKhronosData` + - `[NameAffix("Prefix", "SharedPrefix", "vk")]` from `IdentifySharedPrefixes` + +3. `PrettifyNames` uses the metadata to transform the names according to user-provided configuration. + - The affixes are first stripped → `CreateSwapchain` + - The base name is "prettified" (pascal-casing, removal of underscores) → `CreateSwapchain` (No change in this case) + - Affixes are reapplied according to user configuration → `CreateSwapchainKHR` + - Silk's bindings remove shared prefixes since these represent C namespace prefixes and preserve Khronos vendor + suffixes verbatim for emphasis (notably in contradiction with the Framework Design Guidelines). + +4. Mods strip most metadata from the generated bindings to keep the output clean. + - Silk's bindings keep metadata useful for users, while removing internal generator metadata. + - For example, `[NativeName]` is kept and `[NameAffix]` is removed during the `StripAttributes` mod. + - Tip: Disabling the `StripAttributes` mod can be helpful for debugging unwanted outputs. + +## Test cases + +The behavior for the name processing pipeline is heavily unit tested. Please refer to the unit tests for the +corresponding section of the codebase to see detailed examples of expected inputs and outputs. + +## PrettifyNames + +As seen above, `PrettifyNames` is the mod central to name processing. + +The goal of this mod is to take all of the names from the generated bindings and transform them in bulk. This keeps +other mods performant and simple, as renaming identifiers is a costly operation that involves searching the entire +project for references to that identifier. + +Despite this, `PrettifyNames` also has the goal of remaining dumb and straightforward. It relies on the generator config +for API-specific decisions (eg: removing/reordering affixes, overrides) and other mods for API-specific annotations +(eg: API-specific prefix/suffix conventions). The rest of the processing (eg: prettification), while complex, is done +uniformly. + +This allows `PrettifyNames` to focus strictly on the common case, while edge cases are handled elsewhere. This works +fairly well in practice. Even though the configuration options are limited mostly to how affixes are handled, affixes +are usually where native APIs differ in their naming conventions. Other differences fall outside the common case and are +therefore handled by the generator user or by other mods. + +Furthermore, to keep `PrettifyNames` simple and linear, each step takes the output of the previous step, with no +interweaving of logic. + +`PrettifyNames` works as follows: + +1. All current source code is scraped to gather name information. +2. The names are transformed by a series of name processors. +3. Symbols corresponding to all transformed names are gathered. +4. A symbol-based renamer is used to replace all references to those names with their new versions. +5. Document file names are renamed using the transformed names. + +At time of writing, these are the name processors in use: +```cs +var nameProcessors = new INameProcessor[] +{ + new HandleOverridesProcessor(...), // Overrides are user configurable + new StripAffixesProcessor(...), + new PrettifyProcessor(...), // Acronym threshold is user configurable + new ReapplyAffixesProcessor(...), // Affix reapplication is user configurable + new PrefixIfStartsWithNumberProcessor(), + new ResolveConflictsProcessor(...), + new OutputFinalNamesProcessor(), + new RemoveUnmodifiedFinalNamesProcessor(), +}; +``` + +For specifics on how these processors and other steps work, it is best to refer to the `PrettifyNames` source code. + +## PrettifyNames - Notable Decisions + +Note: It may be helpful to come back to this section after reading about the rest of the name processing pipeline. + +### Strip/Reapply Affixes Scope + +Affixes are stripped and reapplied to create a "scope" where only the base name is visible. + +For example, in the execution order above, `PrettifyProcessor` only affects the base name, but +`PrefixIfStartsWithNumberProcessor` works on the full name with affixes applied. + +Currently, this distinction is not as prevalent as it was when shared prefix trimming was done during `PrettifyNames`. + +Originally, this was implemented so that prefix identification ignores any affixes that have been declared. This +notably affects cases like the `I-` prefix in the Microsoft bindings (the C-style namespace prefix is after the `I-`) +and the vendor suffixes in the Khronos bindings (removing the suffixes before identifying shared prefixes prevents +problematic cases where prefix trimming trims everything except for the vendor suffix, since the vendor suffix was the +only non-shared part of the name; see `OcclusionQueryParameterNameNV` in OpenGL for example). + +Shared prefix identification is now handled by `IdentifySharedPrefixes` by handling affix stripping/reapplication using +the utility methods provided by `NameAffixer`. `PrettifyNames` can be then configured to remove these shared prefixes, +thus matching the original behavior. + +### Strip/Reapply Affixes Configuration + +To keep things simple, only affix reapplication is configurable. This is because the user is expected to configure the +generator output, while mods are expected to handle the process of affix identification. + +Affix reapplication is when common transformations to affixes are applied, such as removing them, reordering them, and +prettifying them. + +## Name Splitting + +Name splitting involves splitting an identifier into separate "tokens" and is handled by the `NameSplitter` class. These +tokens can refer to literal words (as identified by underscore/pascal case separations), but can also refer to groups of +numbers or capitalized letters. + +Note: The codebase is inconsistent when referring to tokens, usually calling them "words" or "fragments" instead. + +The goal of name splitting is to have a consistent representation of a name where each part of the name can be examined +individually. This is helpful when names differ by casing or by different types of separation. + +For example, `VkAccessFlags`, `vkCreateBuffer`, and `VK_MAX_MEMORY_HEAPS` effectively have the same shared prefix. + +For specifics on how this process works and the exact behaviors, it is best to refer to the `NameSplitter` source code +and the `NameSplitterTests` test cases. + +### Name Splitting - Notable Decisions + +#### Handling of Numbers + +Numbers are always split out as their own individual token. This is because this is easier to work with and consistent +than special casing when numbers should "stick" to preceding or proceeding tokens. + +For example: +- `2D` is split as `2_D` +- `R32` is split as `R_32` + +In these two cases, both inputs can be considered one English word, so it can be argued that the output should be the +same as the input. However, this means the name splitting code should have preferences for when numbers should "stick" +one way or the other. + +This gets even messier with names like `Image_2D_RGB16` or `Image2D_RGB16`. Although these exact names have not shown +up in native code, names like `SpvImageFormatR32ui` do in fact exist. + +Because the goal of name splitting is to have a consistent tokenized representation of the name, it can be argued +that it is safer to go for a more naive approach that does not attempt to group numbers with letters together at all. +In this case, a more naive approach means simpler code. It also means less potential surprises since the output is more +resistant to subtle changes in the input. + +## Name Prettification + +As hinted to previous, name prettification is the process of transforming an identifier to follow the Framework +Design Guidelines and is handled by the `NamePrettifier` class. + +This primarily involves pascal casing and the removal of underscore separators. Acronyms are also handled. By default, +acronyms of length 2 are preserved (matching the guidelines), while acronyms of greater lengths are pascal-cased. + +For example, "UI" is prettified as "UI" while "GUI" is prettified as "Gui". +Similarly, "GL" is prettified as "GL" while "EGL" is prettified as "Egl". + +Name prettification takes in a name "fragment" and outputs another fragment representing the prettified version of the +input. The input is first split using `NameSplitter` to get a tokenized representation of the name before being +processed. + +For specifics on how this process works and the exact behaviors, it is best to refer to the `NamePrettifier` source code +and the `NamePrettifierTests` test cases. + +### Name Prettification - Notable Decisions + +#### Output of Fully Capitalized Names + +By default, the `NamePrettifier` disallows outputs that are all caps. +For example, if `GL` is the output and `allowAllCaps` is the default of false, then `Gl` will be the actual output. + +This is to prevent fully capitalized member names, so the codebase typically overrides this behavior when dealing with +type names. This means the `GL` class remains as `GL`. + +#### Handling of Acronyms that contain Numbers + +An acronym includes the capital letters and the numbers immediately following those letters. + +For example: +- `2D` is split as `2_D`. There are 2 acronyms of length 1 here. +- `R32` is split as `R_32`. There is 1 acronym of length 3 here. + +Where this behavior matters is in the following case: +- `RG` is split as `RG` and is prettified as `RG`, however the `NamePrettifier` also disallows outputs that are fully + capitalized by default. This means `RG` is actually output as `Rg`. +- `RG32` is split as `RG_32`. Because this is an acronym of length 4, it is output as `Rg32`. + +Notably, this means that `RG` and `RG32` are consistently output as `Rg-`. + +In the code, this is implemented by merging number tokens with preceding letter tokens. + +For example: +- `2_D` is merged as `2_D`. +- `RG_32` is merged as `RG32`. + +This can be argued to be a hack, but simplifies acronym length calculations and continues to work with the code that +handles pascal casing, which simply uppercases the first character and lowercases the rest for each token. + +#### Acronym Indeterminate Inputs + +These refer to inputs that are fully uppercased, making it hard to tell whether the input is a standalone acronym or +simply written in screaming case. + +The current code handling this behavior was implemented back when the generator used a default long acronym threshold of +3 (and occasionally using 4*), which in turn was ported from the original Humanizer-based prettify implementation. +Therefore, the examples given in the code state a threshold of 4. + +\*4 was used for Khronos APIs as a best effort to preserve vendor suffixes (eg: `KHR`, `EXT`, `NV`, `QCOM`). This is no +longer necessary because the name affix system is now used to preserve these suffixes. + +This behavior notably is less noticeable with the long acronym threshold of 2, but still affects a few names, such as +the `GL` class. Without this, `GL` gets turned into `Gl` since the input is treated as screaming case. + +To learn more about this behavior, please refer to the comments in `NamePrettifier`. + +#### Handling of Consecutive Acronyms + +Consecutive acronyms are pascal-cased, if they both are candidates for being uppercased. + +For example, assuming a long acronym threshold of 4, `RGBA_ASTC` will be prettified as `RgbaAstc`, not `RGBAASTC`. +This is because the latter is much harder to read. + +However, if only one of the two consecutive acronyms is a candidate, for example, with a threshold of 2 and `RG_ASTC`, +the result will be `RGAstc`. + +#### Lowercase "x" between Numbers + +Consecutive numbers are separated by a lowercase "x". Furthermore, if a name already is in the format `2_X_2`, the "X" +will be lowercased. + +The use of the "x" is to ensure that numbers remain separated, especially because prettified names never contain +underscores, which is usually how consecutive numbers are separated in native code. + +The use of a *lowercase* x in particular is a stylistic choice and matches names like `System.Numerics.Matrix4x4`. + +## Name Affixes + +Name prefixes and suffixes are used commonly in both native code and in identifiers created by the SilkTouch generator. + +For example, in `VkPresentInfoKHR` from Vulkan, `Vk-` is a namespace prefix commonly used in C code, while `-KHR` is a +Khronos-style suffix denoting that the type belongs to the `KHR` family of extensions. + +In the generator, suffixes are usually used to denote names that are derived from other names or to prevent name +collisions. For example, `-Handle` is appended to handle types transformed by `TransformHandles`. This means that handle +types like `Buffer` are named as `BufferHandle` instead, thus reducing name collision risks with user-defined types. + +Because of the prevalence of affixes in both native and generated code, the name affix system was added so that names +can be annotated with information about what affixes have been identified or added to the name. This allows mods to +target transformations to a specific, known part of a name. + +Furthermore, because each category of affix can be identified by different mods, it keeps the complex affix +identification process localized to the mod that specializes in that area. For example, C-style namespace prefixes +are handled by `IdentifySharedPrefixes`. + +### Name Affixes - Metadata Format + +The name affixes for a corresponding identifier are stored as C# attributes declared on that identifier. This takes +advantage of the fact that the SilkTouch generator is designed such that mods primarily take Roslyn syntax trees as +input and return new syntax trees as output. + +For example, from the OpenGL bindings: +```cs +public enum InternalFormat +{ + [NameAffix("Prefix", "SharedPrefix", "GL")] + [NameAffix("Suffix", "KhronosVendor", "ARB")] + GL_RGBA32F_ARB = 34836, +} +``` + +In order, the parameters are: + +1. Affix type - Either "Prefix" or "Suffix". +2. Affix category - Used to identify the purpose or source of that affix. + - `PrettifyNames` can be configured to process different affix categories in different ways. For example, + shared prefixes can be removed by targeting the `SharedPrefix` category. +3. Affix value - The affix as it appears in the identifier. + - Note: Currently, affixes need to verbatim match the part of the identifier they represent. For example, stripping + `GL_RGBA32F_ARB` of the `GL-` prefix leads to `_RGBA32F_ARB`, while `GL_-` will lead to `RGBA32F_ARB`. Despite + this, the codebase is written to not include the underscore since it currently does not affect the output and is + arguably cleaner to avoid leading or trailing underscores in affix values. If this does prove to be a problem, + prefer updating the affix stripping code to be tolerant of extra underscores. + +These parameters are all strings for simplicity when parsing. Additionally, the order of the attributes is significant: +name affixes declared earlier in the attribute list represent name affixes closer to the inside of the name. + +However, as a user of the name affix system, the utilities provided by `NameAffixer` should provide everything necessary +for interacting with name affixes without interacting with the exact syntax node representation. + +### Name Affixes - Notable Interactions + +#### IdentifySharedPrefixes and Name Affixes + +`IdentifySharedPrefixes` strips affixes before identifying shared prefixes. Therefore, names like `ID3D12Device` will +appear as `D3D12Device` if the `I-` prefix is identified beforehand. + +#### Deferring Renames + +Renames that involve the addition of affixes can be done simply by adding the affix to the name, assuming that +`PrettifyNames` runs afterward. This is preferable because it avoids a project-wide symbol search to locate and update +where that identifier is used. + +This is because stripping affixes is tolerant of missing affixes and affixes reapplication will then add the newly +declared affix to the final name. + +For example, the `-Handle` suffix is added by `TransformHandles`. This leads to syntax that looks like: +```cs +[NameAffix("Suffix", "HandleType", "Handle")] +public struct Buffer; +``` + +Even though `Buffer` does not have a `-Handle` suffix, it will have it after `PrettifyNames` executes. This is assuming +that `PrettifyNames` is not configured to remove it. + +The removal of affixes can be done similarly, but will involve updating the generator config so that `PrettifyNames` +removes the affix. + +### Referenced Affixes + +Referenced affixes were added to handle compound names where part of the name is actually the name of another +identifier. This ensures that the "referenced" part of the name always matches the name being referenced, in other +words, changes to the referenced name is "synchronized" to the name referencing it (more on this later). + +This occurs primarily in types added to the bindings by Silk. For example: + +- **Nested types** - Nested types are extracted by `ExtractNestedTyping` to be non-nested types. These types have the name + of their parent type plus their original name (for nested types that have proper names in the native code) or the name + of their parent type plus the name of the field that uses them (eg: for `InlineArray` types). + - Example: `GamepadBinding`, `GamepadBindingInput`, and `GamepadBindingInputAxis` in the SDL bindings. The latter are + nested structs. + - Example: `PerformanceCounterDescriptionARM` and `PerformanceCounterDescriptionARMName` in the Vulkan bindings. The + latter is an inline array used by the field, `PerformanceCounterDescriptionARM.Name`. + +- **Derived types** - Derived types refer to types that are generated based on another type (not to be confused with + inheritance). At time of writing, this only refers to function pointer types for which Silk generates a function + pointer struct and a corresponding delegate type. The delegate type has the `-Delegate` suffix appended to it. + - Example: `DebugReportCallbackEXT` and `DebugReportCallbackEXTDelegate` in the Vulkan bindings. + +In other words, referenced affixes are most helpful when dealing with types that are the logical extensions of other +types. + +Most notably, handle type suffixes do not fall into the above categorization. + +For example: `PipelineBinaryHandleKHR` in the Vulkan bindings is not an extension of `PipelineBinaryKHR`, it *is* the +type, just renamed to avoid naming collisions. + +Going back to the idea of synchronizing changes, if `GamepadBinding` was to be renamed using an override, the referenced +affix system ensures that `GamepadBindingInput` and `GamepadBindingInputAxis` are renamed correspondingly. + +Similarly, it ensures that if an affix is configured to be moved to the end of the referenced name, the affix only moves +to the end of the name it was originally declared on. This can be seen in `PerformanceCounterDescriptionARMName` and +`PipelineBinaryHandleKHR`. `ARM` is a Khronos vendor suffix for `PerformanceCounterDescriptionARM`, so it only moves to +the end of that name; however, `KHR` is a vendor suffix for `PipelineBinaryHandleKHR` as a whole. + +Side note: Another benefit of referenced affixes is that it ensures that derived types show up when typing the base +type's name in the IDE. For example, if vendor suffixes always moved to the end of the name, +`PerformanceCounterDescriptionARMName` would become `PerformanceCounterDescriptionNameARM` and would not show up when +autocompleting `PerformanceCounterDescriptionARM`. + +### Referenced Affixes - Metadata Format + +Referenced affixes work the exact same as normal name affixes, but take advantage of C#'s nameof syntax. + +For example, from the SDL bindings: +```cs +public struct GamepadBinding; + +[NameAffix("Prefix", "NestedStructParent", nameof(GamepadBinding))] +public struct GamepadBindingInput; + +[NameAffix("Prefix", "NestedStructParent", nameof(GamepadBindingInput))] +public struct GamepadBindingInputAxis; +``` + +Limitation: Only simple references are allowed because references are resolved manually by `PrettifyNames` and not by +Roslyn. For example, `nameof(GamepadBinding.Member)` will not work because member access expressions are not handled. +Currently, only identifiers that exist in the current scope or parent scope can be referenced. That said, this should be +enough for most use cases. + +## Symbol-based Renamer + +The renamer exists as `NameUtils.RenameAllAsync()` and uses Roslyn symbols to determine whether an identifier needs to +be replaced. + +The renamer has gone through several iterations, mainly due to performance reasons. + +Previously, it used `SymbolFinder.FindReferencesAsync()`, which was replaced since it was far too slow for bigger APIs +like the Microsoft bindings. `FindReferencesAsync` was not designed for mass replacement of all identifiers in a project +and thus suffered an `O(n^2)` scaling (where `n` is the size of the project) since it scanned the entire project for +each symbol replaced. + +The current implementation, which is the `LocationTransformationUtils` class in the codebase, uses a +`CSharpSyntaxRewriter` that visits every syntax node in the project and looks up symbols related to the node before +deciding to replace it. This changes the scaling to `O(n)` with symbol lookup being the primary bottleneck. Symbol +lookup is optimized by checking if the name of the identifier matches a name of the symbols to rename, which doubles the +speed of renaming when it comes to the Vulkan bindings. + +The reason the new renamer is part of the "location transformation" code is because the renamer has also been +generalized to work with any transformation that needs to modify all references of a symbol. This notably was designed +back when `TransformHandles` needed to simultaneously rename all references to a handle type and decrease the pointer +dimension of the references to the type by one (eg: `Buffer**` becomes `BufferHandle*`). This bulk modification ensures +that symbol lookup only needs to occur once. + +Side note: Arguably, "reference transformation" better describes this area of the codebase, but the name originally came +from the `ReferenceLocation` type returned by `SymbolFinder.FindReferencesAsync()`. diff --git a/docs/for-contributors/Generator/using-the-generator.md b/docs/for-contributors/Generator/using-the-generator.md new file mode 100644 index 0000000000..2addf8bd49 --- /dev/null +++ b/docs/for-contributors/Generator/using-the-generator.md @@ -0,0 +1,183 @@ +(This needs clean up before merging. This was originally meant to be a Discord message, but I figured it's a good start to having proper docs.) + +Okay, so short guide (by Exanite): + +Note that this is from my perspective as a new contributor/maintainer. I'm guessing at how some of this stuff works. :sweat_smile: + +The way that Silk 3 works is by taking the output of ClangSharpPInvokeGenerator and modifying the output with a set of mods. +These mods do things such as renaming identifiers, creating types such as handle structs or enums, and adding method overloads. + +**Do note that Silk 3 is in heavy development and things can change without warning.** +This is probably the case until we're a few previews in. + +**Also note that only C bindings are supported right now. COM will be available later.** + +## Generator overview + +There are two main things to configure: +1. Silk 3 - This is the [`generator.json`](https://github.com/dotnet/Silk.NET/blob/develop/3.0/generator.json) file. +2. ClangSharpPInvokeGenerator - This is the [`eng/silktouch`](https://github.com/dotnet/Silk.NET/tree/develop/3.0/eng/silktouch) folder. + +Both are organized by library. + +I suggest referencing the SDL configuration since it best represents your average C library. +Most options there should be applicable after you replace the SDL specific types/paths/etc. + +### `generator.json` + +This defines which mods to run. +- `AddIncludes` adds system header files. You want this. +- `ClangScraper` runs ClangSharpPInvokeGenerator. Only including this is equivalent to running ClangSharpPInvokeGenerator directly. +- The rest do a bunch of other transformations that I won't cover here. Ask if you're interested please. + +One way to learn about the mods and debug them is to add them one by one. +Mods run in the order you define them and work off the output of each other. + +However, you can probably get by just copying the mod order from SDL verbatim. + +Things to note: +- I'm unfamiliar with the test project config. You probably can ignore it. +- Bool types are only transformed for functions. Bool types in structs will likely be left as`int` or similar. + +### `eng/silktouch` + +This folder contains a bunch of `.rsp` files, which hold command line arguments for ClangSharpPInvokeGenerator. + +> To read more about ClangSharpPInvokeGenerator's command line arguments, I recommend installing the tool and using its help options. +> +> ```sh +> dotnet tool install --global ClangSharpPInvokeGenerator +> ClangSharpPInvokeGenerator --help +> ClangSharpPInvokeGenerator -c help +> ``` + +These rsp files can import other rsp files using the `@path` syntax. +Eg: `@../settings.rsp` + +Note that these paths are relative to the `generate.rsp` file (I think... or at least the folder containing that file). + +`@../../remap-stdint.rsp` is my addition that ensures that stdint types behave consistently between Windows and Linux. + +This is the general structure of the `eng/silktouch` folder: + +``` +eng +- silktouch + - opengl <-- This level contains folders per library + - glcompat <-- This level contains folders for each "profile" (I think that's the term) for each variant of the library + - glcore + - gles1 + - gles2 + - sdl + - SDL3 +``` + +You likely don't need to worry about profiles, so we'll just keep focusing on the SDL case. + +This is the structure of the SDL rsps. +Note that you don't necessarily have to structure it this way. + +``` +eng +- silktouch + - sdl + - SDL3 + - generate.rsp <-- The main settings file + - header.txt + - sdl-SDL.h <-- Hand written header file that includes the relevant headers of the library you want to bind + - remap.rsp + - settings.rsp <-- Shared settings for all profiles. Technically can be merged into generate.rsp. +``` + +Let's take a look at the `sdl-SDL.h` file and the `generate.rsp` and `settings.rsp` files. +I'll only include the important parts of the config here. + +`sdl-SDL.h`: +```h +#include +#include +#include +``` + +`generate.rsp`: +```rsp +@../settings.rsp +@../remap.rsp +--exclude +SDL_SetX11EventHook +SDL_SetWindowsMessageHook +SDL_FILE +SDL_LINE +--file +sdl-SDL.h +--methodClassName +Sdl +--namespace +Silk.NET.SDL +--output +../../../../sources/SDL/SDL3 +--traverse +../../../submodules/sdl/include/SDL3/SDL_assert.h +../../../submodules/sdl/include/SDL3/SDL_atomic.h +../../../submodules/sdl/include/SDL3/SDL_audio.h +``` + +`settings.rsp`: +```rsp +@../../common.rsp +--define-macro +TODO_DEFINE_MACROS=HERE +--headerFile +header.txt +--include-directory +../../../submodules/sdl/include +--with-callconv +*=Winapi +--with-librarypath +*=SDL3 +``` + +#### Relevant options from `generate.rsp`: + +`--file` specifies the header file that we first look through. +`--traverse` specifies which header files actually contribute towards the output. (Not sure if you can glob or similar here) + +This separation is because while we need certain header files such as the system headers to compile the library, we don't want to include the system headers as part of our generated bindings. + +`--output` should point to the same `Jobs.JOB_NAME.SourceProject` path you defined in `generator.json`. + +`--methodClassName` specifies which C# class contains the generated methods/constants. +`--namespace` specifies the C# namespace of the generated files. + +`--exclude` allows you exclude types/functions/constants from the output. Usually these are things that aren't useful, don't generate correctly, or are platform-specific. + +#### Relevant options from `settings.rsp`: + +`--headerFile` specifies the header file appended to the top of every generated file. + +`--include-directory` specifies the include directories. This affects all of the headers included, such as in `sdl-SDL.h`. + +`--with-librarypath` is the name of the native library without prefixes/suffixes. If the library name differs outside of the usual `lib` prefix or `.dll`/`.so`/`.dylib` suffixes, the way to handle this is to add `UseAlternativeName` in the generated bindings. An example with Vulkan can be found at `sources/Vulkan/Vulkan/Vk.cs`. + +```cs +static Vk() +{ + LoaderInterface.RegisterHook(Assembly.GetExecutingAssembly()); + LoaderInterface.RegisterAlternativeName("vulkan", "vulkan-1"); + LoaderInterface.RegisterAlternativeName("vulkan", "MoltenVK"); +} +``` + +### Generated bindings output + +All generated binding will be output to the `Jobs.JOB_NAME.SourceProject` path you defined in `generator.json`. + +These generated files all have the `.gen.cs` suffix and most of them are partial type declarations. +This means by creating a similarly named `.cs` file and using the `partial` C# keyword, you can add to the type. + +Do not modify the `.gen.cs` files manually since rerunning the generator will overwrite those changes. + +### Packing the generated bindings + +Haven't done this myself so I'll leave this section as WIP. +I imagine that `dotnet pack` or similar will just work though. diff --git a/sources/SilkTouch/SilkTouch/Mods/Common/ModUtils.cs b/sources/SilkTouch/SilkTouch/Mods/Common/ModUtils.cs index 91dcf7c06b..a8a3f03a30 100644 --- a/sources/SilkTouch/SilkTouch/Mods/Common/ModUtils.cs +++ b/sources/SilkTouch/SilkTouch/Mods/Common/ModUtils.cs @@ -160,17 +160,24 @@ public static string GetMethodDiscriminator(BaseParameterSyntax param) => GetMethodDiscriminator(param.Modifiers, param.Type); /// - /// Gets the relative path for this document. + /// Gets the path relative to the document's project path for the specified document. /// /// The document. /// The relative path. public static string? RelativePath(this Document doc) { - if ( - doc.FilePath is null - || doc.Project.FilePath is null - || Path.GetDirectoryName(doc.Project.FilePath) is not { Length: > 0 } dir - ) + if (doc.FilePath is null) + { + return default; + } + + // Handle projects with no path by simply returning the document path + if (doc.Project.FilePath is null) + { + return doc.FilePath; + } + + if (Path.GetDirectoryName(doc.Project.FilePath) is not { Length: > 0 } dir) { return default; } diff --git a/sources/SilkTouch/SilkTouch/Mods/ExtractNestedTyping.cs b/sources/SilkTouch/SilkTouch/Mods/ExtractNestedTyping.cs index 4d5ff5fb28..273eee7491 100644 --- a/sources/SilkTouch/SilkTouch/Mods/ExtractNestedTyping.cs +++ b/sources/SilkTouch/SilkTouch/Mods/ExtractNestedTyping.cs @@ -17,7 +17,7 @@ namespace Silk.NET.SilkTouch.Mods; /// /// /// Replacing function pointers identified by their s with delegates and -/// Pfn-prefixed structures. +/// function pointer structs. /// /// /// Moving constants into their respective enums. These constants are identified by checking for an enum with diff --git a/sources/SilkTouch/SilkTouch/Mods/MarkNativeNames.cs b/sources/SilkTouch/SilkTouch/Mods/MarkNativeNames.cs index 9f8aa554d1..e47c0ae8b7 100644 --- a/sources/SilkTouch/SilkTouch/Mods/MarkNativeNames.cs +++ b/sources/SilkTouch/SilkTouch/Mods/MarkNativeNames.cs @@ -11,7 +11,7 @@ namespace Silk.NET.SilkTouch.Mods; /// /// /// This mod is currently kept pretty dumb and just applies [NativeName] attributes to almost everything. -/// Syntax nodes not output by ClangSharp are intentionally not processed. +/// Syntax nodes not output by ClangScraper are intentionally not processed. /// This mod is best placed directly after ClangScraper. /// public class MarkNativeNames : IMod diff --git a/sources/SilkTouch/SilkTouch/Mods/TransformHandles.cs b/sources/SilkTouch/SilkTouch/Mods/TransformHandles.cs index c2011b0ba4..b74ffba117 100644 --- a/sources/SilkTouch/SilkTouch/Mods/TransformHandles.cs +++ b/sources/SilkTouch/SilkTouch/Mods/TransformHandles.cs @@ -15,16 +15,19 @@ namespace Silk.NET.SilkTouch.Mods; /// -/// Identifies handle types by finding pointers to empty structs or missing types. +/// Identifies handle types by finding pointers to empty structs. /// In general, a handle type is a struct that wraps an underlying opaque pointer (or some other underlying value). /// These handle types are then transformed by making the struct wrap the underlying pointer and /// reducing the dimension of pointers referencing that handle type by one. /// /// /// Given an empty struct, struct VkBuffer, and all usages of that struct are through a pointer, -/// VkBuffer*, usages of that pointer will be replaced by VkBufferHandle. For a 2-dimensional pointer, -/// VkBuffer**, the resulting replacement is VkBufferHandle*. +/// VkBuffer*, usages of that pointer will be replaced by VkBuffer. For a 2-dimensional pointer, +/// VkBuffer**, the resulting replacement is VkBuffer*. /// +/// +/// The `Handle` suffix is not applied until executes. +/// [ModConfiguration] public class TransformHandles( IOptionsSnapshot config, diff --git a/tests/SilkTouch/SilkTouch/ExtractHandlesTests.SuccessfullyExtractsHandleType.verified.txt b/tests/SilkTouch/SilkTouch/ExtractHandlesTests.SuccessfullyExtractsHandleType.verified.txt new file mode 100644 index 0000000000..b8f9cccb02 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractHandlesTests.SuccessfullyExtractsHandleType.verified.txt @@ -0,0 +1,3 @@ +public unsafe partial struct VkInstance_T +{ +} \ No newline at end of file diff --git a/tests/SilkTouch/SilkTouch/ExtractHandlesTests.cs b/tests/SilkTouch/SilkTouch/ExtractHandlesTests.cs new file mode 100644 index 0000000000..27a4aac718 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractHandlesTests.cs @@ -0,0 +1,56 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.CodeAnalysis; +using Microsoft.Extensions.Logging.Abstractions; +using Silk.NET.SilkTouch.Mods; + +namespace Silk.NET.SilkTouch.UnitTests; + +public class ExtractHandlesTests +{ + static ExtractHandlesTests() + { + if (!VerifyDiffPlex.Initialized) + { + VerifyDiffPlex.Initialize(); + } + } + + [Test] + public async Task SuccessfullyExtractsHandleType() + { + var inputDocName = "Vk.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public struct VkAllocationCallbacks; + public struct VkInstanceCreateInfo; + + public class Vk + { + public static extern VkResult vkCreateInstance( + VkInstanceCreateInfo* pCreateInfo, + VkAllocationCallbacks* pAllocator, + VkInstance_T** pInstance + ); + } + """ + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractHandles = new ExtractHandles(NullLogger.Instance); + + await extractHandles.ExecuteAsync(context); + + // There should be an empty struct named VkInstance_T in a new file + var result = await context + .SourceProject.Documents.Single(x => x.Name != inputDocName) + .GetSyntaxRootAsync(); + await Verify(result!.NormalizeWhitespace().ToString()); + } +} diff --git a/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_MethodParameter.verified.txt b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_MethodParameter.verified.txt new file mode 100644 index 0000000000..e2a2ccf652 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_MethodParameter.verified.txt @@ -0,0 +1,20 @@ +// SDL_BlendMode.gen.cs +public enum SDL_BlendMode : uint +{ + SDL_BLENDMODE_NONE = 0x00000000U, + SDL_BLENDMODE_BLEND = 0x00000001U, + SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U, + SDL_BLENDMODE_ADD = 0x00000002U, + SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U, + SDL_BLENDMODE_MOD = 0x00000004U, + SDL_BLENDMODE_MUL = 0x00000008U, + SDL_BLENDMODE_INVALID = 0x7FFFFFFFU +} + +// Sdl.gen.cs +public unsafe partial struct Sdl +{ + [DllImport("SDL3", ExactSpelling = true)] + [return: NativeTypeName("bool")] + public static extern byte SDL_SetSurfaceBlendMode(SDL_Surface* surface, [NativeTypeName("SDL_BlendMode")] SDL_BlendMode blendMode); +} \ No newline at end of file diff --git a/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_Pointer.verified.txt b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_Pointer.verified.txt new file mode 100644 index 0000000000..3ff556a066 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsCStyleEnumConstants_Pointer.verified.txt @@ -0,0 +1,20 @@ +// SDL_BlendMode.gen.cs +public enum SDL_BlendMode : uint +{ + SDL_BLENDMODE_NONE = 0x00000000U, + SDL_BLENDMODE_BLEND = 0x00000001U, + SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U, + SDL_BLENDMODE_ADD = 0x00000002U, + SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U, + SDL_BLENDMODE_MOD = 0x00000004U, + SDL_BLENDMODE_MUL = 0x00000008U, + SDL_BLENDMODE_INVALID = 0x7FFFFFFFU +} + +// Sdl.gen.cs +public unsafe partial struct Sdl +{ + [DllImport("SDL3", ExactSpelling = true)] + [return: NativeTypeName("bool")] + public static extern byte SDL_GetSurfaceBlendMode(SDL_Surface* surface, [NativeTypeName("SDL_BlendMode *")] SDL_BlendMode* blendMode); +} diff --git a/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsFunctionPointer.verified.txt b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsFunctionPointer.verified.txt new file mode 100644 index 0000000000..bbbbe8facc --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsFunctionPointer.verified.txt @@ -0,0 +1,26 @@ +// PFN_vkDebugReportCallbackEXT.gen.cs +[NativeName("PFN_vkDebugReportCallbackEXT")] +public unsafe readonly struct PFN_vkDebugReportCallbackEXT : IDisposable +{ + private readonly void* _pointer; + public delegate* unmanaged Handle => (delegate* unmanaged )_pointer; + + public PFN_vkDebugReportCallbackEXT(delegate* unmanaged ptr) => _pointer = ptr; + public PFN_vkDebugReportCallbackEXT(PFN_vkDebugReportCallbackEXTDelegate proc) => _pointer = SilkMarshal.DelegateToPtr(proc); + public void Dispose() => SilkMarshal.Free(_pointer); + public static implicit operator PFN_vkDebugReportCallbackEXT(delegate* unmanaged pfn) => new(pfn); + public static implicit operator delegate* unmanaged (PFN_vkDebugReportCallbackEXT pfn) => (delegate* unmanaged )pfn._pointer; +} + +// PFN_vkDebugReportCallbackEXTDelegate.gen.cs +[NativeName("PFN_vkDebugReportCallbackEXT")] +[NameAffix("Prefix", "FunctionPointerParent", nameof(PFN_vkDebugReportCallbackEXT))] +[NameAffix("Suffix", "FunctionPointerDelegateType", "Delegate")] +public unsafe delegate uint PFN_vkDebugReportCallbackEXTDelegate(uint arg0, VkDebugReportObjectTypeEXT arg1, ulong arg2, nuint arg3, int arg4, sbyte* arg5, sbyte* arg6, void* arg7); + +// VkDebugReportCallbackCreateInfoEXT.gen.cs +public unsafe partial struct VkDebugReportCallbackCreateInfoEXT +{ + [NativeTypeName("PFN_vkDebugReportCallbackEXT")] + public PFN_vkDebugReportCallbackEXT pfnCallback; +} diff --git a/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsNestedInlineArray.verified.txt b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsNestedInlineArray.verified.txt new file mode 100644 index 0000000000..42e315ede4 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.SuccessfullyExtractsNestedInlineArray.verified.txt @@ -0,0 +1,16 @@ +// VkPerformanceCounterDescriptionARM.gen.cs +namespace Silk.NET.Vulkan; +public struct VkPerformanceCounterDescriptionARM +{ + [NativeTypeName("char[256]")] + public VkPerformanceCounterDescriptionARMname name; +} + +// VkPerformanceCounterDescriptionARMname.gen.cs +namespace Silk.NET.Vulkan; +[InlineArray(256)] +[NameAffix("Prefix", "NestedStructParent", nameof(VkPerformanceCounterDescriptionARM))] +public struct VkPerformanceCounterDescriptionARMname +{ + public sbyte e0; +} diff --git a/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.cs b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.cs new file mode 100644 index 0000000000..1fcd751155 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/ExtractNestedTypingTests.cs @@ -0,0 +1,331 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.CodeAnalysis; +using Microsoft.Extensions.Logging.Abstractions; +using Silk.NET.SilkTouch.Mods; + +namespace Silk.NET.SilkTouch.UnitTests; + +public class ExtractNestedTypingTests +{ + static ExtractNestedTypingTests() + { + if (!VerifyDiffPlex.Initialized) + { + VerifyDiffPlex.Initialize(); + } + } + + [Test] + public async Task SuccessfullyExtractsNestedInlineArray() + { + var inputDocName = "VkPerformanceCounterDescriptionARM.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + namespace Silk.NET.Vulkan; + + public struct VkPerformanceCounterDescriptionARM + { + [NativeTypeName("char[256]")] + public _name_e__FixedBuffer name; + + [InlineArray(256)] + public struct _name_e__FixedBuffer + { + public sbyte e0; + } + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"Vulkan/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The nested struct should be extracted and named as VkPerformanceCounterDescriptionARMname + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } + + [Test] + public async Task SuccessfullyExtractsFunctionPointer() + { + var inputDocName = "VkDebugReportCallbackCreateInfoEXT.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public unsafe partial struct VkDebugReportCallbackCreateInfoEXT + { + [NativeTypeName("PFN_vkDebugReportCallbackEXT")] + public delegate* unmanaged< + uint, + VkDebugReportObjectTypeEXT, + ulong, + nuint, + int, + sbyte*, + sbyte*, + void*, + uint> pfnCallback; + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"Vulkan/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The function pointer should be extracted as both a struct and a delegate + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } + + [Test] + public async Task SuccessfullyExtractsCStyleEnumConstants_Field() + { + var inputDocName = "Sdl.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public unsafe partial struct Sdl + { + [NativeTypeName("#define SDL_BLENDMODE_NONE 0x00000000u")] + public const uint SDL_BLENDMODE_NONE = 0x00000000U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND 0x00000001u")] + public const uint SDL_BLENDMODE_BLEND = 0x00000001U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND_PREMULTIPLIED 0x00000010u")] + public const uint SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD 0x00000002u")] + public const uint SDL_BLENDMODE_ADD = 0x00000002U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD_PREMULTIPLIED 0x00000020u")] + public const uint SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U; + + [NativeTypeName("#define SDL_BLENDMODE_MOD 0x00000004u")] + public const uint SDL_BLENDMODE_MOD = 0x00000004U; + + [NativeTypeName("#define SDL_BLENDMODE_MUL 0x00000008u")] + public const uint SDL_BLENDMODE_MUL = 0x00000008U; + + [NativeTypeName("#define SDL_BLENDMODE_INVALID 0x7FFFFFFFu")] + public const uint SDL_BLENDMODE_INVALID = 0x7FFFFFFFU; + } + + public class Test + { + [NativeTypeName("SDL_BlendMode")] + public uint Blend; + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"SDL3/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The constants should have been moved from the Sdl clas to the SDL_BlendMode enum + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } + + [Test] + public async Task SuccessfullyExtractsCStyleEnumConstants_MethodParameter() + { + var inputDocName = "Sdl.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public unsafe partial struct Sdl + { + [NativeTypeName("#define SDL_BLENDMODE_NONE 0x00000000u")] + public const uint SDL_BLENDMODE_NONE = 0x00000000U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND 0x00000001u")] + public const uint SDL_BLENDMODE_BLEND = 0x00000001U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND_PREMULTIPLIED 0x00000010u")] + public const uint SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD 0x00000002u")] + public const uint SDL_BLENDMODE_ADD = 0x00000002U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD_PREMULTIPLIED 0x00000020u")] + public const uint SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U; + + [NativeTypeName("#define SDL_BLENDMODE_MOD 0x00000004u")] + public const uint SDL_BLENDMODE_MOD = 0x00000004U; + + [NativeTypeName("#define SDL_BLENDMODE_MUL 0x00000008u")] + public const uint SDL_BLENDMODE_MUL = 0x00000008U; + + [NativeTypeName("#define SDL_BLENDMODE_INVALID 0x7FFFFFFFu")] + public const uint SDL_BLENDMODE_INVALID = 0x7FFFFFFFU; + + [DllImport("SDL3", ExactSpelling = true)] + [return: NativeTypeName("bool")] + public static extern byte SDL_SetSurfaceBlendMode( + SDL_Surface* surface, + [NativeTypeName("SDL_BlendMode")] uint blendMode + ); + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"SDL3/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The constants should have been moved from the Sdl clas to the SDL_BlendMode enum + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } + + [Test] + public async Task SuccessfullyExtractsCStyleEnumConstants_Pointer() + { + var inputDocName = "Sdl.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public unsafe partial struct Sdl + { + [NativeTypeName("#define SDL_BLENDMODE_NONE 0x00000000u")] + public const uint SDL_BLENDMODE_NONE = 0x00000000U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND 0x00000001u")] + public const uint SDL_BLENDMODE_BLEND = 0x00000001U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND_PREMULTIPLIED 0x00000010u")] + public const uint SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD 0x00000002u")] + public const uint SDL_BLENDMODE_ADD = 0x00000002U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD_PREMULTIPLIED 0x00000020u")] + public const uint SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U; + + [NativeTypeName("#define SDL_BLENDMODE_MOD 0x00000004u")] + public const uint SDL_BLENDMODE_MOD = 0x00000004U; + + [NativeTypeName("#define SDL_BLENDMODE_MUL 0x00000008u")] + public const uint SDL_BLENDMODE_MUL = 0x00000008U; + + [NativeTypeName("#define SDL_BLENDMODE_INVALID 0x7FFFFFFFu")] + public const uint SDL_BLENDMODE_INVALID = 0x7FFFFFFFU; + + [DllImport("SDL3", ExactSpelling = true)] + [return: NativeTypeName("bool")] + public static extern byte SDL_GetSurfaceBlendMode( + SDL_Surface* surface, + [NativeTypeName("SDL_BlendMode *")] uint* blendMode + ); + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"SDL3/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The constants should have been moved from the Sdl clas to the SDL_BlendMode enum + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } + + [Test] + public async Task SuccessfullyExtractsCStyleEnumConstants_ReturnType() + { + var inputDocName = "Sdl.gen.cs"; + var project = TestUtils + .CreateTestProject() + .AddDocument( + inputDocName, + """ + public unsafe partial struct Sdl + { + [NativeTypeName("#define SDL_BLENDMODE_NONE 0x00000000u")] + public const uint SDL_BLENDMODE_NONE = 0x00000000U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND 0x00000001u")] + public const uint SDL_BLENDMODE_BLEND = 0x00000001U; + + [NativeTypeName("#define SDL_BLENDMODE_BLEND_PREMULTIPLIED 0x00000010u")] + public const uint SDL_BLENDMODE_BLEND_PREMULTIPLIED = 0x00000010U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD 0x00000002u")] + public const uint SDL_BLENDMODE_ADD = 0x00000002U; + + [NativeTypeName("#define SDL_BLENDMODE_ADD_PREMULTIPLIED 0x00000020u")] + public const uint SDL_BLENDMODE_ADD_PREMULTIPLIED = 0x00000020U; + + [NativeTypeName("#define SDL_BLENDMODE_MOD 0x00000004u")] + public const uint SDL_BLENDMODE_MOD = 0x00000004U; + + [NativeTypeName("#define SDL_BLENDMODE_MUL 0x00000008u")] + public const uint SDL_BLENDMODE_MUL = 0x00000008U; + + [NativeTypeName("#define SDL_BLENDMODE_INVALID 0x7FFFFFFFu")] + public const uint SDL_BLENDMODE_INVALID = 0x7FFFFFFFU; + + [DllImport("SDL3", ExactSpelling = true)] + [return: NativeTypeName("SDL_BlendMode")] + public static extern uint SDL_ComposeCustomBlendMode( + SDL_BlendFactor srcColorFactor, + SDL_BlendFactor dstColorFactor, + SDL_BlendOperation colorOperation, + SDL_BlendFactor srcAlphaFactor, + SDL_BlendFactor dstAlphaFactor, + SDL_BlendOperation alphaOperation + ); + } + """, + // ExtractNestedTyping requires the file path to be set and that the document is under a subfolder + filePath: $"SDL3/{inputDocName}" + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var extractNestedTyping = new ExtractNestedTyping(NullLogger.Instance); + + await extractNestedTyping.ExecuteAsync(context); + + // The constants should have been moved from the Sdl clas to the SDL_BlendMode enum + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } +} diff --git a/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.SuccessfullyInterceptsRequestedFunction.verified.txt b/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.SuccessfullyInterceptsRequestedFunction.verified.txt new file mode 100644 index 0000000000..f956110e13 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.SuccessfullyInterceptsRequestedFunction.verified.txt @@ -0,0 +1,11 @@ +// Vk.gen.cs +public class Vk +{ + [DllImport("vulkan", ExactSpelling = true, EntryPoint = "vkCreateInstance")] + [NameAffix("Suffix", "InterceptedFunction", "Internal")] + private static extern VkResult vkCreateInstanceInternal(VkInstanceCreateInfo* pCreateInfo, VkAllocationCallbacks* pAllocator, VkInstance_T** pInstance); + [DllImport("vulkan", ExactSpelling = true)] + public static extern VkResult vkCreateDevice(VkPhysicalDevice_T physicalDevice, VkDeviceCreateInfo* pCreateInfo, VkAllocationCallbacks* pAllocator, VkDevice_T* pDevice); + [NativeFunction("vulkan", EntryPoint = "vkCreateInstance")] + public static partial VkResult vkCreateInstance(VkInstanceCreateInfo* pCreateInfo, VkAllocationCallbacks* pAllocator, VkInstance_T** pInstance); +} diff --git a/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.cs b/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.cs new file mode 100644 index 0000000000..b1319b2a46 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/InterceptNativeFunctionsTests.cs @@ -0,0 +1,66 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.CodeAnalysis; +using Microsoft.Extensions.Logging.Abstractions; +using Silk.NET.SilkTouch.Mods; + +namespace Silk.NET.SilkTouch.UnitTests; + +public class InterceptNativeFunctionsTests +{ + static InterceptNativeFunctionsTests() + { + if (!VerifyDiffPlex.Initialized) + { + VerifyDiffPlex.Initialize(); + } + } + + [Test] + public async Task SuccessfullyInterceptsRequestedFunction() + { + var project = TestUtils + .CreateTestProject() + .AddDocument( + "Vk.gen.cs", + """ + public class Vk + { + [DllImport("vulkan", ExactSpelling = true)] + public static extern VkResult vkCreateInstance( + VkInstanceCreateInfo* pCreateInfo, + VkAllocationCallbacks* pAllocator, + VkInstance_T** pInstance + ); + + [DllImport("vulkan", ExactSpelling = true)] + public static extern VkResult vkCreateDevice( + VkPhysicalDevice_T physicalDevice, + VkDeviceCreateInfo* pCreateInfo, + VkAllocationCallbacks* pAllocator, + VkDevice_T* pDevice + ); + } + """ + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var interceptNativeFunctions = new InterceptNativeFunctions( + new DummyOptions( + new InterceptNativeFunctions.Configuration() + { + NativeFunctionNames = ["vkCreateInstance"], + } + ) + ); + + await interceptNativeFunctions.ExecuteAsync(context); + + // vkCreateInstance should be intercepted by suffixing the original with -Internal + // and by adding a replacement that makes use of the partial keyword + await TestUtils.VerifyDocumentsAsync(context.SourceProject.Documents); + } +} diff --git a/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared.verified.txt b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared.verified.txt new file mode 100644 index 0000000000..d943cda3a8 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared.verified.txt @@ -0,0 +1,5 @@ +[NameAffix("Prefix", "SharedPrefix", "D3D12")] +public struct D3D12_BUFFER_BARRIER; +[NameAffix("Prefix", "SharedPrefix", "D3D12")] +[NameAffix("Prefix", "Interface", "I")] +public struct ID3D12Device; \ No newline at end of file diff --git a/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared_WithHint.verified.txt b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared_WithHint.verified.txt new file mode 100644 index 0000000000..fffbd6d7f1 --- /dev/null +++ b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.IdentifiesSharedPrefix_WhenPrefixesDeclared_WithHint.verified.txt @@ -0,0 +1,3 @@ +[NameAffix("Prefix", "SharedPrefix", "D3D12")] +[NameAffix("Prefix", "Interface", "I")] +public struct ID3D12Device; \ No newline at end of file diff --git a/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.cs b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.cs index c87e545903..2052076044 100644 --- a/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.cs +++ b/tests/SilkTouch/SilkTouch/Naming/IdentifySharedPrefixesTests.cs @@ -92,7 +92,7 @@ public async Task IdentifiesSharedPrefixGlfw(string? hint) var project = TestUtils .CreateTestProject() .AddDocument( - "VocalMorpherPhoneme.gen.cs", + "Glfw.gen.cs", """ public struct Glfw; public struct GLFWallocator; @@ -185,13 +185,73 @@ public enum OcclusionQueryParameterNameNV await identifySharedPrefixes.ExecuteAsync(context); - // The declaration of the 2 NV member suffixes should make PrettifyNames trim less of the member name + // The declaration of the 2 NV member suffixes should make IdentifySharedPrefixes identify less of the member name as the shared prefix // IdentifySharedPrefixes should only use the unaffixed name for prefix identification // The shared prefix should be "GL_PIXEL" var result = await context.SourceProject.Documents.First().GetSyntaxRootAsync(); await Verify(result!.NormalizeWhitespace().ToString()); } + [Test] + public async Task IdentifiesSharedPrefix_WhenPrefixesDeclared() + { + var project = TestUtils + .CreateTestProject() + .AddDocument( + "D3D12.gen.cs", + """ + public struct D3D12_BUFFER_BARRIER; + + [NameAffix("Prefix", "Interface", "I")] + public struct ID3D12Device; + """ + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var identifySharedPrefixes = new IdentifySharedPrefixes( + new DummyOptions( + new IdentifySharedPrefixes.Configuration() + ) + ); + + await identifySharedPrefixes.ExecuteAsync(context); + + // The declaration of the I- prefix should lead to "D3D12" being identified as the shared prefix + var result = await context.SourceProject.Documents.First().GetSyntaxRootAsync(); + await Verify(result!.NormalizeWhitespace().ToString()); + } + + [Test] + public async Task IdentifiesSharedPrefix_WhenPrefixesDeclared_WithHint() + { + var project = TestUtils + .CreateTestProject() + .AddDocument( + "D3D12.gen.cs", + """ + [NameAffix("Prefix", "Interface", "I")] + public struct ID3D12Device; + """ + ) + .Project; + + var context = new DummyModContext() { SourceProject = project }; + + var identifySharedPrefixes = new IdentifySharedPrefixes( + new DummyOptions( + new IdentifySharedPrefixes.Configuration() { GlobalPrefixHints = ["D3D12"] } + ) + ); + + await identifySharedPrefixes.ExecuteAsync(context); + + // The declaration of the I- prefix should lead to "D3D12" being identified as the shared prefix + var result = await context.SourceProject.Documents.First().GetSyntaxRootAsync(); + await Verify(result!.NormalizeWhitespace().ToString()); + } + [Test] public async Task HintShouldNotAffectSharedPrefixIdentification() { diff --git a/tests/SilkTouch/SilkTouch/TestUtils.cs b/tests/SilkTouch/SilkTouch/TestUtils.cs index f6564f8cc0..9fab20fa81 100644 --- a/tests/SilkTouch/SilkTouch/TestUtils.cs +++ b/tests/SilkTouch/SilkTouch/TestUtils.cs @@ -1,6 +1,7 @@ // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. +using System.Text; using Microsoft.CodeAnalysis; namespace Silk.NET.SilkTouch.UnitTests; @@ -16,4 +17,27 @@ public static Project CreateTestProject() => "TestAssembly", LanguageNames.CSharp ); + + public static async Task VerifyDocumentsAsync(params IEnumerable documents) + { + var builder = new StringBuilder(); + var isFirst = true; + foreach (var document in documents.OrderBy(doc => doc.Name)) + { + if (!isFirst) + { + builder.AppendLine(); + } + + isFirst = false; + + builder.Append("// "); + builder.AppendLine(document.Name); + + var root = await document.GetSyntaxRootAsync(); + builder.AppendLine(root!.NormalizeWhitespace().ToString()); + } + + await Verify(builder.ToString()); + } }