feature: Ability to add custom resources to executor

[vsoch]

It would be good for the executor interface to have, akin to the command line arguments, an easy way to add custom resources to a step. I had tried adding these for the Kueue executor I was working on https://github.com/snakemake/snakemake-executor-kueue#job-resources but ran into bugs / issues (I think I reported somewhere but don't remember off the top of my head now!)

Now that the interface is more established, I am going to go back to Kueue and give it another look, and I can possibly make suggestions about how to go about it. I'm actually starting on the skeleton for Google Batch this morning - I'm just about ready to record my talk (thank goodness) so time should be opening up shortly and I'm super excited to get this underway!

I'll report back here if I make some time to work on the executor interface. My thinking is that we want something that mirrors the command line arguments, eg..,

from snakemake_interface_executor_plugins.resources import ExecutorResourcesBase

# Optional:
# Define custom resources for your executor
# They will be looked for as <executor-name>-<resource-name> in  each step.
# All step resources must be optional
@dataclass
class ExecutorResources(ExecutorResourcesBase):
    container: Optional[string]=field(
        default=None,
        metadata={
            "help": "The name of the container to use for the step.", 
            "env_var": False,
        }
    )

The above would be acceptable as:

rule a:
    input:     ...
    output:    ...
    resources:
        kueue_container=ghcr.io/rse-ops/mamba:app-mamba
    shell:
        "..."

I'll see if I can prototype something today after batch and the kueue refactor!

[vsoch]

Looking at previous notices - I think I like this design slightly better:

rule a:
    input:     ...
    output:    ...
    resources:
        kueue.container=ghcr.io/rse-ops/mamba:app-mamba
    shell:
        "..."

It clearly distinguishes the separation between executor name (.) and the parameter, which can have underscore.

[vsoch]

Here is the bug again - no matter what I try:

$ snakemake --cores 1 --executor kueue --kueue-registry registry-0 --jobs 1
SyntaxError in file /home/vanessa/Desktop/Code/snek/snakemake-executor-plugin-kueue/example/hello-world/Snakefile, line 14:
Expecting rule keyword, comment or docstrings inside a rule definition. (Snakefile, line 14)

[vsoch]

And then when I get it (maybe closer?) it tells me my name is an unexpected keyword (this is the same for kueue.container or kueue_container)

SyntaxError in file /home/vanessa/Desktop/Code/snek/snakemake-executor-plugin-kueue/example/hello-world/Snakefile, line 16:
Unexpected keyword kueue_container in rule definition (Snakefile, line 16)

# By convention, the first pseudorule should be called "all"
# We're using the expand() function to create multiple targets
rule all:
	input:
		expand(
			"{greeting}/world.txt",
			greeting = ['hello', 'hola'],
		),

# First real rule, this is using a wildcard called "greeting"
rule hello_world:
    # Note that this is the default, shown for example only
    output:
		"{greeting}/world.txt"
    resources: 
		kueue.container="ghcr.io/rse-ops/mamba:app-mamba"
	shell:
		"""
		mkdir -p "{wildcards.greeting}"
		echo "{wildcards.greeting}, World!" > {output}
		"""

[vsoch]

Ah! I finally got one working! it doesn't like the dot syntax, and the spacing / characters have to be consistent.

# By convention, the first pseudorule should be called "all"
# We're using the expand() function to create multiple targets
rule all:
	input:
		expand(
			"{greeting}/world.txt",
			greeting = ['hello', 'hola'],
		),

# First real rule, this is using a wildcard called "greeting"
rule hello_world:
	output:
		"{greeting}/world.txt",
	resources: 
		kueue_container="ghcr.io/rse-ops/mamba:app-mamba"
	shell:
		"""
		mkdir -p "{wildcards.greeting}"
		echo "{wildcards.greeting}, World!" > {output}
		"""