Skip to content

Make tokenizer a multi command binary#49

Open
timflyio wants to merge 7 commits intomainfrom
multi-cmd
Open

Make tokenizer a multi command binary#49
timflyio wants to merge 7 commits intomainfrom
multi-cmd

Conversation

@timflyio
Copy link
Copy Markdown
Contributor

@timflyio timflyio commented Apr 7, 2026

The tokenizer command will support subcommands for displaying its version, running the server, displaying the seal key, sealing tokens, and unsealing tokens. This will allow the deployed binary to be used directly for multiple purposes without shipping seperate helpers and will simplify instructions for setting up and using the tokenizer.

  • Changed cmd/tokenizer/main.go to be a framework for launching subcommands. Used args.go for helpers.
  • Broke out the server's debug listener to its own file.
  • Broke out the existing features of tokenizer to separate subcommands in separate files: serve, sealkey, version.
  • Added subcommands for seal and unseal, to replace cmd/seal and cmd/unseal.
  • Allow sealing with the OPEN_KEY or just the SEAL_KEY. This allows the seal command to be run on the tokenizer service right from the OPEN_KEY in the environment without specifying the SEAL_KEY.
  • Changed the server configuration to not use fly-src parser by default, and only use it when asked for. Prior to this it would always try to open and parse /.fly/fly-src.pub, which would fail with a panic when it doesnt exist. This is more convenient for running the server locally. I added the new -use-flysrc=true flag to the Dockerfile, and documentation about running the server, to continue using it with our tokenizer.

The tokenizer command will support subcommands for displaying its version, running the server, displaying the seal key, sealing tokens, and unsealing tokens. This will allow the deployed binary to be used directly for multiple purposes without shipping seperate helpers and will simplify instructions for setting up and using the tokenizer.
@timflyio
Copy link
Copy Markdown
Contributor Author

timflyio commented Apr 7, 2026

As sanity checks, I've deployed to my timkenizer app and tested that it still runs properly. (I found that I forgot to add a new required flag to enable /.fly/fly-src.pub to be used, and fixed that).
I've also manually run some of the example commands in the user manual to verify they work with the updated syntax.

@timflyio timflyio marked this pull request as ready for review April 7, 2026 02:37
@timflyio timflyio requested a review from mjbraun April 7, 2026 07:17
@mjbraun
Copy link
Copy Markdown
Contributor

mjbraun commented Apr 7, 2026

I added tokenizer to my vulnbot scanning (since it's smarter/faster than I am!). Here's what it came back with.

Bug: discarded error in ParseOpenKey (tokenizer.go:112)

if len(privBytes) != 32 {
fmt.Errorf("bad private key size: %d", len(privBytes))
}

This creates an error but never returns it. A wrong-length key silently proceeds and will panic downstream when used as *[32]byte. go vet also flags this. Should be return nil, nil, fmt.Errorf(...).

Bug: double parseFlags in version.go:16

func runVersion(cmd string, args []string) {
fs := flag.NewFlagSet(cmd, flag.ContinueOnError)
parseFlags(fs, versionUsage, args)
parseFlags(fs, versionUsage, args) // duplicate

Copy-paste artifact — parses args twice.

@timflyio
Copy link
Copy Markdown
Contributor Author

timflyio commented Apr 8, 2026

if len(privBytes) != 32 { fmt.Errorf("bad private key size: %d", len(privBytes)) }

lol! good catch.

func runVersion(cmd string, args []string) { fs := flag.NewFlagSet(cmd, flag.ContinueOnError) parseFlags(fs, versionUsage, args) parseFlags(fs, versionUsage, args) // duplicate

yup yup, this one too.

heh. thank you. Both should be fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants