Skip to content

scraper: fix build and re-organize module#4170

Draft
cmitu wants to merge 1 commit intoRetroPie:masterfrom
cmitu:scraper-ftbfs
Draft

scraper: fix build and re-organize module#4170
cmitu wants to merge 1 commit intoRetroPie:masterfrom
cmitu:scraper-ftbfs

Conversation

@cmitu
Copy link
Copy Markdown
Contributor

@cmitu cmitu commented Mar 27, 2026

  • Fixed the build errors caused by the kjk/lzmadec project being removed from Github. The vendor folder contains a locally mirror of the module, but the go version we used ignores the vendored dependencies folder. Bumping the go version fixes the build by using the local version of the package.

  • Refactored the module to remove the golang module. Since the go installation is needed just for building (the result is a statically linked scraper binary), we don't need to have an entire installation kept outside the build step. Given the current lack of changes in the upstream project, I think it's better we provide a binary for installation instead of a source install + local go install. Thus:

    • adopt the golang installation function into scraper as _golang_scraper, installing it in the build folder during a source installation.
    • remove the golang module and add an update hook to remove its installation folder
    • use GOMODCACHE and GOPATH so that go cache folders for packages download and build are not saved under /root. In order to shorten the go commands, I used a subshell to be able to export the vars needed and keep them local.

* Fixed the build errors caused by the `kjk/lzmadec` project being removed from Github.
  The `vendor` folder contains a locally mirror of the module, but the `go` version we used ignores the vendored dependencies folder.
  Bumping the `go` version fixes the build by using the local version of the package.

* Refactored the module to remove the `golang` module.
  Since the `go` installation is needed _just_ for building (the result is a statically linked `scraper` binary), we don't need to have an entire installation kept outside the build step. Given the current lack of changes in the upstream project, I think it's better we provide a binary for installation instead of a source install + local `go` install. Thus:

  - adopt the `golang` installation function into `scraper` as `_golang_scraper`, installing it in the build folder during a source installation.
  - remove the `golang` module and add an update hook to remove its installation folder
  - use `GOMODCACHE` and `GOPATH` so that `go` cache folders for packages download and build are not saved under `/root`.
    In order to shorten the `go` commands, I used a subshell to be able to `export` the vars needed and keep them local.
@cmitu
Copy link
Copy Markdown
Contributor Author

cmitu commented Mar 27, 2026

A couple of other notes, not part of the commit msg:

  • it's possible to use the latest GO version to build scraper, but I haven't found a (portable) set of commands that would use the just the project's vendor folder for the dependent go packages, without downloading them. I found a set of commands that work on an x86, but the same commands don't work on an ARM system.
  • since the included API key for ThegamesDb.net is expired/revoked, I requested a new one and we can patch the included go client package for TGDB to use it.
    Not included yet, should the patching should be done after the go get github.com/J-Swift/thegamesdb-swagger-client-go@43ed8a0b364ed2d8521d0 line. Perhaps using a local patch to replace the downgrade + API key replacement would work best.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant