Skip to content

FP8 Metal carry (#30) follow-ups: E5M2 test, auto-fit sizing, LoRA crashes #31

Closed

Description

@leehack

Follow-ups from the independent audit of #30 (CLEAR at 0a865ab, merged as 1694916). None block; evidence in llamadart-evidence/sd-native-pr-30/.

  • E5M2 not covered by tests/test_fp8_runtime.py. Dropping E5M2 from the patch's type check passes the test, yet that build aborts on an E5M2 model on MTL0. Add an E5M2 copy to the test.
  • auto-fit under-plans FP8 on Metal. backend_fit.cpp estimate_components uses ts.nbytes() (1 byte/elem) while Metal now holds FP8 weights as F16 (2 bytes). sd_turbo-fp8-all: plan 1244 MiB, Metal registers 2583 MB. Risk: large FP8 checkpoints on memory-tight iOS devices. Options: account for the retype in the estimate, or cast in-graph with the existing FP8→F16 copy kernel to keep 1 byte/elem. Document the doubled memory meanwhile.
  • Retirement test is heuristic. It misses an upstream exclusion through a helper (e.g. ggml_is_type_fp8) or a kernel named …_fp8_…; cost is only carrying the patch longer.
  • Pre-existing, not caused by [0.17] Load FP8 weights as F16 on Metal instead of aborting #30 (same on unpatched builds), synthetic LoRA: LoRA immediate mode aborts on Metal even with an F16 base ("pre-allocated tensor … cannot run the operation", ggml-backend.cpp:1089); an FP8 base with a LoRA segfaults on CPU (immediate, and at-runtime with FP8 lora_down).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions