Understanding llama-bench

[aneeshmb02]

I am trying to understand what data(prompts) is being used by the llama-bench tool.
I actually want to compare the performance of different models with different configurations (varying hardware and params). Hence, I need a way to automate the testing the process. Llama-bench seems to be doing that but I want control over the prompts that are used for benchmarking.

[slaren]

The prompts used by llama-bench are randomly generated sequences of tokens. You can configure the length, but not the contents.

[aneeshmb02]

Thanks a lot!

[aneeshmb02]

I tried using mistral 7b with llama-bench but the output shows llama 8b

Why is that?

[slaren]

The heuristic test to detect the model type is wrong and needs to be fixed.

[aneeshmb02]

Oh okay

[kongweiming]

excuse me ,when tg == 128, what is pp's value this moment?

[stepinto]

excuse me ,when tg == 128, what is pp's value this moment?

Looking into the code, I guess it is 1. The decode benchmark is not affected by the -p parameter. It always start with a prompt of single random token. Please correct me if I was wrong.