Draft: Add automated test runner and visualization script for stress testing

[ahzero7d1]

• Add extra arguments for running full test configurations
• Separate test configuration and execution script
• Add visualization script

[JulienVig]

Thanks Ahyoung, the scripts are very handy! I've left a few comments and questions.

• Can you rename scripts to something like python to clarify the difference with the other typescript scripts also in cli/src?
• Can you also add a short README.md in the scripts (python) folder with instructions and example to use the two scripts?

[JulienVig]

Could you add a requirements.txt with the libraries contributors should install?

[JulienVig]

Why were there NaN stds?

[JulienVig]

Could you set the default value to the path to basic_tests.json?

[JulienVig]

I'm getting this error:

cli/dist/args.js:46
    throw Error(`${unsafeArgs.task} not implemented.`);
          ^
Error: mnist_federated not implemented

Did you create new default tasks (mnist_federated, titanic_decentralized, etc.)?

[JulienVig]

Did you choose the "ndjson" name over jsonl for a particular reason? I had only ever seen jsonl until now so unless you particularly prefer ndjson I would change it to jsonl

[JulienVig]

Suggested change

	/**
	* Return validation metrics
	*
	* TODO: currently only works for TFJS, gpt models
	*/
	evaluate(
	_validationDataset?: Dataset<Batched<DataFormat.ModelEncoded[D]>>
	): Promise<ValidationMetrics>{
	throw new Error("Evaluation not supported for this model");
	}
	/**
	* Return validation metrics
	*/
	abstract evaluate(
	_validationDataset?: Dataset<Batched<DataFormat.ModelEncoded[D]>>
	): Promise<ValidationMetrics>;

I think we can make this method abstract rather than throwing