Separate the act of resolving vs loading a dataset

[ashkrisk]

Make it so that the DataSetLoader is no longer responsible for deciding how to make a dataset's files available to the application. It remains responsible solely for resolving a dataset's files, which includes fetching them from a remote if required.

To faciliate this, the DataSetInfo object now knows about the underlying files which make up a dataset and is free to provide methods which allow the caller to access the dataset in a format of their choosing. In particular, this makes it possible to add methods to DataSetInfo which allow for accessing datasets without loading them completely into memory.

Summary of changes:

• Introduces the DataSetFiles class for wrapping the base/query/gt paths of a dataset.
• DataSetInfo no longer accepts a Supplier<Dataset>, instead accepting a DataSetFiles object.
• DataSetInfo is responsible for converting DataSetFiles to an in-memory DataSet or other future formats.

Possible alternatives:

• Introduce an additional abstraction layer responsible for converting DataSetFiles to a DataSet or similar instead of passing the responsibility onto DataSetInfo. This increases complexity without much immediate benefit, so this is left to future patches if and when it becomes necessary (if we start supporting HDF5 again, for example).

[github-actions]

Before you submit for review:

• Does your PR follow guidelines from CONTRIBUTIONS.md?
• Did you summarize what this PR does clearly and concisely?
• Did you include performance data for changes which may be performance impacting?
• Did you include useful docs for any user-facing changes or features?
• Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
• Did you rebase your branch onto the latest main for regression testing and PR submission?
• Did you trigger regression testing via Run Bench Main and review results?
• Did you adhere to the code formatting guidelines (TBD)
• Did you group your changes for easy review, providing meaningful descriptions for each commit?
• Did you ensure that all files contain the correct copyright header?
• Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

[r-devulap]

This seems simple enough as a first step to prgress to running a a larger than memory dataset.

[r-devulap]

@ashkrisk you might want to rebase this to main

[MarkWolters]

I think the code looks fine but it feels like we're losing some flexibility in swapping out the Supplier function for the new DataSetFiles class. Would it not be possible to have DataSetFiles implement Supplier to achieve the same thing?

[ashkrisk]

Would it not be possible to have DataSetFiles implement Supplier to achieve the same thing?

The goal is to be able to use the same DataSetFiles instance to eventually load two different types of datasets - the current one which loads everything into memory and a future one which uses memory-mapping instead.

If DataSetFiles were to implement Supplier (and DataSetInfo treats it as a Supplier without knowledge of the underlying type), we are forced to pick one of those two access methods. That inflexibility is what this PR aims to remove.

I agree there are trade-offs associated with this. As I mentioned in the PR description there are alternatives that retain both kinds of flexibility, but I believe such a design is not worth the extra complexity at this point in time (and can always be added later if we really need it).

[ashkrisk]

@MarkWolters looking at the version of the DataSetInfo class from the last PR in this stack may help explain what I'm going for here. Specifically, there are two methods, getDataSet and getMappedDataSet, and the benchmark utility can pick the one it needs.

https://github.com/datastax/jvector/blob/6a304ba6d084324776a475ca327439ee7876e0e6/jvector-examples/src/main/java/io/github/jbellis/jvector/example/benchmarks/datasets/DataSetInfo.java

[MarkWolters]

Yeah, I see where you're going with it and I think it's a worthwhile effort, I just don't like how it's locked into Fvec/Ivec as the data format. I think it would be fairly cheap and easy to implement DataSetFiles as an interface that defines getBasePath(), getQueryPath() etc. without locking it in to one format, while providing the fvec or hdf5 or whatever implementation as a concrete class.