Fluent API for reading structured data

[renyuneyun]

Search terms you've used

• fluent
• fluent api

Feature suggestion

A fluent API for reading data.

Currently there is only a fluent API for constructing data. But it would be also possible (and maybe more widely usable) to create a fluent API for reading data.
In fact, the current style is almost there, but just with the addition of introducing relevant methods to the relevant types, instead of chaining function calls.

Expected functionality/enhancement

• Allow directly calling methods when reading structured data, instead of chaining function calls, to be more intuitive.
• Relevant methods for dealing with potentially multiple values/objects of the same subject & predicate.
• If the value is a URL (e.g. originally obtained by getUrl), allow "resolving" it as a Thing.
  • This may be provided in the first-level as well, saving the potential duplication of getSolidDataset.
  • Of course, corner-cases may need to be handled.
    • In particular, what if the current dataset already has triples of that URL (as subject)? That's why "resolving" (by fetching the dataset matching that URL) should not be the default behaviour. Maybe should also provide a method/function to "merge" the two datasets.

Use Cases

Easier development.

[NSeydoux]

Hi @renyuneyun , thanks for reaching out. That's a very interesting suggestion, and I know there have been some experiments on such approaches over time in the community. This is not something we'll implement in the near future, but I'm happy to help fleshing it out.

[renyuneyun]

Indeed. I guess you are talking about libraries such as LDflex and LDO?

IMO, there is still much difference. For example, LDO relies on ShEx or SHACL to predefine shapes, and then can make use of that. But the proposal here is to not requiring predefined shapes, but allow developers to quickly query data. This brings no loss to solid-client (as it doesn't verify shapes anyway), but allows flexibility and quick writing of queries.

Would be happy to try an implementation as I imagine that shouldn't take too long if having time (and guess PRs would be welcome?). But not in near future... unfortunately, because of time constraints...

[NSeydoux]

Sorry for the delayed response, I was out on vacations and remaining offine as much as possible :). PRs are absolutely welcome indeed, although it might be easier to start using this issue for some design, so that we are aligned on the general shape of the API before getting into implementation. Usual challenges for such graph-traversal DSL (since that's what we are talking about) are gracefully handling of node filtering versus traversal expressions, and entrypoint definition. There is no entrypoint issue with a Thing, because it s already an identified node of the graph, but traversal may yield different results depenging on where it starts in the graph. Did you have ideas of how you wanted to resolve this? Another way of asking is, could you provide code snippets of what the result of this new API would look like when used?

[renyuneyun]

Sure, I'll draft some examples here. Some shorthand / assumptions:

• I'll omit all namespace definitions, but assume things like NS_RDF('type') to return a term (as string).
• Assume we already have authFetch from a session.
• Getting a single object uses getXXX(), and getting multiple objects uses getXXXs() (note the s at the end).
• The getXXX() and getXXXs() methods won't return the value, but a wrapped version, to assist the fluent API; the value can be obtained by .value.

General idea

Start of query in the "fluent API"

const webId = '...';
const dataset = await getSolidDataset(webId, {fetch: authFetch});
const fluentDataset = fluentQuerier(dataset);  // A new function from the library to support fluent reading; or a method of the `Dataset` class

// Or, equivalently
const fluentDataset = await getSolidFluentDataset(webId, {fetch:authFetch});

Single-object retrieval

const myName = fluentDataset.getThing(webId).getStringNoLocale(NS_FOAF('name')).value;

Multi-object retrieval

const aPodOfMine = fluentDataset.getThing(webId).getUrls(NS_SPACE('storage'))[0].value;

Fluent fetch during traversal (with auth)

const aFriendName = fluentDataset.getThing(webId).getUrls(NS_FOAF('knows'))[0]
    .resolve({fetch: authFetch})
    .getStringNoLocale(NS_FOAF('name')).value;

• .resolve() can only be called on URLs, not on other types (maybe can gracefully accept strings.
• It will fetch the dataset at that URL, and return the Thing at the specific URL
  • IIRC, when URL has hash/segment, getSolidDataset() will get the document without the hash/segment. .resolve() will have the same behaviour when fetching; but when getting the Thing, it will use the hash/segment.

In addition, resolve() may also accept url: "..." as an optional parameter for specifying which URL to query, in case the user wants to override the target for some reason

Back to (fluent) dataset from thing

const aFriendProfileDocument = fluentDataset.getThing(webId).getUrls(NS_FOAF('knows'))[0]
    .resolve({fetch: authFetch})
    .dataset;  // Only callable for a Thing? Or maybe for all types?

Additional comments

• Basically, each step will return a wrapped object/instance, rather than the raw value. This will allow chaining the method calls, and/or assigning to a variable for future use.
  • The wrapper object may not be the same as current data types in the library, but I believe that's fine to define a "twin" of them specifically for the fluent API
• Consistent handling of non-existent values should be done. Probably through JS/TS built-in of undefined, and relevant operators. The examples above didn't explicitly consider this.
• The way the initial fluent dataset is constructed / obtained can be discussed, and I have provided three options above
• I don't feel it's necessary to distinguish between string with or without locale in the default behaviour, particularly for the fluent API; they can be used when needed, but not mandatory (e.g. as optional parameter to getString(), or as additional functions than getString()).
• The fluent dataset may also support querying all subjects, or Things, but probably not to more complex queries.

[NSeydoux]

This looks like a good starting point. One of the things that was challenging with LDFlex was to handle the step equivalent to resolve here: when dealing with a dataset, it is hard to know when you're dealing with a URI pointing to a remote reference vs when you're dealing with a reference present in the current dataset. I'd prefer to initially only support the latter, so I wouldn't include resolve in the fluent API yet, because it would require the whole expression to be async. There are two approaches to deal with this local-only traversal

1. not extending the fluent API

const aFriendWebId = fluentDataset.getThing(webId).getUrls(NS_FOAF('knows'))[0];
// At this point we could use the existing `getThing`
const aFriendName = fluentDataset.getThing(aFriendWebId).getStringNoLocale(NS_FOAF('name')).value;  

2. extending the fluent API

const aFriendName = fluentDataset
  .getThing(webId)
  // getUrls returns a special wrapper binding `getThing` to the URL
  .getUrls(NS_FOAF('knows'))[0]
  .getThing()
  // introducing this fluent API item allows to gracefully deal with a URL not being local to the current dataset, potentially to try fetching it remotely in a separate async call.
  ifPresent((t) ->  
    t.getStringNoLocale(NS_FOAF('name'))
    undefined
  );

The fluent dataset may also support querying all subjects, or Things, but probably not to more complex queries.

That sounds like a very reasonable assumption, at least for a first iteration. We may introduce something like

const withNameWebId = fluentDataset.filter(t -> t.getStringNoLocale(NS_FOAF('name')).value !== undefined)[0]

in the future if needed, but it is not a must-have initially.

[renyuneyun]

One of the things that was challenging with LDFlex was to handle the step equivalent to resolve here: when dealing with a dataset, it is hard to know when you're dealing with a URI pointing to a remote reference vs when you're dealing with a reference present in the current dataset.

Indeed, that's why resolve has to be explicit. If the developer wants to find it (as a subject) in the current dataset, I'd suggest using an alternative:

const aFriendName = fluentDataset.getThing(webId).getUrls(NS_FOAF('knows'))[0]
    .asThing().getStringNoLocale(NS_FOAF('name')).value;  

Thus, there is an explicit semantic difference:

• asThing() retrieves the corresponding Thing from the current dataset
• resolve() fetches the corresponding Thing by an explicit new fetch of another dataset

Of course, this can be extended further. In particular:

• mergeResolve() fetches the other dataset, and merges with the current dataset (for whatever reason the developer wants to do this). Though, I expect this to be rarely used by most developers for most use cases, so don't expect it in the initial version.

... wouldn't include resolve in the fluent API yet, because it would require the whole expression to be async.

That's indeed a fair point. But I would consider it more beneficial (assuming the strategy above), as it's more intuitive, and, fluent. The only place for async is the resolve method, so it should be indicative enough (about its nature in network operation) and also not frustrating other use cases (as they won't be async).

But if async needs to be eliminated, especially if it needs to be eliminated in the middle of the expression, there is an alternative interface design, taking some insights from SQLAlchemy (Python library):

const aFriendName = await fluentDataset.getThing(webId).getUrls(NS_FOAF('knows')).getAt(0).resolve().getStringNoLocale(NS_FOAF('name')).exec();

The implication is:

• We can't get .value arbitrary, as the interface is redesigned
• Only after .exec(), the whole query is executed, and result returned (as value, so no .value needed)
• Each method call is not a query in the usual sense, but "introducing a component/step to the query"
  • It introduces a new part to the query, but won't execute it there yet
  • Because no execution is performed, no await is needed whatsoever
• .exec() should be await-ed
  • We can even differentiate between the interfaces/classes in the middle, so only when/after .resolve() is part of the query (thus changing the returned class/interface after this method call), the .exec() is async, otherwise sync.

Apparently, this requires more code (of the library) to smooth everything. But it has the potential to optimize the query performance, if an engine is able to cut down unnecessary parts when/before executing .exec().

[NSeydoux]

I don't mind the expression being async, as long as it's explicit. If resolve() is only used for async traversal, and doesn't lead to ambiguous behaviors, I'm fine with the approach. I wouldn't start trying to optimize performances with a query engine-like behavior, there are other projects (such as Comunica) which are far better equipped to do this, I don't believe it is a good match for this library at this stage. Getting a clean fluent API for data reading would already be really nice :).

In terms of typing, what would resolve return? I can see two approaches:

• simplistically, it returns a FluentDataset (naming pending) wrapped in a Promise. It reduces the amount of code, because there is a single syncrhonous fluent API, but it means the fluent expressions would not be so fluent, as they would contain promise chains. I actually like leveraging as much of the core language as possible, so this might be nice because it limits the amount of new stuff that would be introduced to developpers using this
• otherwise, it returs a AsyncFluentDataset, which has an API separate from the FluentDataset, probably mirrorred but for async operations, and the .value results in the Promise being exposed so that it can be awaited. It would make expressions more fluent, but it would expand the amount of code to maintain and of concepts to master for developers using this.

I'll ponder which has the best DX/maintenance balance, I'll see if I get time to sketch what the expressions would look like.

I think having fluent expressions such as ifPresent (as sketched above, inspired from Java Optional API) would help dealing with undefined values in the expression chain. Having some sort of wrapper around the value being passed around in the expression will help handling that, given the very diverse nature of data in Solid datasets this approach embraces.