[Feature]: Add TransformVirtual Extensions for IObservableList and IObservableCache

[ktxiaok]

Describe the functionality desired 🐞

New extension methods TransformVirtual<TSource, TResult>(Func<TSource, TResult> transformFactory) for IObservableList<TSource> and IObservableCache<TSource, TKey> that create lightweight views of type IObservableList<TResult>/IObservableCache<TResult, TKey> without intermediate storage collection.

There's an implementation for IObservableList I created and used in my project:

public static class CollectionExtensions
{
    extension<T>(IReadOnlyList<T> source)
    {
        public IReadOnlyList<TResult> TransformVirtual<TResult>(Func<T, TResult> transformFactory)
        {
            ArgumentNullException.ThrowIfNull(source);
            ArgumentNullException.ThrowIfNull(transformFactory);

            return new ReadOnlyListTransformVirtual<T, TResult>(source, transformFactory);
        }
    }

    private sealed class ReadOnlyListTransformVirtual<TSource, TResult> : IReadOnlyList<TResult>
    {
        private readonly IReadOnlyList<TSource> _source;
        private readonly Func<TSource, TResult> _transformFactory;

        public ReadOnlyListTransformVirtual(IReadOnlyList<TSource> source, Func<TSource, TResult> transformFactory)
        {
            _source = source;
            _transformFactory = transformFactory;
        }

        public TResult this[int index] => _transformFactory(_source[index]);

        public int Count => _source.Count;

        public IEnumerator<TResult> GetEnumerator() => _source.Select(x => _transformFactory(x)).GetEnumerator();

        IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
    }
}

public static class ObservableListExtensions
{
    extension<T>(IObservableList<T> source) where T : notnull
    {
        public IObservableList<TResult> TransformVirtual<TResult>(Func<T, TResult> transformFactory) where TResult : notnull
        {
            ArgumentNullException.ThrowIfNull(source);
            ArgumentNullException.ThrowIfNull(transformFactory);

            return new ObservableListTransformVirtual<T, TResult>(source, transformFactory);
        }
    }

    private sealed class ObservableListTransformVirtual<TSource, TResult> : IObservableList<TResult>
        where TSource : notnull
        where TResult : notnull
    {
        private readonly IObservableList<TSource> _source;
        private readonly Func<TSource, TResult> _transformFactory;

        public ObservableListTransformVirtual(IObservableList<TSource> source, Func<TSource, TResult> transformFactory)
        {
            _source = source;
            _transformFactory = transformFactory;
        }

        public int Count => _source.Count;

        public IObservable<int> CountChanged => _source.CountChanged;

        public IReadOnlyList<TResult> Items => _source.Items.TransformVirtual(_transformFactory);

        public void Dispose()
        {
        }

        public IObservable<IChangeSet<TResult>> Connect(Func<TResult, bool>? predicate = null)
        {
            var ret = _source.Connect().Transform(_transformFactory);
            if (predicate is not null)
            {
                ret = ret.Filter(predicate);
            }
            return ret;
        }

        public IObservable<IChangeSet<TResult>> Preview(Func<TResult, bool>? predicate = null)
        {
            var ret = _source.Preview().Transform(_transformFactory);
            if (predicate is not null)
            {
                ret = ret.Filter(predicate);
            }
            return ret;
        }
    }
}

Sincerely thank all of you for creating DynamicData!

The steps the functionality will provide

From the end-user perspective:

1. Call the new extension method TransformVirtual on any IObservableList/IObservableCache to create another stateless IObservableList/IObservableCache as a projected view without intermediate storage collection in a fluent way.
2. Transformation execution is on-demand and should be pure (no side effects). The transformFactory will run if items are accessed via indexer or enumeration. There is no precomputation or materialization, unless a subscription of a downstream observer via Connect or Preview occurs.
3. Disposal of the returned IObservableList/IObservableCache is not required and does nothing because it's just a stateless wrapper.
4. Fully compatible with consumer scenarios that require direct index access, count property and enumeration support.

Considerations

Currently, in order to transform the elements of an IObservableList/IObservableCache, the Connect method needs to be called first and then use the Transform operator of observable change set. After that, the type of the current operating object becomes observable change set and the direct accessors of collection count and items are lost. In some cases, SourceList/SourceCaches are internally used and some transformed IObservableList/IObservableCaches should be exposed as public API. To revert to IObservableList/IObservableCache type, the method AsObservableList/AsObservableCache can be used, which creates intermediate storage collection (e.g. SourceList is created for IObservableList). If we just want to do a simple transformation, such as a type cast (even the implicit cast because covariance is not supported by DynamicData), this method is inefficient and unnecessary. To do this in a better way, we can just create a stateless wrapper of IObservableList/IObservableCache without inherent subscription and collection allocation.

[JakenVeina]

Transformation execution is on-demand and should be pure (no side effects). The transformFactory will run if items are accessed via indexer or enumeration. There is no precomputation or materialization, unless a subscription of a downstream observer via Connect or Preview occurs.

This is probably my biggest concern. Nothing about the surface API signals that the consumer must provide a pure transformation, and there's certainly no way for the operator to check. Existing .Transform() operators do not make the assumption that transformations are pure, except where it's explicitly signaled in the API (I.E. .TransformImmutable(), which in retrospect isn't really the proper name for such an operator). And based on the kinds of issues that get raised here, there's a very large portion of users, if not the majority, that use stateful items in dynamic data collections.

lightweight views of type IObservableList/IObservableCache<TResult, TKey> without intermediate storage collection.

Your stated goal of implementing this API is already falsified by your use of .Transform(), within .Connect() and .Preview(). The .Transform() operator maintains a full internal copy of the upstream collection, plus each associated downstream item, as do most operators. .TransformImmutable() would be what you want to use here, to skip internal storage.

TransformVirtual<TSource, TResult>(Func<TSource, TResult> transformFactory)

Would definitely need different terminology for this than "Virtual" as that already has meaning elsewhere in the library. Maybe .TransformThin()? TransformDynamic()? The key point to convey is that transformation (and filtering and anything else possibly added in the future) is applied at pull-time, rather than push-time. Sorta like IEnumerable<T> and System.Linq operators. Maybe there's something in there.

And actually, that "pull-time, rather than push-time" bit is a key point that makes me think this API doesn't really belong in DynamicData. The core premise of DynamicData, really, is about having collections, but built on top of reactive programming (I.E. System.Reactive and IObservable<T>). This concept of a virtualized collection is fundamentally the opposite, and doesn't do anything to leverage reactive programming at all. It's lazy evaluation, rather than reactive evaluation.

So, call me undecided. The idea sounds useful. But my immediate impression is that it seems like it should be part of its own library, built on the premise of lazy-evaluation. I'll have to think on it some more, and see what the other maintainers think.

[JakenVeina]

Alright, after some ruminating I realized something: this actually aligns really closely with something I've already had in mind for a while. We have one operator that I reworked a while back, .ExpireAfter() which has two versions: one that attaches to IObservable<IChangeSet<TObject, TKey>> and one that attaches directly to ISourceCache<TObject, TKey>. It occurred to me that quite a few other operators could benefit from having direct access to an upstream IObservabeCache<TObject, TKey>, rather than having to maintain their own copy, internally. So, the idea would be to have specialized versions of operators that are largely transparent to the consumer, just depending on whether they're being called upon a cache, or upon a stream.

I.E.

using var subscription = source
    .Filter(...)
    ...

versus

using var subscription = source
    .Connect()
    .Filter()
    ...

So, really, your proposal is just that, but taking it a step further. Not only can the operator implement internal optimizations, by having access to an upstream cache, it can also provide for possible DOWNSTREAM optimizations to other operators, by returning IObservableCache<TObject, TKey>.

Ultimately, I think the way this fits into our existing API is as follows:

public static IObservableCache<TIn, TOut, TKey> TransformImmutable(
        this    IObservableCache<TIn, TKey> source,
                Func<TIn, TOut>             transformFactory)
    where TObject : notnull
    where TKey : notnull;

I.E. it's just the same as our existing .TransformImmutable() operator, but when it's called upon an IObservableCache<TObject, TKey> directly, it gets the internal optimizations, and carries collection methods forward.

On a separate note, I 100% want to change that operator name to .TransformPure(), and deprecate TransformImmutable(). That's 100% a better name for what's going on with that operator. So, if that happens to go in before this new operator, then the name should match.

[ktxiaok]

That's a good insight. Giving operators access to source collections opens up more room for optimizations.