Designing LINQ operators

I've started a small project (I'll post a link when I've actually got something worthwhile to show) with some extra LINQ operators in - things which I think are missing from LINQ to Objects, basically. (I hope to include many of the ideas from an earlier blog post.) That, and a few Stack Overflow questions where I've effectively written extra LINQ operators and compared them with other solutions, have made me think about the desirable properties of a LINQ operator - or at least the things you should think about when implementing one. My thoughts so far:

Lazy/eager execution

If you're returning a sequence (i.e. another IEnumerable<T> or similar) the execution should almost certainly be lazy, but the parameter checking should be eager. Unfortunately with the limitations of the (otherwise wonderful) C# iterator blocks, this usually means breaking the method into two, like this:

public static IEnumerable<T> Where(this IEnumerable<T> source,
                                   Func<T, bool> predicate)
{
    // Eagerly executed
    if (source == null)
    {
        throw new ArgumentNullException("source");
    }
    if (predicate == null)
    {
        throw new ArgumentNullException("predicate");
    }
    return WhereImpl(source, predicate);
}

private static IEnumerable<T> WhereImpl(IEnumerable<T> source,
                                        Func<T, bool> predicate)
{
    // Lazily executed
    foreach (T element in source)
    {
        if (predicate(element))
        {
            yield return element;
        }
    }
}

Obviously aggregates and conversions (Max, ToList etc) are generally eager anyway, within normal LINQ to Objects. (Just about everything in Push LINQ is lazy. They say pets look like their owners...)

Streaming/buffering

One of my favourite features of LINQ to Objects (and one which doesn't get nearly the publicity of deferred execution) is that many of the operators stream the data. In other words, they only consume data when they absolutely have to, and they yield data as soon as they can. This means you can process vast amounts of data with very little memory usage, so long as you use the right operators. Of course, not every operator can stream (reversing requires buffering, for example) but where it's possible, it's really handy.

Unfortunately, the streaming/buffering nature of operators isn't well documented in MSDN - and sometimes it's completely wrong. As I've noted before, the docs for Enumerable.Intersect claim that it reads the whole of both sequences (first then second) before yielding any data. In fact it reads and buffers the whole of second, then streams first, yielding intersecting elements as it goes. I strongly encourage new LINQ operators to document their streaming/buffering behaviour (accurately!). This will limit future changes in the implementation admittedly (Intersect can be implemented in a manner where both inputs are streamed, for example) but in this case I think the extra guarantees provided by the documentation make up for that restriction.

Once-only evaluation

When I said that reversing requires buffering earlier on, I was sort of lying. Here's an implementation of Reverse which doesn't buffer any data anywhere:

public static IEnumerable<T> StreamingReverse<T>(this IEnumerable<T> source)
{
    // Error checking omitted for brevity
    int count = source.Count();
    for (int i = count-1; i >= 0; i--)
    {
        yield return source.ElementAt(i);
    }
}

If we assume we can read the sequence as often as we like, then we never need to buffer anything - just treat it as a random-access list. I hope I don't have to tell you that's a really, really bad idea. Leaving aside the blatant inefficiency even for sequences like lists which are cheap to iterate over, some sequences are inherently once-only (think about reading from a network stream) and some are inherently costly to iterate over (think about lines in a big log file - or the result of an ordering).

I suspect that developers using LINQ operators assume that they'll only read the input data once. That's a good assumption - wherever possible, we ought to make sure that it's correct, and if we absolutely can't help evaluating a sequence twice (and I can't remember any times when I've really wanted to do that) we should document it in large, friendly letters.

Mind your complexity

In some ways, this falls out of "try to stream, and try to only read once" - if you're not storing any data and you're only reading each item once, it's quite hard to come up with an operator which isn't just O(n) for a single sequence. It is worth thinking about though - particularly as most of the LINQ operators can work with large amounts of data. For example, to find the smallest element in a sequence you can either sort the whole sequence and take the first element of the result or you can keep track of a "current minimum" and iterate through the whole sequence. Clearly the latter saves a lot of complexity (and doesn't require buffering) - so don't just take the first idea that comes into your head. (Or at least, start with that and then think how you could improve it.)

Again, documenting the complexity of the operator is a good idea, and call particular attention to anything which is unintuitively expensive.

Conclusion

Okay, so there's nothing earth-shattering here. But the more I use LINQ to answer Stack Overflow questions, and the more I invent new operators in the spirit of the existing ones, the more powerful I think it is. It's amazing how powerful it can be, and how ridiculously simple the code (sometimes) looks afterwards. It's not like the operator implementation is usually hard, either - it's just a matter of thinking of the right concepts. I'm going to try to follow these principles when I implement my extra operator library, and I hope you'll bear them in mind too, should you ever feel that LINQ to Objects doesn't have quite the extension method you need...

...Full Article.