Faster Coroutine Pipelines

A paper presented at ICFP'17.

Abstract

Coroutine pipelines provide an attractive structuring mechanism for complex programs that process streams of data, with the advantage over lazy streams that both ends of a pipeline may interact with the I/O system, as may processes in the middle. Two popular Haskell libraries, Pipes and Conduit, support such pipelines. In both libraries, pipelines are implemented in a direct style by combining a free monad of communication events with an interpreter for (pseudo-)parallel composition that interleaves the events of its argument processes. These implementations both suffer from a slow-down when processes are deeply nested in sequence or in parallel. We propose an alternative implementation of pipelines based on continuations that does not suffer from this slow-down. What is more, the implementation is significantly faster on small, communication-intensive examples even where they do not suffer from the slow-down, and faster even than comparable programs based on lazy streams. The continuation-based implementation may be derived from the direct-style implementation by algebraic reasoning.

Download

• Text of the paper.
• Archive containing source code (see the enclosed README.txt for instructions).
• Slides for a talk.