Pipes & Filters Architecture Pattern

1. Pipes & Filters Architecture Pattern Source: Pattern-Oriented Software Architecture, Vol. 1, Buschmann, et al

2. Problem • You are designing a system that needs to perform several transformations on some data • The transformations can be performed sequentially • It should be easy to: • Reorder the sequence of the transformations • Add new kinds of transformations • Remove transformations • There are different sources of input data (files, network, programs) • There are different destinations for output data (files, network, programs)

3. Solution • Organize the system using Pipes & Filters • Input data originates from Data Source components • Output data is consumed by Data Sink components • Each transformation required to convert the input into the output is implemented by a Filter component • Data Sources, Filters, and Data Sinks arranged in a pipeline • Pipes connect adjacent components, the output of one component becoming the input to the next component

4. Solution • Sources, Filters, and Sinks can be Active or Passive • Active components run on their own thread of control • Passive components execute only when invoked, directly or indirectly, by an Active component

5. Dynamic Behavior : Scenario I • Active Source / Passive Filter / Passive Sink

6. Dynamic Behavior : Scenario II • Passive Source / Passive Filter / Active Sink

7. Dynamic Behavior : Scenario III • Passive Source / Active Filter / Passive Sink

8. Dynamic Behavior : Scenario IV Buffering Pipe • Passive Source / Multiple Active Filters / Passive Sink

9. Implementation • Divide the system into a sequence of processing stages • Define the data format to be passed along each pipe • Decide how to implement each pipe connection • The simplest Pipe is direct Read/Write method calls between adjacent filters • If buffering or synchronization is required between filters, actual Pipe objects will be needed to perform these functions

10. Implementation • Design and implement the filters • Passive filters can be implemented as regular methods • Active filters can be implemented as processes or threads • Design the error handling • In pipelines with one active component, standard error handling mechanisms can be used (exceptions, return codes, etc.) • In pipelines with multiple active components, how will an error in one thread of control become visible to other threads? • If an error occurs, we could: • Tear down the entire pipeline and restart it from scratch, or • Restart only the parts that failed and resynchronize the pipeline • Setup the pipeline

11. Known Uses: UNIX Command Pipelines cat fall.txt win.txt | sort | gzip | mail fred@byu.edu

12. Known Uses: Image Processing

13. Known Uses: Compilers

14. Consequences • Very flexible • Filters can be reused and recombined in arbitrary ways • Individual filters can be easily replaced (e.g., plug in a different kind of compression) • No intermediate files necessary between processing stages • Benefits from efficiencies inherent in parallel processing (multiple active components) • Some filters don’t produce any output until they've consumed all of their input (e.g., sort), which is not conducive to parallelism • Data transfer and context switching between processes and threads can be expensive