Pipes and Filters Design Pattern: A Practical Approach

When it comes to data processing, one of the common challenges developers face is how to write maintainable and reusable code. The pipes and filters design pattern is a powerful tool that can help you solve this problem by breaking down a complex data processing task into a series of simple, reusable filters that can be combined in different ways to achieve different results.

In this blog post, we will take a look at how to use the pipes and filters pattern to validate incoming data using C#, We will start by defining the basic building blocks of the pattern, then we will implement a simple example that demonstrates how the pattern works in practice.

Building Blocks

The pipes and filters pattern consists of three main components: filters, pipes, and the pipeline.

  • A filter is simple piece of code that performs a specific task on the input data and returns the result. In our example, we will have two filters: a validation filter that checks if the input data is valid and a transformation filter that converts the input data to uppercase.
  • A pipe is a data structure that connects the output of one filter to the input of another. In our example, we will not use pipes explicitly, but the pipeline will be responsible for connecting the filters together.
  • The pipeline is the main component of the pattern that holds all the filters and connects them together. It is responsible for applying the filters to the input data in the correct order and returning the final result.

Implementing the Pipes and Filters Pattern in C#

Now that we have a basic understanding of the components of the pipes and filters pattern, let’s take a look at how we can implement in C#.

First, we will define and interface for filters called IPipeFilter<T> that has a single method called Process that takes in a input of type T and returns output of type T.

interface IPipeFilter<T>
{
    T Process(T input);
}

Next we will create two filters that implement this interface. The first one is DataValidationFilter that checks if the input data is valid and throws an exception if it is not.

class DataValidationFilter : IPipeFilter<string>
{
    public string Process(string input)
    {
        if (string.IsNullOrWhiteSpace(input))
            throw new Exception("Invalid input data");

        return input;
    }
}

The second filter is DataTransformationFilter that converts the input data to uppercase.

class DataTransformationFilter : IPipeFilter<string>
{
    public string Process(string input)
    {
        return input.ToUpper();
    }
}

Finally, we will create a class called DataProcessingPipeline that takes a list of IPipeFilter<T> as a constructor argument, and it applies each filter in the list to the input data in the order they are provided.

class DataProcessingPipeline<T>: IPipeLine<T> {
  private readonly List <IPipeFilter<T>> _filters;

  public DataProcessingPipeline(List <IPipeFilter<T>> filters) {
    _filters = filters;
  }

  public T Process(T input) {
    foreach(var filter in _filters) {
      input = filter.Process(input);
    }
    return input;
  }
}

with the above classes we are ready to implement the pipeline and use it to validate and transform incoming data.

class Program
{
    static void Main(string[] args)
    {
        var pipeline = new DataProcessingPipeline<string>(new List<IPipeFilter<string>>
        {
            new DataValidationFilter(),
            new DataTransformationFilter()
        });

        try
        {
            var processedData = pipeline.Process("valid input data");
            Console.WriteLine(processedData);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }
}

In this example, we first create an instance of DataProcessingPipeline<string> with a list of filters that contains DataValidationFilter and DataTransformationFilter, Then we apply the pipeline to the input data “valid input data”, the output of this pipeline will be “VALID INPUT DATA”.

Conclusion

The pipes and filters pattern is a powerful tool for breaking down complex data processing tasks into simple, reusable components. It can help you write maintainable and reusable code that is easy to understand and modify. In this blog post, we have seen how to use the pipes and filters pattern to validate incoming data using C#, but this pattern can be used in many other scenarios as well. I hope this example will give you a good starting point for using this pattern in your own projects.

Dynamic objects in C#

Dynamic objects in C# are objects that can have properties and methods added to them at runtime, as opposed to traditional objects which have a fixed set of properties and methods defined at compile time. There are a few different ways to create dynamic objects in C#, including the dynamic keyword and the ExpandoObject class.

Here’s and example of using the dynamic keyword to create a dynamic object:

dynamic obj = new ExpandoObject();
obj.Name = "John Smith";
obj.Age = 30;

console.WriteLine($"Name: {obj.Name}, Age: {obj.Age}");

In the example above, we create a dynamic object using the ExpandoObject class, which is part of the System.Dynamic namespace. We then add a Name and Age property to the object, and print them out.

Here’s an example of using the ExpandoObject class to create a more powerful and flexible dynamic object:

dynamic obj = new MyDynamicObject();
obj.Name = "John Smith";
obj.Age = 30;

console.WriteLine($"Name: {obj.Name}, Age: {obj.Age}");

public class MyDynamicObject : DynamicObject
{
    private Dictionary<string, object> _properties = new Dictionary<string, object>();

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        _properties[binder.Name] = value;
        return true;
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        return _properties.TryGetValue(binder.Name, out result);
    }
}

In the example above, we define a MyDynamicObject class that inherits from DynamicObject, and overrides the TrySetMember and TryGetMember methods. These methods allow us to add and retrieve properties from the object at runtime.

Dynamic objects can be useful in a variety of situations, such as when you need to work with data that has a flexible schema, or when you want to add properties and methods to an object at runtime based on user input. However, it’s important to keep in mind that using dynamic objects can make your code less predictable and more difficult to debug, so it’s usually best to use them sparingly. The ExpandoObject is a more powerful and flexible option than the dynamic keyword, but it may be more complex to use in some cases.

dynamic vs expando

  • dynamic is a type, whereas ExpandoObject is a class: The dynamic keyword is a type that can be used to declare a variable. When you declare a variable as dynamic, you can assign any value to it at runtime. The ExpandoObject class, on the other hand, is a class that you can use to create a dynamic object
  • dynamic objects are resolved at runtime, whereas ExpandoObject objects are resolved at runtime and compile time: When you use the dynamic keyword, the compiler does not check the type of the object at compile time. Instead, the type is resolved at runtime, which can make it more difficult to catch errors. The ExpandoObject, on the other hand, is resolved at both runtime and compile time, which means that the compiler can catch some errors that might occur when using the object.
  • dynamic objects have limited functionality, whereas ExpandoObject objects have more functionality: The dynamic keyword is limited to the functionality provided by the .NET runtime. This means that you cannot add your own methods or properties to a dynamic object. The ExpandoObject, on the other hand, is part of the System.Dynamic namespace, which provides a number of classes and methods for working with dynamic objects. This means that you can use the ExpandoObject to create more powerful and flexible dynamic objects.

Conclusion

In general, the ExpandoObject is a more powerful and flexible option than the dynamic keyword, since it provides more functionality and is resolved at both runtime and compile time. However, the dynamic keyword can be a simpler and more lightweight option in some cases, particularly when you don’t need the additional functionality provided by the ExpandoObject.

How to create dotnet console application in 3 steps

Introduction

.NET core is a Cross Platform Microsoft Open Source framework which supports variety of platforms including Ubuntu, Debian, Fedora, CentOS, RHEL, MAC and of course Windows.

You can use either Visual Studio Community or Visual Code. Both are completely free to use.

Install .Net Core SDK

Download .Net Core SDK based on the platform that you are working on and install it.

Create App

Go to the terminal and create a new directory and run following command

dotnet new

Run App

To run the app you need to restore the packages using the below command and run the next command to see the output of the hello world program

dotnet restore

dotnet run

if you are looking for more samples you can visit the dotnet github page.

Thanks,