Cloud Native Patterns Illustrated: Fan-in and Fan-out | by Ozan Sazak | Oct, 2022

A single processor data pipeline will eventually run into a bottleneck. Let’s use concurrent processors with fan-in/fan-out queues!

I have started reading technical books after graduating from CS to expand my knowledge on many subjects, such as System Design, Low-Level CS, Cloud Native, etc. now i am studying cloud native go, which provides an intuitive introduction to the cloud native world using the benefits of Go. The book starts with Go Basics and Cloud Native Patterns, and that’s where I thought “I should write about these patterns!”.

In this series of blog posts, I’ll write about the Cloud Native patterns I’ve learned – as well as using pictures moneymi,

Let’s say we have a continuous data source (such as a Go Channel), and we need to process the data and put the processed data into a destination channel with as low latency as possible.

In the general case, we have a processor function that intersects the source and destination, which processes the data packets (here packet is an abstraction for simplicity) one by one.

Animation of single-processor data pipeline
Animation of single-processor data pipeline

However, each processing function has latency. This may be due to network or CPU usage, blocking syscalls, etc. If we send enough packets per second for the processor to function, then VOICE! Now we have a hitch!

Animating a single-processor data pipeline with congestion
Animating a single-processor data pipeline with congestion

There is a fairly simple solution to this problem: using multiple processors inside the data pipeline. In this way, we can process data streams concurrently, which reduces overall latency and reduces pipeline congestion.

Animation of multi-processor data pipeline
Animation of multi-processor data pipeline

We can implement this solution using shared memory, such as message queues.

With this approach, we will split the incoming data packet into separate input queues. Then, each queue’s own processor will take the packets one by one, process the data, and put the processed data into the corresponding output queue. Finally, the destination (another processor, queue, or any other system) will take the processed data packets from each output queue, and collect them into a single data source.

The first approach to split the data source (input) into multiple data sources (input queues) is called Fan-out pattern. Second, aggregating multiple data sources (output queues) into one data source (destination) is called Fan-in pattern.

For simplicity, we have specified an input queue and an output queue for each processor in our pipeline. We can use multiple input/output queues per processor, or a shared queue between a few processors, depending on the overall system requirements.

Animation of multi-processor data pipeline with input and output queues (channels)
Animation of multi-processor data pipeline with input and output queues (channels)

Let’s get our hands dirty with some concurrent Go! First, let’s define our data sources:

Program stringGenerator Creates a receive-only string channel, a . makes goroutine which puts prefixed strings in the channel, and returns the channel. We read in fan-out code later from this channel.

Our processor function will also be quite simple:

Inside the processor function, we will wait for a random amount of time to simulate processor latency.

In the fan-out implementation, we will take the receive-only channel, and return a fragment of the receive-only channel:

goroutines created inside Splitter The function will handle the data routing logic. Note that inside goroutine, we used single range statement to get from source Channel:

for data := range source 
dest <- data

This means that each goroutine will try to read from the channel in the loop, and the one that reads first will receive the next item. In other words, Each goroutine will fight on the next data instance,

We can use a centralized solution for fan-out instead of competing goroutines. In that case, we would define a master process To distribute each incoming data instance (string in our case) between all output channels.

In fan-in, we’ll basically do the reverse of fan-out with a few differences:

Aggregator The function takes a slice of receive-only input sources and returns a single receive-only output channel. Inside, we created a goroutine for each input source, which continuously reads from the source and populates the output channel (destination) with the obtained data.

Note that we have a . have used sync.WaitGroup To wait for the aggregator goroutines to finish. After an input source (channel) is closed, the loop inside the corresponding goroutine will terminate, and it will complete its task.

After all input sources are closed, we are done with aggregation, and now we can close the destination channel. This is an important stepAnd if we don’t close the channel we created, the Go runtime will complain of a fatal error:

fatal error: all goroutines are asleep - deadlock!

With all of our functions together, we’re ready to run our code:

  • A single processor data pipeline will eventually run into a bottleneck
  • It is helpful to split the input source into multiple queues and do simultaneous processing
  • This split-process-aggregate pattern is called fan-in/fan-out

Thanks for reading.

Want to Connect?

This story was originally posted at

Leave a Reply