Skip to content

Concurrency ​

goflow provides several primitives for parallel stream processing. All concurrency operators respect context cancellation and propagate gofuncy.GoOption settings for tracing and error handling.

FanOut / FanIn ​

The FanOut/FanIn pattern distributes work across multiple goroutines and merges the results back into a single stream.

FanOut distributes elements round-robin across n output streams. FanIn merges multiple streams into one, with elements arriving in non-deterministic order.

FanOut(3), then n * n on each part, then FanIn
go
ctx := context.Background()

s := goflow.Of(ctx, 1, 2, 3, 4, 5, 6, 7, 8, 9)

// Fan out to 3 workers
partitions := s.FanOut(3)

// Apply a transformation to each partition
transformed := goflow.MapEach(partitions, func(ctx context.Context, n int) (int, error) {
    return n * n, nil
})

// Merge results back
result := goflow.FanIn(transformed).Collect()
fmt.Println(result) // order is non-deterministic

WARNING

FanOut sends to unbuffered channels. It blocks on the slowest consumer. If one partition is not being consumed, the entire upstream pipeline stalls.

FanMap -- The Shorthand ​

FanMap combines FanOut, MapEach, and FanIn into a single call. It is the most common way to parallelise a transformation.

FanMap runs fn on 3 workers and merges the results
go
results := goflow.FanMap(s, 4, func(ctx context.Context, url string) (Response, error) {
    return httpGet(ctx, url)
})

This is equivalent to:

go
results := goflow.FanIn(goflow.MapEach(s.FanOut(4), fn))

FanMapFilter does the same but with a MapFilter function, allowing you to skip items during the concurrent transformation:

FanMapFilter runs fn on 3 workers, skips b and merges the results
go
results := goflow.FanMapFilter(s, 4, func(ctx context.Context, id string) (User, bool, error) {
    u, err := fetchUser(ctx, id)
    if err != nil {
        return User{}, false, nil // skip on error
    }
    return u, true, nil
})

Process -- Parallel Consumption ​

Process is a terminal operator that consumes the stream with n long-lived workers. Unlike FanMap which produces a new stream, Process consumes the stream and collects errors.

Process runs fn on 4 workers at the same time
go
err := goflow.Of(ctx, urls...).Process(8, func(ctx context.Context, url string) error {
    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    return saveResponse(ctx, url, resp)
})
if err != nil {
    log.Fatal(err) // errors.Join of all worker errors
}

Process runs n workers in a gofuncy.Group; it does not start one goroutine per element. Configure the group with goflow.WithGroupOptions(...). All errors are collected and returned via errors.Join, so you see every failure rather than just the first. For long-lived streams, pass goflow.WithErrorHandler(h) instead: errors go to h and are not retained.

Tee -- Broadcasting ​

Tee sends every element to all n output streams. This is useful when you need to process the same data in multiple ways simultaneously.

Tee(2) sends every element of [1, 2, 3] to 2 output streams
go
copies := s.Tee(2)

// Consumer 1: log everything
go func() {
    copies[0].ForEach(func(_ context.Context, item Item) error {
        log.Printf("received: %v", item)
        return nil
    })
}()

// Consumer 2: process items
err := copies[1].ForEach(func(ctx context.Context, item Item) error {
    return process(ctx, item)
})

WARNING

Tee blocks on the slowest consumer. Every output stream must be actively consumed, or the pipeline will deadlock. If consumers run at different speeds, consider inserting a buffered Pipe between Tee and the slower consumer.

Tuning with WithOptions and Buffer Sizes ​

For high-throughput pipelines, you can tune behavior with buffer sizes and gofuncy options.

Buffer sizes on Pipe reduce blocking between producers and consumers:

A Pipe with buffer 2 accepts a and b; Try(c) returns ErrFull
go
send, s := goflow.Pipe[int](ctx, goflow.WithBuffer(1024)) // 1024-element buffer

WithOptions propagates gofuncy options (error handlers, tracing) to all downstream operators:

go
s := goflow.Of(ctx, items...).WithOptions(
    gofuncy.WithErrorHandler(func(err error) {
        log.Printf("stream error: %v", err)
    }),
)

Real-World Example: Parallel HTTP Fetching ​

FanMapFilter fetches 3 URLs at the same time and skips the failed one
go
package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"

    "github.com/foomo/goflow"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    urls := []string{
        "https://example.com/api/1",
        "https://example.com/api/2",
        "https://example.com/api/3",
        "https://example.com/api/4",
        "https://example.com/api/5",
    }

    // Fetch up to 3 URLs concurrently, skip failures
    bodies := goflow.FanMapFilter(
        goflow.Of(ctx, urls...),
        3,
        func(ctx context.Context, url string) (string, bool, error) {
            req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
            if err != nil {
                return "", false, nil
            }

            resp, err := http.DefaultClient.Do(req)
            if err != nil {
                return "", false, nil // skip failed requests
            }
            defer resp.Body.Close()

            body, err := io.ReadAll(resp.Body)
            if err != nil {
                return "", false, nil
            }

            return string(body), true, nil
        },
    )

    results := bodies.Collect()
    fmt.Printf("fetched %d responses\n", len(results))
}