SideInputs materialize the entire join dataset to every worker, so once the joined data grows beyond the recommended size (~100 MB in streaming, under ~1 GB in batch) they become the bottleneck and slow the pipeline. Google's own Dataflow use-case guidance says to use SideInputs only for small lookup tables and to switch to a CoGroupByKey (a shuffle-based join) when both collections are larger, which distributes work evenly across workers. Switching to Avro (A) helps splittability of the source read but does not address the join cost that is described as the problem, reducing batch size (B) hurts throughput, and retrying errored records (C) adds work rather than removing it. Therefore replacing the SideInput join with CoGroupByKey is the correct optimization.
Community Discussion