Scaling streams is generally left to the streaming platform itself. Available options vary based on the selected backend.
Replicante Core works on the assumption that streams provide some level of ordering guarantees:
This expectation usually makes it more difficult to scale streams without, at least partial, downtime.
Kafka’s scaling is based on the idea of topic partitions.
Events are streamed to a single topic and partitioned as follow:
This has implications to scaling kafka topics by adding partitions: it is difficult to add partitions while maintain ordering guarantees unless no writes occur and all events have been processed prior to scaling partitions.
The initial implementation of event streams will not provide support for changes to the partitions counts while the system is operating.
While future versions may support online partition scaling they will still be a complex operation. You are advised to conservatively over-provision the number of partitions on event streams to limit, or even avoid, the need to change the number of partitions while the system is running.
At the same time be mindful of over-provisioning too much as that will over-burden Kafka.