Extending flink state serialization
Operations with Flink state are a common source of performance issues for a typical stateful stream processing application. One tiny mistake can easily make your job to spend most of a precious CPU time in serialization and inflate a checkpoint size to the sky. In this talk we’ll focus on a Flink serialization framework and common problems happening around it:
- Is Kryo fallback is really that expensive from the CPU and state size perspective?
- How to plug your own or existing serializers into the Flink (like protobuf).
- Using Scala sealed traits without Kryo fallback.
- Using custom integer variable-length encoding and delta encoding for primitive arrays to further reduce the state size.
Presented at Flink Forward Europe 2019