Roman Grebennikov

Roman Grebennikov

A personal blog & info page

09 Oct 2019

Extending flink state serialization

Operations with Flink state are a common source of performance issues for a typical stateful stream processing application. One tiny mistake can easily make your job to spend most of a precious CPU time in serialization and inflate a checkpoint size to the sky. In this talk we’ll focus on a Flink serialization framework and common problems happening around it:

  • Is Kryo fallback is really that expensive from the CPU and state size perspective?
  • How to plug your own or existing serializers into the Flink (like protobuf).
  • Using Scala sealed traits without Kryo fallback.
  • Using custom integer variable-length encoding and delta encoding for primitive arrays to further reduce the state size.

Presented at Flink Forward Europe 2019

Link | Video | Slides