Distributed Locks in Multi-DC: How to Stop Duplicating CronJobs

In this article, Timofey Saltymakov from Tochka Bank addresses the challenge of CronJob duplication when deploying applications across multiple data centers (Multi-DC). The author explains that standard Kubernetes mechanisms do not prevent the same task from running simultaneously in different clusters, leading to synchronization conflicts. The piece details the team's evolution: from using database advisory locks to developing their own custom library. The key takeaway is a shift in paradigm: instead of trying to synchronize the processes themselves, the team focused on synchronizing task execution time. The article provides a practical look at architectural solutions for ensuring idempotency and fault tolerance in distributed systems, highlighting the importance of selecting the right tools for managing background processes in complex infrastructures.