When logs scream and you don't hear: how I put logs in order in an ETL connector

In this article, the author shares practical experience in optimizing a logging system using a Python connector for Airflow as an example. The focus is on the challenges developers face when debugging ETL processes, where unstructured or redundant logs make troubleshooting difficult. The author analyzes metrics from before and after implementing changes, discusses controversial aspects of choosing logging levels, and proposes the concept of a 'log map' as a contract between system components. The article is useful for data engineers who want to make their pipelines more transparent and easier to monitor. The main takeaway is that high-quality logging is not just about recording events, but a tool for effective incident management and improving the reliability of software products.