Technologies
Back
Data & Analytics

-1000% occupancy: how I found holes in my own published dataset

Habr
Advertisement468 × 90
-1000% occupancy: how I found holes in my own published dataset

The author shares their experience of discovering critical errors in a published open dataset regarding parking occupancy. After an anomalous occupancy value of -1000% was recorded, an investigation began, revealing serious data quality issues. During the process, the author found that standard recalculation methods were ineffective, and finding the missing denominator required a non-standard approach. The article details why automated checks, which had been running for a year and a half, failed to detect the anomaly in time. This case serves as an important reminder of the need for thorough data validation before publication and highlights that even tested systems can harbor hidden defects. The author analyzes their mistakes and offers lessons to help other developers and analysts avoid similar situations when working with large datasets.

This is a summary. Read the full article at the original source:

Habr
Advertisement468 × 90
Share
Data & Analytics

Related stories

Advertisement970 × 250