A customer sent malformed data, leading to unnoticed ClickHouse insert errors.
Lack of monitoring for asynchronous inserts prevented early detection.
We did the following measures to resolve this issue:
- Implemented stricter input validation on the usage endpoint.
- Deployed fixes, restoring expected metric levels.
- Integration tests and alerts for asynchronous insert errors were added.