Tracking down a device service that locked up at midnight

Problem

In one customer environment, the device service that connects to the screens stopped accepting new connections overnight. The screens couldn’t receive content updates; in the morning, restarting the service fixed everything. There was no infrastructure failure; the fact that a restart solved it showed the problem was inside the application itself.

What happened?

Tracing the logs backwards revealed a chain of events:

• At midnight, the storage for the daily logs was rotated and a small setup step ran. That step ran inside a lock that held up every incoming request, and it couldn’t recover when it failed.
• Over a few hours, the waiting requests exhausted the server’s resources; the database and cache connections stopped responding.
• At the same time, a scheduled job that kept timing out produced logs that grew with every failure, writing gigabytes of logs within a few hours, and into the application’s own database at that.

Fixes

• We moved the setup step out of the request path; even if it fails, it no longer blocks requests or retries endlessly.
• We put log uploads from devices into a bounded queue. Under load, a device retries shortly after, and the server doesn’t get overwhelmed.
• We removed calls that made requests wait unnecessarily.
• We added an in-memory cache for frequent device queries.
• We capped log size so a single error can’t turn into a log storm.

The next step is to separate logs from the application’s database and add alerts on resource usage.

Lessons learned

• Never do slow work inside a lock that holds up every request, and plan for the failure case too.
• Writing logs to the same database as the system they monitor ties two problems together when an incident hits.

My contribution

I did the log analysis, found the source of the problem and implemented the fixes.