<h1>Changelist</h1>
- We've been experiencing production issues for a while regarding stalled workers. After a rigorous investigation, we saw there were background threads that kept running. The issue with such threads is that sometimes they were non-daemonic threads, meaning that the Python's interpreter will not exit unless these threads will finish their execution. These threads were stuck in a deadlock and would have never been finished. We've added a background threads protection at the end of the Slave's execution. The protection consists of two phases. The first tries to stop the threads mildly by raising a SystemExit exception in the threads using an internal Python API, then joining the threads for 2 seconds waiting for the thread to finish. If there are unkillable threads left, and slave would kill itself with a SIGKILL resulting in a non-interruptable exit.
- Tests coverage has increased
- Slave logic has been refactored resulting in a clearer more debuggable version
- Supervisor code changes to reflect the Slave's logic change