Yesterday I was stuck in a hellish debugging cesspool – you know what I mean – when you are debugging a mysterious crash right down embedding line by line print statements and nothing is making sense. And I don't mean that its not making a lot of sense; I mean that its making NO DAMN SENSE. I was debugging on a GPU instance on AWS that was running a TensorFlow application and what I kept seeing was output appearing before the routine invoking it was executing. And this was a worst case scenario right down to the very first line of my Python code being:
Happily I was pairing with my colleague Grant and he did that wonderful pairing step of "this makes no damn sense; let me apply my Google Fu to this" and he discovered that Docker buffers output at times and he recommend this change to the Dockerfile:
# rest of Dockerfile appears here CMD [ "python3", "./main.py" ]
# rest of Dockerfile appears here CMD [ "python3", "-u", "./main.py" ]
Here is the relevant Stack Overflow post.
Thank you Grant!