We have this one customer with whom we've got a weird relationship which comes down to everyone walking on eggshells around them, and we're trying really hard to not leave anything to chance, make no mistakes and give them no excuse to complain.
A week ago I made a small change to one of our scheduled runs for my script; basically it ran an hour earlier. That shouldn't affect anything regarding its functionality.
Now keep in mind I don't believe in coincidences.
So that same night things started getting freaky. Our separate remote logging service wasn't picking up the script's run at all - it was as if it was never ran in the first place, but I could see on local logs that it did run, and the files that it created and transferred.
Not believing in coincidences I started digging in my off hours to figure out wth happened. Then another item came to my attention; a second separate system which looked into a different database to pull some statistics about the script's activitity broke and it was pulling nothing.
Now I got really worried because two coincidences were too many. I went at it harder, broke the script down and went over it line by line to figure out why the hell changing the time would cause such havoc. This took days, and I could find nothing - but the strange thing is neither did our customer. They had no issues.
And then it came to light that each of those two separate systems broke down around the same time as mine... independently.
So it was a coincidence, but I busted my ass before that was clear.
Randomness fucked me over even after I accounted for it.