This is a second-hand story, so take it with a grain of salt. My first tech job was for a local computer shop owned by a guy who’d been around a bit. This is his story, from before I knew him, with some added flourishes:
A call came in on a Friday afternoon, around 2 pm, from one of the larger customers with a support contract: the network is down, nobody can get to the fileserver. We’re dead in the water, you have to come out.
Andy makes haste to arrive on-site, but it takes a while due to starting on the other end of town. As he’s arriving, the network has miraculously recovered.
These things shouldn’t just solve themselves, but then again they shouldn’t randomly happen, either (but this is back when hardware was touchier than it is now). Everything checks out now, nothing looks amiss – the netware server is running like nothing happened and everyone has file access again. Chalk it up to solar activity or something.
Next Friday afternoon, 2 pm: same call, same problem. And as Andy arrives, the network is coming back to life. Check again, netware indicates no downtime. Clearly something happened, and un-happened before anyone could try to fix it.
This happens a couple of more times, and the Andy decides that this calls for a pre-emptive strike. He clears his calendar on Friday afternoon and shows up at the client’s site just after lunch. He’s going to wait it out.
Now, this is in the days of Netware, IPX, and 10-base2 cabling: one long common circuit of coax cable to join everyone, running at a staggering 10 megabits. Netware is pretty solid but 10-base2 is touchy: the cable must be unbroken and terminated at both ends, or else it doesn’t work. It’s slow because of all the cross-talk but nobody complains because it’s cheap to install and files are still relatively small. Nobody has email and the internet is unheard of.
Precisely 2:05 pm, on schedule, the network goes out. The office is small enough that he can see everyone, and confirm that there isn’t someone doing something nefarious. Just people going about their business – working on documents, having meetings, neatening up the office before the weekend, watering their plants…
It turns out there was a stress fracture in the cable’s sheathing. It wasn’t causing a problem most of the time. But this crack happened to be behind a secretary’s desk, under her new plant.
Every Friday she would overwater that plant, causing the excess to overflow down the back of her desk and over the cracked sheathing, effectively un-terminating the network.
After an hour or so it would dry and things went back to normal.
I originally posted this on reddit: http://www.reddit.com/r/linux/comments/2p6qy5/4_impossible_bugs_any_other_stories_like_these/cmu7hte and realized that it really belongs here. Andy, if you read this, you still owe me some paychecks!
