My final year taking a B.Sc. I'm writing up my Distributed Systems project, the day before handing it in. It's on top of Transis, and source code is "stored" in RCS (yes, I'm that old). The project is a reliable system administration tool, that performs the same action across a cluster with guaranteed semantics.
I'm very proud of the semantics, but cannot figure out why the subdirectory installation stuff works almost but not quite. Here's my sequence of actions:
1. Install across all machines.
2. Manually see it's broken.
3. "rm -rf *".
4. Repeat.

What in to discover is that the subdirectory installation always finishes off in a current directory 1 level higher than where it started. Oh, and the entire cluster sees my NFS home directory. Oh, and I'm running each cluster member in a deep subdirectory of my dev directory. Oh, and my RCS files live in a subdirectory of my dev directory.

All of a sudden, my 5 concurrent "rm -rf *"s were printing weird error messages about ENOENT and not being able to find some inodes. In a belated flash of brilliance, I figure out all the above, and also that I've just deleted my dev directory. 5 times, concurrently. And the RCS files.

That was the day a kindly sysadmin taught me than NetApps have these .snapshot directories.