Last weekend was horrendous. A site I maintain had a fluky disk.

Got it configured for raid 5, so that was good. One of the disk died, and after a reboot, had to e2fsck the disk.

The box booted ok after a few fsck, but lost some data. One of them is my Data.fs. Had a backup of Data.fs.old, but that was also fscked.

If I tried to start zope with it, zope b0rked with CorruptedTransactionError. Tried truncating, but that led to PosKeyError. Truncating more resulted in the same thing.

fsrecover didn't work either. Ended up with a smaller Data.fs, that wouldn't load.

I was doomed.

Lucky I still have an earlier copy that worked. But that copy was too early and we had to do more dev :(

So, the lesson learnt, backup to different disk. Backup to different host. Often.

Now, I've setup up repozo to backup to the same host, and rsync to another host every day.

Hope to not go through this experience again.

And to experience this after 10 years of good uptime and no real crash. Would've loved to eperience this earlier, while developing :P