Random ...
 
December 2017
S M T W T F S
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            
Tags ...
Links



python+zope++:: Don't backup? you should!
Posted at 24.Oct,2010 00:02  Comments 2 / Trackbacks 0 / Like this post!

Last weekend was horrendous. A site I maintain had a fluky disk.

Got it configured for raid 5, so that was good. One of the disk died, and after a reboot, had to e2fsck the disk.

The box booted ok after a few fsck, but lost some data. One of them is my Data.fs. Had a backup of Data.fs.old, but that was also fscked.

If I tried to start zope with it, zope b0rked with CorruptedTransactionError. Tried truncating, but that led to PosKeyError. Truncating more resulted in the same thing.

fsrecover didn't work either. Ended up with a smaller Data.fs, that wouldn't load.

I was doomed.

Lucky I still have an earlier copy that worked. But that copy was too early and we had to do more dev :(

So, the lesson learnt, backup to different disk. Backup to different host. Often.

Now, I've setup up repozo to backup to the same host, and rsync to another host every day.

Hope to not go through this experience again.

And to experience this after 10 years of good uptime and no real crash. Would've loved to eperience this earlier, while developing :P


Bookmark and Share

Is this entry helpful? Comments/Donate/Click some google ads.  
Trackback is http://myzope.kedai.com.my/blogs/kedai/257/tbping 

Comments
Will never happen during development

You'll never experience this during development. It only happen during production, and after very looong years in production :)

Sharuzzaman  at 24.Oct,2010 23:46
true, that

haha. now i'm ok since i know this happened to others as well
but having to tell the boss, is another experience all together :P

kedai  at 25.Oct,2010 15:25
Post a comment