Valium hamster lives in a dreamworld, in a cage of his own

[venta]

Well, the hard-drive mentioned in the last post is not looking what you'd call well. Sysadmin #1 has thus far failed to extract anything but melodious beeps from it. Oh, and the odd crunching noise.

Best guess so far is that something has caused the platters to stick, and the inevitable bad shit has ensued. Delicate shaking and tapping have so far failed to free things up; I believe out-and-out brutality is next on the agenda.

Apparently this is a common failure with Maxtor HDs. I didn't know that, and reckon I bought it under the advice of grumblesmurf some years ago. Interestingly, Maxtor apparently claim that their HDs don't bleep, and that it's your PC you're hearing. This is demonstrably a total lie.

So, while not wishing to seem unduly pessimistic, I wondered if anyone wanted to recommend a storage solution to me ?

What do I need... Blimey, HDs have come down in price since I last bought one. So let's assume that size isn't really an issue. The fjord-pining HD was 120Gb and I don't think it was more than half full.

Primarily I listen to mp3s at work (I have the real CDs at home), so I need something that'll plug into my work laptop via USB. If anyone's got any opinions on the virtues of things sold as USB drives (like the Western Digital MyBook) versus just buying a HD and a USB caddy for it, please speak up.

Also, any specific recommendations of makes to go for or to avoid ? Maxtor have incurred my wrath; the caddy also came from what turned out to be a well-dodgy retailer on eBay. It was extremely shoddy, had no ventilation, and an illegal plug, so I won't be doing that again.

And the knotty subject of back-ups. Yes, stable doors, etc. But faced with the task of ripping hundreds of CDs I don't want to do it again again. To me it makes sense to have two HDs, one at home and one at work, thus allowing each to use the other to back up files of all kinds. My home PC is also a laptop, so will probably want some form of external HD as well - but the question is how to transfer easily between them.

Obviously I can, once I've ripped my music, transfer the bulk of it all in one fell swoop. But if I'm adding files to both my home and work HDs, I'll want to do some form of incremental copying of changes in both directions. On the whole, my two laptops are never awake at the same time, and if they were I have no idea how to access one from the other, which makes any form of over-the-air transfer a bit painful. My work laptop goes home each day, so could perhaps transfer some sort of patch in each direction ?

Is this a problem already solved ? My own scripting skills are a bit rubbish, so it might be beyond me to roll my own. Or is there a better approach to ensure I can store masses of files at work, small amounts of files at home, and be reasonably backed up ?

Comments

[timeplease.livejournal.com]

Hardware: have you considered a 2.5" drive and enclosure? These are mostly quite happy to run off the USB power supply without needing an external supply, which may be more convenient for you.

Software: have you looked at rsync?

[hughe.livejournal.com]

yes, avoid crapxtor. i think most makes have gone through a phase when they made a bunch of bad hard disks and everyone swore never to by them again.

i've always thought that if i dont go for the latest/very best capacity, then that might mean a couple of bugs have been ironed out? though that might just be superstition.

i have two hard disks of different makes on my home computer and I rsync between them every night.

(rsync only coppies the changes, so if you've added 3 mp3s and changed one then only those will be coppied)

[venta.livejournal.com]

Hardware: good point. The last HD came out of my desktop machine at my old place of employment so was a "normal" size. I hadn't considered that I could look at other sizes.


Software: rsync would do the job, but I've no idea how to get the two machines to talk to each other.

[bateleur.livejournal.com]

You may recall I used to have endless problems finding a satisfactory way to do backups. The solution I eventually adopted was Amazon S3 (by means of Jungle Disk), which would be too expensive for your MP3s I expect (it's $0.15/GB/month). However, during my investigations I did manage to determine that long term backup is a completely unsolved problem.

As such I'd recommend going for something relatively cheap and cheerful and just making sure you always have multiple copies.

[keris.livejournal.com]

I got something like this,
http://www.dabs.com/productview.aspx?Quicklinx=4HMN&CategorySelectedId=11157&PageMode=1&NavigationKey=11157,50410
It's small and portable and travels with me. I don't expect it to last years what with being shoved in my bag while I cycle to work and back but it's holding up fine so far!

[undyingking.livejournal.com]

If they're never both on at the same time, then I suspect you're going to have to use your work laptop as a third copy, which can be synced with both of them. I don't know of a way that you can use it just as a rolling increment.

Or use an online backup as suggested by bateleur -- I also use JungleDisk, but not for MP3s as it would be rather costly. But there are eg. free things that use a GMail account as an online storage device.

[broadmeadow.livejournal.com]

I also have an external drive at work for my music collection. It is 2.5" and USB-powered and looks very much like this one: http://www.ebuyer.com/product/113876. In order to protect against failure I go for multiple copies; the master copy is on my home PC and the work copy is a backup. To sync between the drives I use robocopy from Microsoft's Windows Server 2003 Resource Kit Tools (also suitable for XP; downloadable from http://www.microsoft.com/Downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en) which, like rsync, only copies files it needs to (robocopy /mir srcdir\ tgtdir\ does the trick). To actually make the copy means taking the drive home from work to do it.

[broadmeadow.livejournal.com]

This is essentially what I do between my servers (one of which is just a clone of the other and can be switched in if the main one fails or I want to work on it), but be warned: it is possible for a drive to fail in a way that makes it appear to just be empty but otherwise ok. When that happens, rsync helpfully erases the copy to match. Ask me how I know this.

[venta.livejournal.com]

I'd recommend going for something relatively cheap and cheerful and just making sure you always have multiple copies.

This is the plan. However, the how of the multiple copies is what's currently baffling me.

[venta.livejournal.com]

So, Broadmeadow, how do you know that ?

[venta.livejournal.com]

That looks like a good bet, actually! Still doesn't solve the how-to-do-backups problem, but is the kind of thing I think I want.

[bateleur.livejournal.com]

I've never been much for clever backup software myself. I favour copying files into directories and copying the directories to the backup device and then calling the job done!

(This only gets messy if you have files which you update often and are fairly large but you don't also want to be holding backups of earlier versions.)

[keris.livejournal.com]

It comes with its own software - Ceedo, which I haven't found the point of yet, and Freeagent, which supposedly does back-ups (but is a bit slow over USB1)...

[broadmeadow.livejournal.com]

Well, it went a bit like this. Just as I was off to bed I noticed my websites were not working, investigated a bit and discovered that the drive in the server which contained all the web content was empty. So I switched everything over to the backup server and went to bed, happy knowing that it could wait until morning and chuffed at my strategy of having a backup machine. But then shortly after the main server started up its overnight rsync jobs and erased all the web content on the now-live backup server. How I chuckled when I realised the enormity of my ineptitude the next morning. Fortunately neither server had the master copies - that was on my desktop. And there were DVD backups. And it was also all mirrored offsite on my desktop PC at work ....

... although this smugness can only lead to a very big fall, probably very soon.

[broadmeadow.livejournal.com]

As you use Windows I refer you to my previous comment about the Windows Resource Kit Tools. You could plug both your external drives into the same machine periodically and update the mirror from your master copy with a single DOS command. The only real downside to this strategy is that if you unknowingly erase or corrupt files from your master copy you'll end up propagating those problems. As you'll presumably only ever be adding to the mp3 collection you could devise some fairly straightforward script which checksums the files and diffs the results against the last run (which should show up only the additions you've made in the meantime) before you make a backup.

[shrydar.livejournal.com]

We've got a pair of external firewire drives for backups (a LaCie BigDisk and an IceCube IIRC), and at any given time at least one of them is in a bank vault. We changed to that offsite location when our laptops were stolen the same week there was a burglary at the house we'd previously kept our offsite backup at - fortunately for us the drive was not stolen, though other items from the same room were.

At the moment our backup procedure consists of creating fresh disk images of the contents of our respective home directories, which at 30-80GB is very much a 'start it up before going to bed' procedure, though firewire 800 helps (at least, on J's laptop. Mine is slower). I keep meaning to look into using rsync instead, but have never quite got around to it.

I've transferred a subset of my mp3 collection to work using a 4GB USB key, but that copy is considered entirely expendable, and is not part of my backup strategy.

The rips we've done from our DVD collection and the stuff we've recorded from free to air is as vulnerable as your mp3 collection was. One media drive, not backed up.

[rabbit1080.livejournal.com]

... the media drive has DVD backups of hard-to-replace stuff, but most of it isn't backed up.

[broadmeadow.livejournal.com]

(The bit about checking the drive integrity is possibly overkill in this case. If a file gets corrupted you'll notice eventually and you'll still have the original CD to restore from.)

BTW - this business with having a master / backup drive is probably most useful if you want to be able to access your CD collection on your PC at home as well as work. If you don't then our work notebooks can write DVDs so you could just backup to them - having essentially the same setup as you used to use, with DVDs to restore from if needed.

[hughe.livejournal.com]

ah thats interesting to know...

I put a check in for that....

[venta.livejournal.com]

I had considered just writing mp3s to DVD. However, I acknowledge that I'm pretty poor about backing up files at home as well, and though there are far fewer of them I'd probably be more upset if they were lost. There are plenty of solutions for backing up a few text files, but I don't do them. Hence the wish to evolve some sort of system to ensure I actually do my backing-up.

Thanks also for your emails about drives and such.

[broadmeadow.livejournal.com]

I have a slightly different strategy for backing up system drives on my non-server (Windows) PCs at home; it may be a more useful one to consider for your notebook backups. The problem here is not only the possibility of a drive failure, there is also the strong liklihood of Windows getting itself in a complete mess. Rebuilding a Windows PC takes ages what with all the Windows updates, configuration and reinstallation of apps that you have to do. What I do here is obtain a second drive of the same type and connect it via a USB adaptor (essentially an external case without the actual case) and use a program called Casper XP (a cheaper thing that does the same thing as Norton Ghost - hence the name!) to make an exact clone of the drive, and then store it away somewhere safely. If needed the backup drive can then be swapped in to restore the machine to the last "good" state. However, I don't update this clone very often and still use a separate backup policy of periodically mirroring important directories onto my server (or an external USB drive would do) more frequently.

Going slightly off at a tangent, has anyone here used a NAS device rather than a USB-connected external drive? A problem I have is that my desktop PCs are running Windows and the servers Linux, and they handle timestamping differently. There is a different granularity in the times stored but robocopy has an option (/fft) which makes it allow up to a 2s difference in modification times to be considered "equal" which resolves that. The problem comes when the clocks change - they handle this differently and see a 1 hour difference in modification times between pre-existing and identical files, so _everything_ gets updated. Now, NAS devices generally seem to run Linux and use the Linux ext3 filesystem so I presume would have the same issue, but can anyone confirm this?