Data Redundancy/Backups - how are people managing this? Share your system

[nikko1974]

Hi,

I thought I'd start a thread on discussing how people are backing up/put some form of redundancy in place with their data & also systems.

Most of us will have suffered a loss of some important data during our lives and there's generally nothing better to make you understand the value of a good backup system than exactly this....how ironic!

Personally I lost around 5 HDDs a few yrs ago due to a faulty PSU - and lost a LOT of data some of which was personal material (eg photos, tax retrun stuff) that was either a real PITA to do again or irreplaceable.

With seemingly more and more data these days people get confused on how they should backup, how often, to where etc etc - they just know they should but often never do as they're so uncertain. Or alternatively they do backup but the system they use is so cumbersome that its very inefficient or restrictive. So hopefully this threads atleast offers a few easy options or perhaps a new idea....worst case tells you you're a genuis for doing it the perfect way!

SO PLEASE POST UP HOW YOU'RE MANAGING DATA REDUNDANCY/BACKUPS IN YOUR SYSTEMS & ANY OTHER APPLICABLE THOUGHTS.

Try and outline:

• Basic structure of your systems
• Amount of data you backup
• Software utilised
• Hardware utilised (for backups)
• Regularity of backups
• Thoughts on your system - why you chose it - how easy to manage etc?

Also the odd tale of woe over lost data is always appreciated (shared grief and all that!).....So no set way to reply - just post up anything that you think is useful to others.

:-)

[dgaust]

I basically have 3 tiers of redundancy - maybe a bit too much, but hey what are you going to do.

Level 1

An AMD 4850e based server, with 2x dedicated hardware RAID controllers. They're not top end controllers, but they do a reasonable job with each controller running 4x1tb hdd's in RAID5.

I also have another 2x tb drives in the server that don't hold critical information and 2x 500gb drives in a RAID 1 (using the mobo RAID) for the OS.

The server is my media storage machine (and torrent box), and most media is stored on the RAID arrays. Family pictures and video are also stored there.

Stuff I can stand to lose goes on the non-critical drives.

Level 2

A WHS machine. This machine is very low end, an AMD Sempron 3100+ I believe with 1gb of RAM.

This machine stores nothing but backups of the OS partitions on all my PCs at home (last count there were 6) as well as all the family photos. This machine is stored in another part of the house to the media server.

The WHS machine is only awake from 1am to 4am to do the backups before it puts itself to sleep.

Level 3

Carbonite - I use this to backup (again) my family pictures and stuff I really don't want to lose.

This is a last resort system, as I'd prefer the local backups, but you never know if you're house is going to burn down.

I basically chose these systems for their ease of use, everythings automated, redundancy is handled by the RAID controllers, the WHS controls the backups and the carbonite software handles the uploads. It's all hands off, and I like to think (I hope) bomb proof.

[ozbear]

My development system I back up to tape nightly, usually around 1.5 GB. I choose tape because it is pretty much infinitely extendable, reliable, reusable, and cheap. The backup software is Retrospect so everything is in its schedule (which machines to backup, when, and where to) and manages to tape catalog. In the event of an emergency, say a fire, I only need to grab the small box of tapes and the cat and then out the door. I just the load the tape it will need to perform the backup when I am done for the day. The backup will run sometime overnight and be finished in the morning.

Having a backup has saved my arse more than once, and not because of any sort of virus but because of human error, like deleting the wrong file, reverting last week's changes to documents, and so forth.
Oz

[nikko1974]

Yes, isn't it true Ozbear - the backups are most to save you from yourself!!!

Wow Dgaust....nice system setup indeed - goodness me it'll take a cataclysmic event to have tou lose a single byte of data! 6 systems in the house though! Wow - that power bill must be a killer.

Very impressed...haha slowly tapping mine out - but gawd it pales when compared to that.

[patto]

OK this is how I manage data:

Main file server (XP Pro box) 1 TB mirrored array (1st level of redundancy) all data is stored in directory called "DATA"
emails are on a exchange server (offsite) and POP3 emails are on a laptop and main machine.

HTPC - this has a spare 400GB drive which the contents of "DATA" (excluding movies) are copied each night to using a robocopy bat file (2nd level of redundancy). It is essentially mirrors the files system with only changes copied each night, takes less than 2 mins to run. About 250GB data in the backup. I don't do a seperate backup of movies as I still have discs.

Operating systems are only manually backed up by me when I remember and generally stored on different machines around the house.

I have just ordered a 5.25" SATA hot swap tray so I can use the spare HDD's floating (720GB combined) around the house for backups to create my third level of redundancy. I plan to use these around once a month for backups, hopefully include operating system and data in this step.

I use robocopy as it is easy to get the data back (file structure is unchanged), it is quick and reliable. I found with the windows backup system it is slow, I don't understand it very well and I haven't found it reliable and more difficult to get the data back. I have been using ROBOCOPY to backup work data across a VPN on the internet each night to my house onto an external hardrive and that takes around 20mins to run if there is no major data to transfer (that is 4th level of redundancy for the office). This has been reliable for 3 years. Every time I check to log files it has always run.

Luckily I have not been subject to data loss yet. My main worry is fire or theft - this could wipe me out completely!!! I have been thinking about getting a fire resistant safe to store the monthly backups (it doesn't help if someone steals the safe though).

Patto

[nikko1974]

Ok - I'll lead off - am just making the reply structure up so suffice to say reply as you like. ;-)

SYSTEMS
Pretty simple house compared to others I'm sure. 2 systems. One is the Main PC - thats on 24/7 and acts as the data storage for the dedicated HTPC, a torrentbox and all the other general stuff thats needs to be done from video encoding to just browsing the net or playing the odd game etc. That has 2 HDDs - one for the system & general work/nonsense/usage - the other is a 1tb with all the data on it for streaming to the HTPC.

I only keep the movies on there that I haven't watched yet or know that I regularly like to watch - as I've just too many (around 1400) - but all my music is on there (about 7500 tracks).

The other system is a dedicated HTPC - its networked to the Main PC via CAT5 ethernet.

AMOUNT OF DATA
Hmm well I've around 800 DVDs (all dvd5's - and on great quality media (Taiyo Yuden))...so with them there is no redundancy - as I can't be bothered burning extra copies or ripping to HDD as it'd take up too much room. I lost a bunch of them a few yrs ago as they were burnt on poor media (Sony,TDK, Laser) and they've all be reburnt onto very good archival media. Are stored according to DVD archive reommendations (upright position, in folders, cool/dark area). Everynow and then I'll run Nero CD-DVD tool on them and check the PI scans for errors - but they're always very good.

I've also got 600 or so compressed video files (eg .avi, mkv etc) - I'd imagine around 1.5tb in all. 100gb or so of music - and prolly the same of other files.

BACKUP TECHNIQUE/STRATEGY
I guess I have 2 levels of redundancy. For all my personal files and music I keep those on my main system and also back them up every few weeks onto an external 1tb HDD using a HDD dock(esata) and Karen's Replicator. So there are always 2 copies of these files at all times (both on HDDs).

With the DVD5's - no redundancy.

With the compressed video files any favourites or not watched material is on the main PC....in addition all of them are burnt to quality DVD media (as files - not converted or anything) and stored as the DVD5's are.

In addition, I've just added an extra layer - with every compressed video file being 'archived' to 1tb HDDs - which will be written to only once. These will only have material added to them - never removed. These are stored in static bags with moisture absorbers in a secure location.

I had bought brand new HDDs for this but then was told by a guy that was silly and to use any other HDDs I had that were a few months etc old as they'd BASICALLY gotten out of that 'potential DOA/new HDD failing period'.

Which is true....as if I'd just written all the data to them and stored - who's to say the flaw in the HDD would be shown up until the 3rd or so time I use them (maybe the equivalent of a few days for a HDD within a system). So there are 2 archiving HDDs - both 'proven'.

I use 5400rpm HDDs for all my external and storage HDDs.

For the system partitions themselves, I use Acronis True Image 10. I only ever make full backups (never incremental or that other one).....and I only make these by booting from the rescue disc and making the image from here. Have never had an image fail on me.

I'd make new image of each system anytime that I make considerable changes to it. In the past I deleted a lot of the images but now I will keep all of them as I find the ones you think are useless often can be handy - particularly with tinkering on the HTPC.

SOFTWARE USED
- Karen's Replicator: Good synching software - used to copy 'backups'. Can be run to replicate deletions - matches files perfectly - setup profiles etc - well regarded by users.
- Acronis True Image: only from the boot CD though _) I found the full version and whole 'secure zone' thing a pain and restrictive.

HARDWARE USED
- Vantec NexStar external esata HDD dock
- 5400rpm 1tb HDDs (Samsung and WD)
- Quality DVD media (Taiyo Yuden)

TIPS
- Never fully trust a HDD until you've used it for a reasonable period - as if its going to fail the risk is highest in the early days. So partition it the SLOW way - not a 'quick partition'. And maybe even consider running the manufacturer's diagnostic tester on it (Seagate's Seatools runs on all HDDs)
- Get to understand what DVD media ID codes are...make sure you're using decent ones - as the brand on the packaging doesn't necessarily mean anything.
- Make your backup system easy to run, otherwise you won't do it and its essentially worthless.

I need to do somekind of offsite backup for my most vital stuff as Dgaut has. Might have to look into that soon. I've resisted going down the path of a RAID solution as all the computer folks I speak with endorse this only if you're prepared to invest in the proper hardware and have a dedicated controller etc - and I can't justify that with just 2 systems in the house. Also hear of RAID arrays gone wrong so personally I apply the KISS rule on that one.

So while I know the system isn't perfect I feel the minimum 2 levels of redundancy it offers (sans the DVD5s) is solid in those levels. Its also very cost effective & flexible - in terms of portability and the amount of space it takes up. Impressed by some other people's setups though.

:-)

[ozbear]

TIPS
- Never fully trust a HDD until you've used it for a reasonable period - as if its going to fail the risk is highest in the early days. So partition it the SLOW way - not a 'quick partition'.

Did you mean "format" instead of partition?

Oz

[nikko1974]

Did you mean "format" instead of partition?

Oz

Yes, so when you're making a partition the tendency of many people is to select the 'quick format' option - instead of the slower one which will check for dead sectors etc.

:-)

[pantonious]

I have no redundancy :P

my file server uses a Dynamic Span of disks...that way i can add any size/brand of disks i like to it, i don't have to keep it all the same. (sorta like the WHS storage pool)
Offers no data protection. I've been told that if i loose 1 drive i loose everything, but i don't think thats true...i would loose the data on THAT drive, but i think it can 'rebuild' itself without the dead drive

There is nothing 'critical' on my server at the moment. If i were to loose data, it would be annoying, but not devistating!

Im not one for photos and home movies (at the moment).
If that starts happening, then im sure i can think of something

[Mike]

I basically have 3 tiers of redundancy - maybe a bit too much, but hey what are you going to do.

I'm with you...

[sort of] 3 tiers here as well

1. MediaServer (3TB - WHS) holds all my stuff - photos, music, movies, TV shows, documents, website backups etc. etc.

2. MediaBackup (3TB - XP Pro) 3 year old Dell machine is a mirror of Mediaserver and has multiple drives set up as a spanned volume. This server is literally buried under a pile of crap in the corner at the back of the garage with disguised cabling running to it in case the house gets burgled and they clear off with the obvious targets. This mirror setup runs Goodsync daily via a cron job so that everything on "MediaServer" get synced to "MediaBackup".

3. Carbonite - runs continuously and backs up selected dirs containing all the really important stuff (currently 38gb) to a remote server in case the house burns down - just like dgaust

Not sure what else I could do, but a number of really bad experiences a few years ago "persuaded" me to become pretty anal about backups - and so far so good.

Mike

[arkay]

I have no online data redundancy at all. All drives are concatenated to give me maximum storage space.

Setup includes:

1 x Linux based server with 4 tb storage (all media centralised including nzb downloads)
7 x other Linux based PC's about the house, each with internal boot drive but no sensitive data.

The 7 pc's I could care less about. I image backup the main HTPC front end only before making any changes to it. The server houses all recorded TV / Episodes / personal files / mp3's and mail for all users around the house. The Pc's mount data volumes from the server so there is no local files on any PC of any importance.

The server has an LTO 3 tape drive connected to it. I use HP Data Protector to back my data up. Data is separated into two categories:

1. Important cannot be lost data. Backed up incrementally at my discretion, about once a month (or on addition of new photo's/documents). The quantity of this data is around the 200GB mark so it fits on a single tape. I make this backup twice and the second tape is stored in my desk drawer at work.

2. Everything else. Mostly TV/Movies/Games is backed up to tape as I see fit. Usually I will back up an entire series on completion of watching it. It is then removed from the drive. Any static media is backed up in group 1. Transient media (such as tv series that I won't watch again for some time), is put on tape and restored next time I want to watch it, then deleted again afterwards.

The tape is failry cost effective and is enterprise grade storage. It's easy to work with though the initial layout of the drive can be expensive but for personal use you will never need another drive. LTO3 can store 400GB per tape uncompressed (800 GB compressed).

OS level image backups of the HTPC and other machines are stored on the server and are captured in the incremental backups in 1 above. Image backups of the server itself back up to a share on my desktop machine.

Anything on the server that has yet to be watched and is not backed up is of no consequence to me if I lose it. I can just get it again from the same source.

Cheers,

Arkay.

[GlenR]

Backups are basically insurance, and as such you can pay as much as you like to get the security you like. I've gone down a middle appraoch for mine.

Core in a WHS machine that backs up my workstaions and HTPC as well as keeps all my media and other important stuff. File duplication is enabled to allow protection against a HDD failure.

The HTPC and my two main PCs (mine and SWMBO's) are backed up daily buy WHS. the other couple of unimportant ones have no valuable data so they are backed up only if there is a major change. Rare as these are fairly static builds. The only stuff not backed up is the Recorded TV volume of the HTPC. I consider this expendable data, and no big deal if it's lost. Anything I want to keep is edited to remove ads, converted to XviD and put on the server.

The server is further protected by a UPS to ensure it shuts down cleanly in the event of power fail. Got hit by this once (fortunately no really vital data lost).

No off-site storage as yet. Basically, if I need to I can just pick up the server and go to save all my data.