From: Gordon Henderson on
In article <gigcjh$pfg$1(a)localhost.localdomain>,
Martin Gregorie <martin(a)see.sig.for.address.invalid> wrote:
>On Fri, 19 Dec 2008 13:45:43 +0000, Ian Rawlings wrote:
>
>> Can you not use rsync instead of your .tgz backup?
>>
>I could, but I wanted to use compressed backups. The TGZ files are 8.8 GB
>vs a tad under 15 GB uncompressed and the USB disk is only 120 GB and I
>wanted to automatically fit as many backups as possible onto the disk.
>Using compressed archives and a big empty container gives the easiest way
>of doing this.

Slightly harder to restore though.

The multi-day rsync method:

Do the first one by hand:

mkdir /backupdevice/00
rsync --stats -aHx --delete /path /backupdevice/00

Then daily:

cd /backupdevice
rm -rf 99
rename 98 to 99
rename 97 to 98
etc. (obviously in a loop)
rename 00 to 01

cp -al 01 00
rsync --stats -aHx --delete /path /backupdevice/00

So while not compressed, each day will only store what gets changed,
so you should very probably be able to store even more copies using this
method than compressed backups.

What you end up with is NN copies of the data, each from
the day before. Files that don't change are hard linked to the
"originals". Restoring is easy because the files are "just there" with
no compression and no searching through tar indexes and so on.

(And remember that rsync can send to a remote host, given the right ssh
credentials, etc.)

And if you want an archive, on the 1st of each month,
cp -al 99 archive.month.year

An example of space used:

Filesystem Size Used Avail Use% Mounted on
/dev/md5 35G 9.9G 24G 30% /var
/dev/md6 38G 29G 7.7G 79% /archive

The /archive here has 60-days worth of backups. So if you got a 50%
compression on 10GB of data to 5GB, 60 * 5GB is 300GB, yet the rsync
method is only using 29GB.


If you want the script, have a look at
http://unicorn.drogon.net/dailyBackup

That's just the "get out of jail free" accidental deletion backup -
the partition it backs up to is on the same box - it's part of a series
of scripts that do things like mysql database dumps, take the archive
to other servers both on-site and off-site... However if doing it to
removable media, it can be changed a little. And yes, it's in csh. Works
for me.

Gordon
From: Ian Rawlings on
On 2008-12-19, Martin Gregorie <martin(a)see.sig.for.address.invalid> wrote:

> I could, but I wanted to use compressed backups.

Yes I did wonder about that. Mind you, with your system if you do get
an error then you can lose the entire backup rather than just one
file, and I'm told (but have never tested it) that with uncorrectable
bit error rates, and the amount of bits on a disc these days, you've
got about a 50% chance of being able to read an entire disc without
finding at least one error.

> I met it with my compressed archiving system. As soon as the archive grew
> to 4GB it started failing: tar thought it had hit run out of diskspace
> and quit. The solution was simple: I merely reformatted the USB drive as
> a single ext3 partition.

Last time I hit that problem, the FS reported that the write happened
fine, but it actually had looped around and was over-writing the start
of the file ;-) I think this was accessing a disc across SAMBA though.

> The idea in both schemes is to preserve enough stuff so that, if a disk
> dies, I can just flop the latest dump back onto a new disk.

I do exactly the same with my systems, a script that mounts /boot and
an exclude list that includes /proc/, /tmp/ and a few others, although
I do include /dev now as one of the restores I did came up with "can't
access /dev/console" and died, requiring a bit of furtling!

On my laptop I rsync the encrypted filesystem across unencrypted to a
server and do an emerge sync in the script that I use to connect it to
my home network, the emerge sync (gentoo) generally takes much longer
than the backup.

--
Blast off and strike the evil Bydo empire!
http://youtube.com/user/tarcus69
http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on
On 2008-12-19, alexd <troffasky(a)hotmail.com> wrote:

> Have you thought of using rsnapshot? It uses rsync, but with hard links
> instead of creating new copies of files, so it could save you a lot of
> space.

That protects you against accidental deletion and some metadata
corruption, but if the file contents get corrupted or a file in the
backup is edited, then it happens to all the "copies" in the backups
so that's no good. Each hard link points to the same data on the
disc, so if the data on the disc changes, then provided that the inode
remains the same all your backups will be pointing at the now changed
data.

E.g.;

echo "this is textfile1.txt" > textfile1.txt
ln -s textfile1.txt textfile2.txt
cat textfile2.txt
this is textfile1.txt
vi textfile1.txt (change the textfile1 to a textfile3)
cat textfile2.txt
this is textfile3.txt

So you change one of the files and all the hard links now point to the
changed data, just what you *don't* want if you're making historic
backups. A hard link is not a backup, it's just a pointer to the same
data.

--
Blast off and strike the evil Bydo empire!
http://youtube.com/user/tarcus69
http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on
On 2008-12-19, Gordon Henderson <gordon+usenet(a)drogon.net> wrote:

> cp -al 01 00
> rsync --stats -aHx --delete /path /backupdevice/00
>
> So while not compressed, each day will only store what gets changed,

Hard links (cp -al) are not copies, they're pointers to the same data,
so a corruption of that data knackers every one of your backups of
that file, so only of use to protect against accidental deletion.

--
Blast off and strike the evil Bydo empire!
http://youtube.com/user/tarcus69
http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on
On 2008-12-19, Ian Rawlings <news06(a)tarcus.org.uk> wrote:

> echo "this is textfile1.txt" > textfile1.txt
> ln -s textfile1.txt textfile2.txt

Err, ignore the ln -s, it works the same even if you use a hard link
(which I did do when I tested it, just not in the post!)

--
Blast off and strike the evil Bydo empire!
http://youtube.com/user/tarcus69
http://www.flickr.com/photos/tarcus/sets/
First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4 5 6 7 8 9
Prev: Compaq NS5000...
Next: Backup to DVD