Prev: 2.5" drives for server instead of 3.5"?
Next: new 4K sector hard drive on Windows 2003 server
From: David Brown on 28 Apr 2010 04:59
On 27/04/2010 22:12, Kremlar wrote:
> I've had it with tapes and am trying to come up with a good disk-based
> backup solution for servers (running AD, Exchange, etc.).
>
> We currently use Symantec Backup Exec with mostly DLT-V4 drives,
> previously using Sony AIT. Experiencing too many early tape drive
> failures, bad tapes, bad tapes ruining drives (!), etc.
>
> What are others doing for disk-based backup solutions for small/medium
> size businesses? I don't mind sticking with Backup Exec. External USB
> drives plugged in via USB cable can work, but seems a bit hack to me.
>
> I'm envisioning some kind of caddy containing 2.5" or 3.5" SATA drives
> (to cover the electronics and make them a bit more durable) that inserts
> into either an external USB/eSATA base unit or internal drive bay.
>
> Something like this on NewEgg, but ideally I'd want a caddy for the
> drive that will protect the electronics as we often times need the
> ability to take backups off-site.
>
> http://www.newegg.com/Product/Product.aspx?Item=N82E16817998038&cm_re=2.5%22_disk_enclosure-_-17-998-038-_-Product
>
>
> We need the ability to rotate between 5+ hard drives, plus take at least
> one backup off-site.
>
> What are others doing?
>
>

What sort of data size are you looking at? It's important to know how
much data you have in total, how much changes in a given timeframe, and
also how much network bandwidth you have to an offsite backup.

My preferred solution is to use an rsync-based backup (you can use
rsnapshot, dirvish, etc., to automate it, or write the scripts
yourself). When the server data is copied over to the backup system,
only the changes are actually transferred and stored. But for each
backup run, you have a snapshot of the server data at the backup time.
Thus you can always go back in time and see exactly what you had at any
point - restoration of data is a simple file copy. And if your network
bandwidth matches your data change per day, you can run offsite backups
over the Internet overnight automatically.

From: David Brown on 28 Apr 2010 06:44
On 28/04/2010 11:39, Kremlar wrote:
> 1-2TB for a week's worth of data would be fine, maybe 200-300GB would be
> a full nightly backup. I prefer taking full backups every night.
>
> RSYNC sounds great, and I will check into it, but for off-site copies of
> at least 1 full backup (200-300GB), I was envisioning more of a
> removable media that can be physically taken off-site. Like a tape, or
> removable hard disk.
>

Taking full backups is a hangover from the sort of backup system you are
used to - it is unnecessary with an rsync style backup. I only ever do
full backups when I set up a new backup server, and even then it's
really an incremental backup but with nothing there from before.

There are two reasons for doing full backups. One is when you consider
your backup media to be unreliable - with rsync backups you avoid that
by having the redundancy of at least two backup servers, and possibly
using raid on the servers (though three backup servers is better than
two servers with mirrored raid).

The other reason for doing full backups is so that you can easily
restore lost data. Restoration of data is the critical part of any
backup system, though it is often overlooked or given only brief testing.

When you are thinking in terms of "backup sets", such as making a full
backup once a week and incremental (or differential) backups every
night, it's clear that full backups are much easier for restoring data.
This is independent of the media - be it tape, removable hard drives,
optical disks, etc.

With rsync backups as I describe them, you are keeping a full copy of
all your data as a normal directory on the backup server. Each time you
run a backup, you use rsync to update that copy from the main server -
new or changed files are copied over, and files that are deleted from
the main server are deleted on the backup copy. On the backup server
you then make a hardlinked snapshot copy of the sync directory ("cp
-al") to a dated directory - metadata for directories are copied, but
file data is not as the snapshot and the sync copies share the same
hardlinked file.

The result is that you have a series of directories on the backup server
that hold snapshots of the main server at given dates, and you can
access all the files directly as though you had multiple independent
full backups. Yet the backup transfer time, and the space requirements,
are as though you had done simple incremental backups each night.

mvh.,

David




>
> "David Brown" <david(a)westcontrol.removethisbit.com> wrote in message
> news:4bd7f9ce$0$11807$8404b019(a)news.wineasy.se...
>> On 27/04/2010 22:12, Kremlar wrote:
>>> I've had it with tapes and am trying to come up with a good disk-based
>>> backup solution for servers (running AD, Exchange, etc.).
>>>
>>> We currently use Symantec Backup Exec with mostly DLT-V4 drives,
>>> previously using Sony AIT. Experiencing too many early tape drive
>>> failures, bad tapes, bad tapes ruining drives (!), etc.
>>>
>>> What are others doing for disk-based backup solutions for small/medium
>>> size businesses? I don't mind sticking with Backup Exec. External USB
>>> drives plugged in via USB cable can work, but seems a bit hack to me.
>>>
>>> I'm envisioning some kind of caddy containing 2.5" or 3.5" SATA drives
>>> (to cover the electronics and make them a bit more durable) that inserts
>>> into either an external USB/eSATA base unit or internal drive bay.
>>>
>>> Something like this on NewEgg, but ideally I'd want a caddy for the
>>> drive that will protect the electronics as we often times need the
>>> ability to take backups off-site.
>>>
>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16817998038&cm_re=2.5%22_disk_enclosure-_-17-998-038-_-Product
>>>
>>>
>>>
>>> We need the ability to rotate between 5+ hard drives, plus take at least
>>> one backup off-site.
>>>
>>> What are others doing?
>>>
>>>
>>
>> What sort of data size are you looking at? It's important to know how
>> much data you have in total, how much changes in a given timeframe,
>> and also how much network bandwidth you have to an offsite backup.
>>
>> My preferred solution is to use an rsync-based backup (you can use
>> rsnapshot, dirvish, etc., to automate it, or write the scripts
>> yourself). When the server data is copied over to the backup system,
>> only the changes are actually transferred and stored. But for each
>> backup run, you have a snapshot of the server data at the backup time.
>> Thus you can always go back in time and see exactly what you had at
>> any point - restoration of data is a simple file copy. And if your
>> network bandwidth matches your data change per day, you can run
>> offsite backups over the Internet overnight automatically.
>>
>>

From: David Brown on 28 Apr 2010 09:10
On 28/04/2010 13:32, Kremlar wrote:
> You're right, it is a hangover from my current mentality and something I
> need to open my mind about.
>
> How does RSYNC handle things like Active Directory and Exchange?
>

I don't know - I use Linux on servers (and though I use samba for file
sharing, we don't use active directory at all).

Basically, rsync backups are about doing a snapshot of a directory. I
know that works perfectly well for the maildir directory I have for my
imap server (dovecot).

I gather Exchange uses some sort of mishmash of huge email store files,
"jet" databases, and other bits and pieces that are apparently designed
to make life difficult - it probably even stores important information
in the windows registry. This is one of the reasons why I don't
consider Exchange a good choice for an email server. But obviously you
are looking for a backup solution for the email server you have, not an
email server to suit the backup system!

I doubt if a direct rsync copy from the relevant directories on the
email server would be successful. At best, backup and restore would be
like somewhat like giving the server an unexpected hard reset.
Theoretically Exchange can use its logs to recover from this and fix any
database corruption - I would not want to trust that in a real system.

So for backup of Exchange data, you'll have to have some other software
that is designed for the purpose, and writes the backup files somewhere.
You can then use rsync to copy these files. You gain most if these
backups are split into many different files, with only some files
changing each day as unchanged files cost nothing in time or space to
the rsync backup. If you have large backup files that change each day,
the rsync backup will have to make new copies each time - but it is able
to avoid a full transfer over the network (it only transfers the
difference between the files).

Note also that the backup servers need to be Linux (or at least some
form of *nix), since the snapshot copies use hard links which windows
can't do properly. The source of the backups can be anything that is
accessible to the Linux server.

mvh.,

David


>
> "David Brown" <david(a)westcontrol.removethisbit.com> wrote in message
> news:4bd81273$0$11799$8404b019(a)news.wineasy.se...
>> On 28/04/2010 11:39, Kremlar wrote:
>>> 1-2TB for a week's worth of data would be fine, maybe 200-300GB would be
>>> a full nightly backup. I prefer taking full backups every night.
>>>
>>> RSYNC sounds great, and I will check into it, but for off-site copies of
>>> at least 1 full backup (200-300GB), I was envisioning more of a
>>> removable media that can be physically taken off-site. Like a tape, or
>>> removable hard disk.
>>>
>>
>> Taking full backups is a hangover from the sort of backup system you
>> are used to - it is unnecessary with an rsync style backup. I only
>> ever do full backups when I set up a new backup server, and even then
>> it's really an incremental backup but with nothing there from before.
>>
>> There are two reasons for doing full backups. One is when you consider
>> your backup media to be unreliable - with rsync backups you avoid that
>> by having the redundancy of at least two backup servers, and possibly
>> using raid on the servers (though three backup servers is better than
>> two servers with mirrored raid).
>>
>> The other reason for doing full backups is so that you can easily
>> restore lost data. Restoration of data is the critical part of any
>> backup system, though it is often overlooked or given only brief testing.
>>
>> When you are thinking in terms of "backup sets", such as making a full
>> backup once a week and incremental (or differential) backups every
>> night, it's clear that full backups are much easier for restoring
>> data. This is independent of the media - be it tape, removable hard
>> drives, optical disks, etc.
>>
>> With rsync backups as I describe them, you are keeping a full copy of
>> all your data as a normal directory on the backup server. Each time
>> you run a backup, you use rsync to update that copy from the main
>> server - new or changed files are copied over, and files that are
>> deleted from the main server are deleted on the backup copy. On the
>> backup server you then make a hardlinked snapshot copy of the sync
>> directory ("cp -al") to a dated directory - metadata for directories
>> are copied, but file data is not as the snapshot and the sync copies
>> share the same hardlinked file.
>>
>> The result is that you have a series of directories on the backup
>> server that hold snapshots of the main server at given dates, and you
>> can access all the files directly as though you had multiple
>> independent full backups. Yet the backup transfer time, and the space
>> requirements, are as though you had done simple incremental backups
>> each night.
>>
>> mvh.,
>>
>> David
>>
>>
>>
>>
>>>
>>> "David Brown" <david(a)westcontrol.removethisbit.com> wrote in message
>>> news:4bd7f9ce$0$11807$8404b019(a)news.wineasy.se...
>>>> On 27/04/2010 22:12, Kremlar wrote:
>>>>> I've had it with tapes and am trying to come up with a good disk-based
>>>>> backup solution for servers (running AD, Exchange, etc.).
>>>>>
>>>>> We currently use Symantec Backup Exec with mostly DLT-V4 drives,
>>>>> previously using Sony AIT. Experiencing too many early tape drive
>>>>> failures, bad tapes, bad tapes ruining drives (!), etc.
>>>>>
>>>>> What are others doing for disk-based backup solutions for small/medium
>>>>> size businesses? I don't mind sticking with Backup Exec. External USB
>>>>> drives plugged in via USB cable can work, but seems a bit hack to me.
>>>>>
>>>>> I'm envisioning some kind of caddy containing 2.5" or 3.5" SATA drives
>>>>> (to cover the electronics and make them a bit more durable) that
>>>>> inserts
>>>>> into either an external USB/eSATA base unit or internal drive bay.
>>>>>
>>>>> Something like this on NewEgg, but ideally I'd want a caddy for the
>>>>> drive that will protect the electronics as we often times need the
>>>>> ability to take backups off-site.
>>>>>
>>>>> http://www.newegg.com/Product/Product.aspx?Item=N82E16817998038&cm_re=2.5%22_disk_enclosure-_-17-998-038-_-Product
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> We need the ability to rotate between 5+ hard drives, plus take at
>>>>> least
>>>>> one backup off-site.
>>>>>
>>>>> What are others doing?
>>>>>
>>>>>
>>>>
>>>> What sort of data size are you looking at? It's important to know how
>>>> much data you have in total, how much changes in a given timeframe,
>>>> and also how much network bandwidth you have to an offsite backup.
>>>>
>>>> My preferred solution is to use an rsync-based backup (you can use
>>>> rsnapshot, dirvish, etc., to automate it, or write the scripts
>>>> yourself). When the server data is copied over to the backup system,
>>>> only the changes are actually transferred and stored. But for each
>>>> backup run, you have a snapshot of the server data at the backup time.
>>>> Thus you can always go back in time and see exactly what you had at
>>>> any point - restoration of data is a simple file copy. And if your
>>>> network bandwidth matches your data change per day, you can run
>>>> offsite backups over the Internet overnight automatically.
>>>>
>>>>
>>

From: Lynn McGuire on 28 Apr 2010 13:12
> What are others doing for disk-based backup solutions for small/medium
> size businesses? I don't mind sticking with Backup Exec. External USB
> drives plugged in via USB cable can work, but seems a bit hack to me.

We use both internal dedicated hard drives and external hard drives
for our Windows network. We use robocopy for the backup tool and
run it once per day (when I leave the office). Robocopy works fine
and I have it set to keep deleted files. This forces reformatting
of the backup drives occasionally depending on the size. Our backup
drives are 750 GB to 1.5 TB. Our backup is currently 550 GB and is
growing 10+ GB per month.

We do have one constantly open database (ACT!) but it produces a zip
backup file once per day. I rotate the usb external drives once per
week to my home. I have a total of 7 usb external drives of which
only one is in the office at any given moment.

I prefer robocopy ( http://en.wikipedia.org/wiki/Robocopy ) because
then each backup device contains a direct copy of the files. You
do not have to use a program like backup exec to read the backup
device.

I use the following batch file:

robocopy \\sc\c sc_c /s /e /v /r:0 /xj
attrib -h -s sc_c
date /t
time /t
echo "" > lasttime
dir

good luck,
Lynn
From: David Brown on 28 Apr 2010 15:48
Arno wrote:
> David Brown <david(a)westcontrol.removethisbit.com> wrote:
>> On 28/04/2010 13:32, Kremlar wrote:
>>> You're right, it is a hangover from my current mentality and something I
>>> need to open my mind about.
>>>
>>> How does RSYNC handle things like Active Directory and Exchange?
>>>
>
>> I don't know - I use Linux on servers (and though I use samba for file
>> sharing, we don't use active directory at all).
>
>> Basically, rsync backups are about doing a snapshot of a directory. I
>> know that works perfectly well for the maildir directory I have for my
>> imap server (dovecot).
>
> The thing here is that a Linux installation expects to be backed up
> with tar/rsync/... while running. After all it is a server OS and
> backups without stopping services or reboots are just the way to do it.
> I have done backups/copies with tar of running systems for years
> under Linux and never had any problems. This is what a server OS
> gives you. The jokers in Redmont are (again) decades behind in
> this. I bet they do not even know that.
>

Backing up is easy - have you tested restoring? I know for a fact that
this sort of backup is fine for dovecot and maildir storage - most Linux
mail servers are happy that way. But it might not work so well with,
for example, a database server. It will /probably/ work - but you have
to check it.

In general, of course, you are right - Linux services are designed to be
robust in the light of failure, and typically store files in a way that
makes them easily recoverable. Using a simple file copy
(tar/rsync/etc.) for backup therefore mostly works fine. Windows style
servers are a different matter - they may store all sorts of stuff in
different places, and because you generally have a single huge program
trying to do everything, they don't have the necessary intermediate data
stored in files.

>> I gather Exchange uses some sort of mishmash of huge email store files,
>> "jet" databases, and other bits and pieces that are apparently designed
>> to make life difficult - it probably even stores important information
>> in the windows registry. This is one of the reasons why I don't
>> consider Exchange a good choice for an email server. But obviously you
>> are looking for a backup solution for the email server you have, not an
>> email server to suit the backup system!
>
> Exchange is a toy, for various reasons. That said, for exchange the
> only real choice is to stop the mail server for backup. Then you
> can use rsync again.
>

That should probably work. But in that case, I'd make a fast local copy
of the data before doing an offsite rsync.

> This is not too bad actually, as the email system was not (fortunately!)
> designed by MS and has redundancy. If a mail server is unreachable,
> the sending MTA will try again, hence no mail lost. The only issue
> I see is if this server is also used outbound. In that case you probably
> need to either provide a failover or tell your users that there can be
> problems sending mail in the backup time window.
>
>> I doubt if a direct rsync copy from the relevant directories on the
>> email server would be successful. At best, backup and restore would be
>> like somewhat like giving the server an unexpected hard reset.
>> Theoretically Exchange can use its logs to recover from this and fix any
>> database corruption - I would not want to trust that in a real system.
>
> What is worse, you cannot be sure until you try it. And even then
> some backups may work and others may not. Not a good situation that
> speaks of lack of experience and foresight on the part of the
> software designers.
>
> I think arranging a backup time windown (e.g. 4:00-5:00) and then
> stopping exchange for a backup is your best bet. From what I hear
> this may also increase the stability of exchange in addition.
>
> Arno
 |  Next  |  Last
Pages: 1 2
Prev: 2.5" drives for server instead of 3.5"?
Next: new 4K sector hard drive on Windows 2003 server