Prev: CreateFileMapping failing for large files - ERROR_NOT_ENOUGH_Q
Next: Heap Corruption error - please help
From: David K David on 13 Jun 2008 19:00
I'm not sure if I'm posting this in the correct location, but I wasn't sure
where else I would post it. Let me know if it needs to be posted in a
different location.

I am creating an application for managing mass storage devices that are
plugged into a certain USB duplication hub (which is identified by it's VID
and PID). It enumerates all the mass storage devices on the hub and allows
the user to perform file copies, delete, format, eject, etc on the drives. A
feature that this application needs to have is being able to do a raw copy at
the device level from one USB drive to another. The idea is to get a mirror
image from the source device and write it onto one or more destination
devices.

Now, being very new to the technologies used to interact with USB devices
(beyond just the basic .NET framework file copy and delete and such), I have
very little knowledge about the plausibility of this feature or how
complicated it would be. From the knowledge I do have, I planned on opening
a device handle to the source and destination drives using the CreateFile API
function and then using ReadFile and WriteFile to copy the drive bit for bit
to the new drive.

I wrote the code for this and ran it successfully only to find that my
destination USB drive has been corrupted beyond repair. Obviously it is not
quite as simple as I had anticipated. I used the same code, substituting
files for the path when calling the CreateFile functions (instead of the
"\\.\PhysicalDeviceXX" path) and my code copied the files just fine so I know
my code runs correctly and isn't corrupting the data.

I have searched all over the web trying to find examples of a bit-level copy
of USB drives (or any drives for that matter) using ReadFile/WriteFile and
have come up with nothing. Can anyone provide some help and/or direction on
this issue? My time on this project is running VERY short and I need to get
this completed asap.

Any help is greatly appreciated!

David K.
From: David Craig on 13 Jun 2008 20:19
Most of us working at this level and lower do not use a managed language.

It is a rather simple task to do copies as long as the 'new' drives can only
be mounted by the RAW filesystem. If they have a filesystem on them, or it
is the source drive which may have a filesystem that Windows would recognize
it requires they be dismounted so the RAW filesystem will claim them. This
also implies that they are not either the boot or system drive for the
Windows that is currently running.

Dismounting and locking of drives/volumes appears in some of the header
files that come with the WDK. I believe most of them are also present in
the SDK headers. Using CreateFile for the physical drive and not one of the
volumes is the correct way to begin. Be careful to follow the rules for the
appropriate bits to be set in the CreateFile parameters.

You neglected to mention the Windows version you are required to support.
It gets a little more difficult with Vista and later because of UAC. If
they have to be supported, I would recommend that you add a manifest
indicating you need administrator rights.

Read some of the documents about formatting/partitioning a drive from
Sysinternals and in the MSDN Library. Your basic task is to partition and
then format the drive, but instead of just writing the system areas you will
be writing most or all of the drive.

If time is a major requirement, there are several who participate in the
newsgroups who can be hired. I suspect some have a toolkit that can speed
their delivery of the product, but don't forget that they get paid for not
only their time but their knowledge. It will take you some time to
understand all the implications of the various options in CreateFile,
ReadFile, WriteFile, and the IoCtls required to ensure you have exclusive
access to the drive(s).

Using USB is easy, but not nearly as fast as SATA 300 or 1394. Doing a copy
from one master to multiple targets will speed the duplicating process but
USB is a highly sharable design and not designed primarily for mass storage.


"David K" <David K(a)discussions.microsoft.com> wrote in message
news:D5C197FC-66F6-406F-88DC-9B5632DF5C1D(a)microsoft.com...
> I'm not sure if I'm posting this in the correct location, but I wasn't
> sure
> where else I would post it. Let me know if it needs to be posted in a
> different location.
>
> I am creating an application for managing mass storage devices that are
> plugged into a certain USB duplication hub (which is identified by it's
> VID
> and PID). It enumerates all the mass storage devices on the hub and
> allows
> the user to perform file copies, delete, format, eject, etc on the drives.
> A
> feature that this application needs to have is being able to do a raw copy
> at
> the device level from one USB drive to another. The idea is to get a
> mirror
> image from the source device and write it onto one or more destination
> devices.
>
> Now, being very new to the technologies used to interact with USB devices
> (beyond just the basic .NET framework file copy and delete and such), I
> have
> very little knowledge about the plausibility of this feature or how
> complicated it would be. From the knowledge I do have, I planned on
> opening
> a device handle to the source and destination drives using the CreateFile
> API
> function and then using ReadFile and WriteFile to copy the drive bit for
> bit
> to the new drive.
>
> I wrote the code for this and ran it successfully only to find that my
> destination USB drive has been corrupted beyond repair. Obviously it is
> not
> quite as simple as I had anticipated. I used the same code, substituting
> files for the path when calling the CreateFile functions (instead of the
> "\\.\PhysicalDeviceXX" path) and my code copied the files just fine so I
> know
> my code runs correctly and isn't corrupting the data.
>
> I have searched all over the web trying to find examples of a bit-level
> copy
> of USB drives (or any drives for that matter) using ReadFile/WriteFile and
> have come up with nothing. Can anyone provide some help and/or direction
> on
> this issue? My time on this project is running VERY short and I need to
> get
> this completed asap.
>
> Any help is greatly appreciated!
>
> David K.


From: David K on 13 Jun 2008 22:39
Thank you David for the quick response and detailed explanation/options.

To answer your question about operating system support, I only need the
application to run on Windows XP and I already expect that the user of the
application will need administrative access to the computer due to the
methods I am using for enumeration of the USB drives and the eject
functionality as well.

Also, I can give you a little more detail about the reason for wanting this
functionality. The software I am writing is for management of a USB
duplicator. Many of the potential users of this duplicator want to be able
to duplicate very quickly to multiple drives at once and with this method I
should be able to use multiple threads to write to each USB drive from a FIFO
read buffer asynchronously. This way I only have to read through once and
yet I can be copying all 20 USB drives at once.

Another reason for this functionality is that some USB drives have more than
just a basic partition and filesystem on them. For instance, there are USB
drives which have a second partition that Windows recognizes as a CD drive
and this allows them to get around the "AutoRun" security restrictions that
have been placed on USB pen drives.

We figured that if we could get a bit-level duplication of such a drive, the
destination drive(s) would all be partitioned the same as the source drive
and contain the same files as well. I imagine it would be much more
difficult (and in some cases slower) to try and duplicate the partitioning
information first and then copy each partition's contents.

Anyway, It sounds like what I'm missing then is the dismounting and locking
of the destination drive before I try to copy the RAW data (and then an
unlock and remount afterwards I suppose as well). I will definitely check
out the Sysinternals documents on formatting/partitioning.

Also is it possible you know a good link to start with on the MSDN library
as my searches on formatting/partitioning have not been very productive to
this point.

I have one other question, and that is about "dd for windows". It is the
Windows version of the GNU dd project which enables RAW image creation and
device duplication from a command line interface. It sounds like it already
has the feature set I need, in which case I could just include that with the
application and use it to perform just what I am currently trying to do
myself. If anyone has experience with using this, I'd like to hear what they
have to say...

Thanks again for the quick reply and useful information...

- David K

"David Craig" wrote:

> Most of us working at this level and lower do not use a managed language.
>
> It is a rather simple task to do copies as long as the 'new' drives can only
> be mounted by the RAW filesystem. If they have a filesystem on them, or it
> is the source drive which may have a filesystem that Windows would recognize
> it requires they be dismounted so the RAW filesystem will claim them. This
> also implies that they are not either the boot or system drive for the
> Windows that is currently running.
>
> Dismounting and locking of drives/volumes appears in some of the header
> files that come with the WDK. I believe most of them are also present in
> the SDK headers. Using CreateFile for the physical drive and not one of the
> volumes is the correct way to begin. Be careful to follow the rules for the
> appropriate bits to be set in the CreateFile parameters.
>
> You neglected to mention the Windows version you are required to support.
> It gets a little more difficult with Vista and later because of UAC. If
> they have to be supported, I would recommend that you add a manifest
> indicating you need administrator rights.
>
> Read some of the documents about formatting/partitioning a drive from
> Sysinternals and in the MSDN Library. Your basic task is to partition and
> then format the drive, but instead of just writing the system areas you will
> be writing most or all of the drive.
>
> If time is a major requirement, there are several who participate in the
> newsgroups who can be hired. I suspect some have a toolkit that can speed
> their delivery of the product, but don't forget that they get paid for not
> only their time but their knowledge. It will take you some time to
> understand all the implications of the various options in CreateFile,
> ReadFile, WriteFile, and the IoCtls required to ensure you have exclusive
> access to the drive(s).
>
> Using USB is easy, but not nearly as fast as SATA 300 or 1394. Doing a copy
> from one master to multiple targets will speed the duplicating process but
> USB is a highly sharable design and not designed primarily for mass storage.

From: David Craig on 14 Jun 2008 01:59
Must be SanDisk or another U3 drive company. I use them and am very
grateful that I can just wipe the U3 stuff out including the CD. You can
try a copy of WinHex to see if it can copy the entire media to a file. It
can also be used to write it to one drive at a time. If it works, then you
can expect success with a good program.

Look at ntdddisk.h & ntddstor.h in the WDK. Also look at the full SDK.

I will reply to this message later with a few hints.


"David K" <DavidK(a)discussions.microsoft.com> wrote in message
news:523522EA-8793-41EA-9BBC-7CD5F37C7D57(a)microsoft.com...
> Thank you David for the quick response and detailed explanation/options.
>
> To answer your question about operating system support, I only need the
> application to run on Windows XP and I already expect that the user of the
> application will need administrative access to the computer due to the
> methods I am using for enumeration of the USB drives and the eject
> functionality as well.
>
> Also, I can give you a little more detail about the reason for wanting
> this
> functionality. The software I am writing is for management of a USB
> duplicator. Many of the potential users of this duplicator want to be
> able
> to duplicate very quickly to multiple drives at once and with this method
> I
> should be able to use multiple threads to write to each USB drive from a
> FIFO
> read buffer asynchronously. This way I only have to read through once and
> yet I can be copying all 20 USB drives at once.
>
> Another reason for this functionality is that some USB drives have more
> than
> just a basic partition and filesystem on them. For instance, there are
> USB
> drives which have a second partition that Windows recognizes as a CD drive
> and this allows them to get around the "AutoRun" security restrictions
> that
> have been placed on USB pen drives.
>
> We figured that if we could get a bit-level duplication of such a drive,
> the
> destination drive(s) would all be partitioned the same as the source drive
> and contain the same files as well. I imagine it would be much more
> difficult (and in some cases slower) to try and duplicate the partitioning
> information first and then copy each partition's contents.
>
> Anyway, It sounds like what I'm missing then is the dismounting and
> locking
> of the destination drive before I try to copy the RAW data (and then an
> unlock and remount afterwards I suppose as well). I will definitely check
> out the Sysinternals documents on formatting/partitioning.
>
> Also is it possible you know a good link to start with on the MSDN library
> as my searches on formatting/partitioning have not been very productive to
> this point.
>
> I have one other question, and that is about "dd for windows". It is the
> Windows version of the GNU dd project which enables RAW image creation and
> device duplication from a command line interface. It sounds like it
> already
> has the feature set I need, in which case I could just include that with
> the
> application and use it to perform just what I am currently trying to do
> myself. If anyone has experience with using this, I'd like to hear what
> they
> have to say...
>
> Thanks again for the quick reply and useful information...
>
> - David K
>
> "David Craig" wrote:
>
>> Most of us working at this level and lower do not use a managed language.
>>
>> It is a rather simple task to do copies as long as the 'new' drives can
>> only
>> be mounted by the RAW filesystem. If they have a filesystem on them, or
>> it
>> is the source drive which may have a filesystem that Windows would
>> recognize
>> it requires they be dismounted so the RAW filesystem will claim them.
>> This
>> also implies that they are not either the boot or system drive for the
>> Windows that is currently running.
>>
>> Dismounting and locking of drives/volumes appears in some of the header
>> files that come with the WDK. I believe most of them are also present in
>> the SDK headers. Using CreateFile for the physical drive and not one of
>> the
>> volumes is the correct way to begin. Be careful to follow the rules for
>> the
>> appropriate bits to be set in the CreateFile parameters.
>>
>> You neglected to mention the Windows version you are required to support.
>> It gets a little more difficult with Vista and later because of UAC. If
>> they have to be supported, I would recommend that you add a manifest
>> indicating you need administrator rights.
>>
>> Read some of the documents about formatting/partitioning a drive from
>> Sysinternals and in the MSDN Library. Your basic task is to partition
>> and
>> then format the drive, but instead of just writing the system areas you
>> will
>> be writing most or all of the drive.
>>
>> If time is a major requirement, there are several who participate in the
>> newsgroups who can be hired. I suspect some have a toolkit that can
>> speed
>> their delivery of the product, but don't forget that they get paid for
>> not
>> only their time but their knowledge. It will take you some time to
>> understand all the implications of the various options in CreateFile,
>> ReadFile, WriteFile, and the IoCtls required to ensure you have exclusive
>> access to the drive(s).
>>
>> Using USB is easy, but not nearly as fast as SATA 300 or 1394. Doing a
>> copy
>> from one master to multiple targets will speed the duplicating process
>> but
>> USB is a highly sharable design and not designed primarily for mass
>> storage.
>


From: David Craig on 14 Jun 2008 02:24
Here is some code snipped from a format program I wrote:

1. m_hDevice = CreateFile(
m_csDeviceName,
GENERIC_READ | GENERIC_WRITE, // Need full access
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH,
NULL
);

2. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_LOCK_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);


3. if (!(WriteFile(
m_hDevice,
pBuffer,
dwNumSects * dwSectorSize,
&dwBytesWritten,
&ovl
)))
{
m_dwStatus = GetLastError();
}
else if (dwBytesWritten != (dwNumSects * dwSectorSize))
{
m_dwStatus = ERROR_WRITE_FAULT;
}


4. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_DISMOUNT_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);


5. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_UNLOCK_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);

6. bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_UPDATE_PROPERTIES,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);


7. // This function, available with Windows XP & 2003 does
// work correctly and the physical drive is properly
// remounted with the correct file system driver.
//
bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_GET_DRIVE_LAYOUT,
NULL,
0,
&outBuf,
sizeof(outBuf),
&BytesReturned,
NULL
);

bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_SET_DRIVE_LAYOUT,
&outBuf,
BytesReturned,
NULL,
0,
&BytesReturned,
NULL
);


Here are a lot of hints. I leave it to you to supply the error code
handling which I omitted. I would read about each function call and IoCtl
used. Items 6 & 7 are complementary and only one should be used, though I
think I used both. I only tested on XP Pro and I am certain it won't
function properly under Vista because of UAC though invoking from an
elevated cmd.exe prompt could solve that problem.




"David Craig" <drivers(a)nowhere.us> wrote in message
news:OQzhDPezIHA.1240(a)TK2MSFTNGP02.phx.gbl...
> Must be SanDisk or another U3 drive company. I use them and am very
> grateful that I can just wipe the U3 stuff out including the CD. You can
> try a copy of WinHex to see if it can copy the entire media to a file. It
> can also be used to write it to one drive at a time. If it works, then
> you can expect success with a good program.
>
> Look at ntdddisk.h & ntddstor.h in the WDK. Also look at the full SDK.
>
> I will reply to this message later with a few hints.
>
>
> "David K" <DavidK(a)discussions.microsoft.com> wrote in message
> news:523522EA-8793-41EA-9BBC-7CD5F37C7D57(a)microsoft.com...
>> Thank you David for the quick response and detailed explanation/options.
>>
>> To answer your question about operating system support, I only need the
>> application to run on Windows XP and I already expect that the user of
>> the
>> application will need administrative access to the computer due to the
>> methods I am using for enumeration of the USB drives and the eject
>> functionality as well.
>>
>> Also, I can give you a little more detail about the reason for wanting
>> this
>> functionality. The software I am writing is for management of a USB
>> duplicator. Many of the potential users of this duplicator want to be
>> able
>> to duplicate very quickly to multiple drives at once and with this method
>> I
>> should be able to use multiple threads to write to each USB drive from a
>> FIFO
>> read buffer asynchronously. This way I only have to read through once
>> and
>> yet I can be copying all 20 USB drives at once.
>>
>> Another reason for this functionality is that some USB drives have more
>> than
>> just a basic partition and filesystem on them. For instance, there are
>> USB
>> drives which have a second partition that Windows recognizes as a CD
>> drive
>> and this allows them to get around the "AutoRun" security restrictions
>> that
>> have been placed on USB pen drives.
>>
>> We figured that if we could get a bit-level duplication of such a drive,
>> the
>> destination drive(s) would all be partitioned the same as the source
>> drive
>> and contain the same files as well. I imagine it would be much more
>> difficult (and in some cases slower) to try and duplicate the
>> partitioning
>> information first and then copy each partition's contents.
>>
>> Anyway, It sounds like what I'm missing then is the dismounting and
>> locking
>> of the destination drive before I try to copy the RAW data (and then an
>> unlock and remount afterwards I suppose as well). I will definitely
>> check
>> out the Sysinternals documents on formatting/partitioning.
>>
>> Also is it possible you know a good link to start with on the MSDN
>> library
>> as my searches on formatting/partitioning have not been very productive
>> to
>> this point.
>>
>> I have one other question, and that is about "dd for windows". It is the
>> Windows version of the GNU dd project which enables RAW image creation
>> and
>> device duplication from a command line interface. It sounds like it
>> already
>> has the feature set I need, in which case I could just include that with
>> the
>> application and use it to perform just what I am currently trying to do
>> myself. If anyone has experience with using this, I'd like to hear what
>> they
>> have to say...
>>
>> Thanks again for the quick reply and useful information...
>>
>> - David K
>>
>> "David Craig" wrote:
>>
>>> Most of us working at this level and lower do not use a managed
>>> language.
>>>
>>> It is a rather simple task to do copies as long as the 'new' drives can
>>> only
>>> be mounted by the RAW filesystem. If they have a filesystem on them, or
>>> it
>>> is the source drive which may have a filesystem that Windows would
>>> recognize
>>> it requires they be dismounted so the RAW filesystem will claim them.
>>> This
>>> also implies that they are not either the boot or system drive for the
>>> Windows that is currently running.
>>>
>>> Dismounting and locking of drives/volumes appears in some of the header
>>> files that come with the WDK. I believe most of them are also present
>>> in
>>> the SDK headers. Using CreateFile for the physical drive and not one of
>>> the
>>> volumes is the correct way to begin. Be careful to follow the rules for
>>> the
>>> appropriate bits to be set in the CreateFile parameters.
>>>
>>> You neglected to mention the Windows version you are required to
>>> support.
>>> It gets a little more difficult with Vista and later because of UAC. If
>>> they have to be supported, I would recommend that you add a manifest
>>> indicating you need administrator rights.
>>>
>>> Read some of the documents about formatting/partitioning a drive from
>>> Sysinternals and in the MSDN Library. Your basic task is to partition
>>> and
>>> then format the drive, but instead of just writing the system areas you
>>> will
>>> be writing most or all of the drive.
>>>
>>> If time is a major requirement, there are several who participate in the
>>> newsgroups who can be hired. I suspect some have a toolkit that can
>>> speed
>>> their delivery of the product, but don't forget that they get paid for
>>> not
>>> only their time but their knowledge. It will take you some time to
>>> understand all the implications of the various options in CreateFile,
>>> ReadFile, WriteFile, and the IoCtls required to ensure you have
>>> exclusive
>>> access to the drive(s).
>>>
>>> Using USB is easy, but not nearly as fast as SATA 300 or 1394. Doing a
>>> copy
>>> from one master to multiple targets will speed the duplicating process
>>> but
>>> USB is a highly sharable design and not designed primarily for mass
>>> storage.
>>
>
>


 |  Next  |  Last
Pages: 1 2 3
Prev: CreateFileMapping failing for large files - ERROR_NOT_ENOUGH_Q
Next: Heap Corruption error - please help