External hard drives...

[tim...]

making the point that the terminology you used isn't common currency

[John Rumm]

Indeed, and that was the point. Drop and drag will incur the overhead of dealing with lots of small files, as well as the additional overhead of pre-enumeration to allow the progress dialogue to give some meaningful feedback.

Command line copy will be a little faster since it does not need to enumerate all the files in advance.

Yes, so that is why, if you want to copy a whole disk quickly, cloning it using intelligent block transfer software will be significantly quicker than any file by file copy.

[Paul]

Original post Nov6,2020

Current date Nov9,2020

Information dribble so far.

"I have all my stuff on a 1tb external drive.... yesterday I bought another to back it up"

"says it will take three and a half hours"

Source drive 1TB Destination drive 1TB Transfer time estimated 12600 seconds

"it is transferring at 12mb/s" # I didn't see that message at the time

"good idea to have a back up back up with everything on it about 150gb"

MB

12-- * 12600 SEC = 151,200 MB =~ 150GB checks out SEC

This is not good performance on a USB2 port.

Transfer speed could be improved (doubled) by cloning.

if the source drive is doing a lot of retries and error correction, this can account for sub-optimal performance.

USB2 ports should manage 35MB/sec. The disk is transferring at 12MB/sec, about 1/3rd the expected rate.

If the source drive has extremely small files, like a million 4KB files, file-by-file transfer time is then dominated by seek time, and it's not unusual for a transfer to occur at 1MB/sec and the estimate to be 24 hours or more. This is an example of a pathological case. Nobody in the year 2020 has

4KB files. There will be a mix of larger files instead.

In such cases, and particularly as the drives are the same size, cloning software will give a faster transfer rate. Cloning takes seek time out of the picture, for best result.

Having USB3 ports on the computer would help, but first we have to figure out where the 12MB/sec is coming from. Since the transfer is not currently bus limited (35MB/sec), in fact a USB3 port used in the current way (drag and drop copy) will not help.

Summary:

A combination of cloning software and USB3 ports would give an optimal result. The transfer cannot go faster than floor(txspeed1, txspeed2).

A combination of cloning software and existing USB2 port would cut the transfer time in half.

If the transfer time is not cut in half by cloning, there is a good chance the source drive is damaged and in danger of data loss.

*******

To evaluate the source drive, try the following.

Install. Select the Benchmark tab. Adjust the menu at the top to select the source drive. Run the benchmark. The test is read-only and will not harm the data on the drive. Only the paid version of the software has read/write testing.

The expected read result, is a flat line at 35MB/sec. The HDTune benchmark is sequential. Seek time plays no part. If the drive is sick and has bad sectors, the rate will be less than 35MB/sec.

This test will establish a baseline, a health check, which adds to the information summarized above.

You cannot use the SMART tab on USB enclosures, unless there is passthru for SMART data. Consequently I have not pointed out the Health tab or given instructions for use, as I don't expect the dialog will load properly.

Paul

[bilou]

This is only a good idea if the partitions are almost full and de-fragmented. Cloning 2 brand new empty disks takes a lot of time.

[whisky-dave]

I backed up my iMac over the weekend (as I'm installing W10 on it)

Using just the finder I got an average speed of 2.1 GB per minute. A 2014 iMac with 512 GB SSD to an external 2 GB drive in a caddy over USB 3.0

Both time machine and carbon copy cloner too longer at about 1.5 GB per miniute

[Paul]

Cloning only copies the occupied clusters on NTFS. If you have 1GB of files on a 1TB disk, it takes no time at all to clone.

Cloning is not limited to "dd" transfers. Macrium uses "smart" transfer to only copy the occupied clusters. Virtually all the other 20+ programs for Windows do the same thing. It's harder to find a working "dumb" transfer method. You have to use "dd.exe" to do that.

Backup time - 5min 50sec Clone time - 7min 35sec Copy - 18.7MB/Sec estimated 24 minutes

In this case, the transfer of a fragmented file, copying is three times slower.

I could demonstrate a similar thing with small files, as I have a disk image around somewhere with a million files on it for testing.

Cloning can be faster, if done right. Because the clusters identified for transfer, are transferred in numerical order. Cluster 0, Cluster 5, Cluster 6, and so on.

Paul

[Ian Jackson]

In message <robnh5$iu0$ [e-mail]>, Paul [e-mail] writes

It possibly depends on the cloning program. I'm sure that at least one of those that I've tried gave the option of copying only the occupied clusters, or copying the lot even when empty.

>

[tim...]

most windows users are numpties and just use D&D

Some don't even know that doing D&D between directories on the same device does a move and between devices does a copy (you have to fiddle around with Shift/Ctrl key to swap)

[John Rumm]

Most users of computers of any type!

If you drag with the right mouse button rather than the left, you will get a popup context menu when you "drop" that lets you copy, move, or create shortcut overriding the normal default actions.

[John Rumm]

Perhaps you use your intuition and throw in some of the things that you think might help influence the answer... you know like how much data you are copying etc.

Bit like if someone asked you "What size joist do I need?" but did not mention the span or loading, how are you the professional going to answer?

[John Rumm]

Of course, you would not stoop so low... oh hang on:

"Nobody taking the piss out of tim lamb then?....even I know he hasn't got a clue about phones and sim cards...missed a good chance there for some fun chaps..... "

Perhaps you could make your mind up what you actually want?

You already have something that "bloody works", but you asked if there was a way to make it happen faster.

So we give you some more advanced techniques that will make it go faster, and you are still whining it all too complicated for you.

[Jim GM4DHJ ...]

complicated is bad......

[John Rumm]

Better make do with simple and slow then...

[Graeme]

In message <roc7ka$klb$ [e-mail], tim... [e-mail] writes

Well, the Windows numpty does at least know that the easiest way to move files between devices or drives is a simple cut and paste :-)