Hard Drive Speed

"Hard drive write speed, measured in megabytes per second (MB/s), typically ranges from around 75 MB/s for a 5400 RPM drive to about

Sounds pretty spot on to me

See above

Why? A terabyte is a lot of bytes. Guess why people are heading for NVME SSDS

"NVMe drives can usually deliver a sustained read-write speed of 4.0 GB/s in contrast with SATA SSDs that limit at 600 MB/s."

The numbers look close enough - depending on if we are talking about real GB or decimal TB (i.e. 1,073,741,824 vs 1,000,000,000)

about 13MB/s

Well that is slow in terms of raw HDD transfer speeds (130MB/s for a

Copying many small files, at the file system level, will be *much* slower than fewer large files. Thousands of small "few kB" text and config files etc you might not get more than a few hundred kB/sec up to the low MB/sec average throughput. For intermediate size files (photos etc), then 10 to 50MB/sec might be expected. Large 100MB+ files should copy at close to raw drive speed.

If you want to copy a whole disk, then drive cloning software will be very much faster - it will be able to run the drive at close to max throughout much or all of the time.

Many thanks TNP/John :-)

I had forgotten how slow spinning disks are, the Iron Wolves were advertised as being suitable for a NAS, which is where they started life, and are in an HP N54L now. The 45,000 45 KB texture files for Flight Simulator may make a difference as well!

I am struggling with the final piece of my "how to organise storage" puzzle. I have decided a NAS doesn't add anything as my NVidia TV Pro is happy to read files from normal file shares without resorting to DNLA so I don't need a streaming app.

SSDs are still relatively expensive, close to £1K to replace the Iron Wolves. In addition I know from recent experience that when they fail they fail, totally, absolutely and no warning :-(

The replacement for the first failed WD SSD is due here on Wednesday, the second one should be a week or so behind. Horrendous process, taken 2 months so far. Their instructions are incorrect (I think pre Brexit) and I had to kick their customer service into action. I have 2 more form the same time, warranties will expire by the end of this year.

I am running a second copy process from my main machine to a USB 3 DAS, started this morning around 11 o/c and has a couple of hours to go. Also an Iron Wolf spinner so may have to think about using the DAS instead of the N54L as it seems much quicker.

Keeps the brain active :-)

If both drives are in the same computer there is also the possibility that the HDD read/write speed is not the main bottleneck.

It's called "seek time".

Moving the disk drive heads around, during file by file data transfer, is slow. The seek time is the "rate limiting step". It might take a millisecond, to move the data from one disk to the other. It might take 15 milliseconds to move the heads for the next file. suddenly the transfer is 16 times slower than we would like.

SSDs still have access time, but it is a lot smaller number. Still, if you use CrystalDiskMark, and you evaluate the 4K block transfer performance, the number shown in the rating for the device, is

There is *no good answer* for small files. Small files suck. If you insist on using file-by-file transfer, you're doomed to be doing this arithmetic, figuring out the transfer speed and moaning that it is so so slow.

But, there is a solution. When you "clone" a drive, data transfers are done at the cluster or inode level. The important part of cloning though, is the access to the data is in "cluster order". The head starts at 0, moves to 3, moves to 7, the head always moves in the one direction. Disks "like" that sort of movement. And, if the clusters or inodes are contiguous (shoulder to shoulder), the cloning software can read 256,000 clusters in a row, then blast those across the cable to the other device. Suddenly, the sequential speed of the device is the rate limit. (200MB/sec on an average hard drive.)

Hard drives have a track buffer. When we do the sequence I made as an example, of 0 3 7, the disk drive can buffer up the whole track. It is possible, that the track buffer contains 0,3, and 7, all acquired in the same (unbidden) read. The disk can safely read more data than the application asks for. On the track read, we immediately clone over cluster 0. when the application says "now move 3 for me", the disk goes "hey, I already got your

The end result is, that cloning can proceed at 200MB/sec, when file-by-file is doing 1MB/sec.

I have an image file with 64,000,000 files in it. Like cloning, restoring from backup (which is also done in cluster or inode order), I can restore 64,000,000 files in *4 minutes*. This is the benefit of working at the cluster level, and NOT file-by-file.

When you put files in a ZIP ? You pay at both ends. The seek time issue occurs when loading up the ZIP file. The seek time issue occurs when you unpack the ZIP file (in a file-by-file manner of course). A ZIP file still has advantages, but because it is a file-by-file thing, it still sucks. For example, over Ethernet, you may notice computers have lousy protocols for transfer over the wire and while a ZIP container moves faster, the preparation of the ZIP container sucks sucks sucks. Thus the ZIP container is hardly ever a totally-preferred solution.

Whether you smart-clone or smart-backup, the capture of data sequentially at the cluster or inode level, is a *lot* faster than what you've been doing. Look at my example, of moving 64,000,000 files in *4 minutes* for a hint. It wold have gone faster than that, except the software has checksums on the materials, to ensure no corruption has occurred, and it would run faster if there were no checksums to calculate. I could do better than the four minute number. And, there is actually one brand of software, that does not do checksums (so it just screams).

Via cloning, a 200MB/sec hard drive should transfer

In the example here, the speed limit was the checksum activity. And no, the disk is not fiber channel. The OS "makes up" that bullshit description, based on measuring the speed and deciding "Ha! That's gotta be a fiber channel". It is not a fiber channel device.

Paul

That would be 0.13, not 0.013?

Many thanks :-)

I Googled cloning software and there's a lot of it but all now subscription (how did that creep up on us so quickly?).

Any suggestions? Currently I use SmartSync Pro because I paid for it years ago and it runs from a batch file, it may be once I've done the first copy it may be the easiest solution but I am happy to look at alternatives.

Suitable for win10/11 64 bit Macrium Reflect V7

I've been using this software for backups for a few years and performed a SSD to SSD clone a few weeks back. One SSD was in the laptop and the other in a USB adapter.

Clonezilla will always be free, but it's a bit scary. Not point and click.

It should be pointed out, that Clonezilla uses other packages to do the heavy lifting.

A little Wiki examination, will give the names of some component parts.

See also

List of disk cloning software Partclone Clonezilla FSArchiver

The reason PartImage does not support Ext4, has to do with when support stopped.

create-the-image temp.img # This might be necessary, if the application restore-the-image temp.img # needs random access to the .img file, rather than pure-sequential

Restoration like that, involving the usage of an intermediate file, is going to be slower because of the additional step, but it still gets the job done.

While "CloneZilla is scary" (agreed), that should still not prevent you from looking around and being creative.

Macrium Reflect (Windows, make the Rescue CD for other usage), can clone EXT4, but it is unclear to me whether it supports C12 properly or not. Say, Ubuntu 24.04.2 might make a EXT4 with C12 switched on. Whereas a Linux Mint 21.3, would not have C12 switched on. Tune2FS can provide info on features.

Paul

Iron Wolf are designed for NAS - so designed for 24/7 operation and also tolerant of vibration typical in multi drive enclosures.

Yup, that is the kind of thing that hammers HDD file access time. Note also that on write, not only have you got the write time, but also the OS overhead in creating and updating the directory entries for the new file.

The main NAS advantage for that kind of thing is going to be low cost of running 24/7 and shared access without needing to keep a computer on.

For bulk storage, it is still hard to beat HDDs for cost per GB, and performance is more than adequate for streaming etc.

Often you will find most SSD makers will do free cloning software for making HDD to SSD migration easy. Most will just do something like a badge engineered version of Acronis or similar. Some used to check if the machine actually contained a drive of the right brand[1], but that seems less common these days (probably because they can't always sense it when the SSD is on a USB to SATA/M2 adaptor).

Macrium reflect also has a free version. EaseUS Todo has a free tier. Aomei also have a free one. You can also make a bootable image of Clonezilla (runs outside of windows)

Use DD in linux?

Another option is that some SSDs or NVMEs usually come with a copy of Ghost

Or you could download and burn to a DVD a Ultimate Boot CD ISO or Hirens Boot CD ISO and that has a gazillion utlities some of which will be disk cloning software....

Delivered by the SH of Ventoy fame :-D

P.S. as it happens I have both UBCD and Hirens in my list of ISOs on my NVME M2 stick in a USB-C adapter......

my current list of ISOs on my Ventoy NVME is:

H:\>dir Volume in drive H is Ventoy Volume Serial Number is 4E21-0000

Directory of H:\

H:\>

dban is Dariks Boont 'n' nuke, Bliss is an Android OS that can run on a PC, HBCD is Hirnes Boot CD, mk86 is Memtest 86 and UBCD is ultimate boot CD. ANd I still have lots of space still to stick more ISOs on.

Another incredibly useful utlity is Ninite but that is specific to Windows.

No. In the end the limiting factor is how fast the data can be read off the drive.

usually te interface to the drive.

Sigh. Small files are cached into sectors. And written in blocks

Or even boot a live linux distro and let it do the copying. Its pretty efficient

"A copy of Ghost" ???

Shirley my leg is being pulled.

This advice might be OK, if you were dealing with a Win2K or WinXP era machine. A machine that has *never* had drives from other OSes connected to it.

Try to find materials of a bit more modern vintage. Because you never know what changes that an older tool can't handle properly.

cloning candidates would include (somewhat free):

Macrium Reflect Free Easeus (look for a slightly older version perhaps) AOMEI

Those three have had Free Editions, suited to doing Full Backups and likely Cloning as well.

On the Linux side, you could attempt to use gparted to clone a partition onto a second disk drive. Or, you could experiment with FSArchiver and see if it has an option that results in cloning.

Some of the really old utilities, do not know about

Rather than attempting to make lists of this sort (you can see I'm not good at it), I try to tell people to "dial in" and "test" these softwares, until confident that they work. As an example, I could find a posting about a certain product, where it could not even manage to copy a FAT32 successfully. The reason I do not name and shame them, is failure cases like this can be caused by not doing a CHKDSK /f on a partition before carrying out a sensitive procedure. Modern Windows is doing some checks in the background, which is why this is less of an issue than it used to be. Even Linux has checking utilities, such as fsck. But if you were running Win2K, and planning on cloning, I would run a CHKDSK /f on the partition first. It might never ever have been checked before.

Paul

If there are an extremely large number of small files, file-by-file copy is not the answer. In such cases, "dd" may actually be faster (5 hours versus 24 hours).

Smart Cloning is fastest, because only the inodes needed to do the job are copied and white space is ignored. And the inodes are scooped off the disk, in increasing LBA order (so the heads on the disk do short seeks, not long seeks, a head switch takes 1 millisecond on a HDD).

On the Windows side, there are several tools for cloning. Macrium Reflect Free can do NTFS and it can also do EXT4. Normally, Macrium stops, if it detects a detail that it does not support. it usually carries out checks early in the operation and throws a red code on the screen if conditions are not right. Since it is a Windows tool, its "best" support is for Windows file systems (NTFS, FAT32, EXFAT, ReFS-probably-not). I've not seen any statements about how thorough it is with EXT4 detection and analysis. For example, TUNE2FS can report what "features" are enabled on an EXT4.

The extent to which Smart Cloning will be faster, will depend on how full the disk is. "dd" copies every sector. A Smart Clone would also copy every sector... if the partition was absolutely full. Smart Clone is faster than dd, if there is some white space on the disk that does not need to be copied.

For the average disk drive in the room here, they are seldom absolutely full. Cloning will be a little bit better, in each case.

If a partition does not have an abnormally large number of files on it (like my pathological 64 million file test partition), then any olde method of copying would work. If you had three 500GB backup files stored in a 2TB partition, nothing special is required there. You can do whatever works for you. Wanna use XXCOPY ? Go right ahead. But when someone tells me "I've been waiting for 24 hours and this sucks", I have my suspicions as to what happened. You can be patient and wait for it to finish, or you can make a new plan. Your choice.

Hardware failures are also possible on storage devices. Some people do not know how to check for that, and for them, their "first sign of trouble", is when the drive is no longer detected.

Paul