Faster file uploads to SD card

I have been working on improving the file upload speed from Ethernet to the SD card on the Duet. Although I have managed to get a useful improvement in speed by improving the firmware, it has become apparent to me that there are things you can do to improve upload speed without firmware changes. So I'd like to share them with you. Here are my recommendations

1. Choose a low-capacity (2Gb or 4Gb) SD card. Re-format it using the FAT or FAT16 (not FAT32) system, and the largest cluster size you can select (which is normally 64kb). Cards larger than 4Gb can only be formatted using FAT32, so are best avoided for now.

2. Class 10 cards will often perform poorly. Some brands of Class 4 cards (e.g. Sandisk) perform much better. The reason is that the speed you get when uploading a large file is dominated by how fast the card handles non-sequential writes when updating the FAT, and Class 10 cards typically handle these about 1000 times slower than they handle sequential writes. See [forum.xda-developers.com] for some measurements.

The best speed I have obtained so far is 306Kbytes/sec, using a 2Gb Sandisk card formatted with 64kb cluster size, and some firmware tweaks which are not yet ready for release. I am looking to increase this further with additional firmware changes, but it may take a while.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Very interesting as not long ago you could read that you have to use at least a class 6 card for the Duet board and everyone cried why you get a class 4 card from Kingston with the Duet board.
If I did not get in trouble using a class2 card?
What about UHS1-cards? I switched to those cards. I use Toshiba 8GB UHS-1-cards. I can later run some tests. With your firmware 1.09j I can achieve up to 240kb/s with these cards using FileZilla.

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

The additional data transfer speed provided by UHS-1 cards is only available when using the UHS hardware interface, which the Duet does not support. In any case, there is no need for faster data transfer because even Class 4 cards are eaaily fast enough. However, if UHS-1 cards handle random writes faster than Class 10 cards do, then they could be a good choice.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

I bought those cards because I got it cheap. I use 16GB cards especially for my RPi as i hoped that the reading speed is faster even if UHS-1 is not supported. However I do not want to have several different cards so if I need a card I just take one. Okay sometime I am changing the brands or size but I only buy microSD-cards as they are not much more expensive as SD-cards.
Besides I have Intenso 16GB cards with UHS-1 and a poor sequential writing speed (as UHS-1 is more intended for reading speeds).
So now I am curious and I will run some tests soon but the only "slow" card I have is the delivered Kingston with my MK1.
But how fast you get with your card and the normal firmware? As I am getting 240kb/s where I got only 200kb/s with my previous card (Intenso class 10).

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

The Toshiba 8GB card with UHS-1 costs only 0,60€ more than the cheapest card with 4GB and Class4. However I ran several tests with Crystalmark 5.0.2 under Windows 10 64-Bit and I got exactly the results what I had expected. It seems my cardreader is not capable of UHS but that is good. BTW the Kingston card which was delivered with my MK1 failed completely. The random writing speed was really poor. So I used an old SD-card with Class2 for comparison. Maybe the Kingston card is not fully functional as I replaced it because I got some trouble.
I hope it is okay to post the results here.

Toshiba 8GB UHS-1


Transcend 32GB UHS-1


Transcend 32GB Class10


Intenso 16GB UHS-1


Intenso 16GB Class10


Kingston 4GB Class 4 Fat32


Kingston 4GB Class4 Fat


Hama 2GB Class2 Fat (Full SD-card)


The Toshiba wins at 4k writing. So I think it was a good choice as I get upload results of 240kb/s with this card and with the Intenso Class10 I only got 200kb/s (but using an older firmware version).

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

I just got my best ever upload speed: 840kb/sec on a 6.3Mb file.

The way I got this speed was by using the original unbranded 2Gb card that came with my red Ormerod 1 kit! It's the same card that can't always be read when the Duet boots up.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

I would really like to compare your upload speed with my Toshiba card. For me it is very strange and it does not make any sense if I look at my tests with CrystalDiskMark. Normally the sequential write speed shozld be the only thing that matters while uploading and the random read speed when reading the card content.
My original ard was a Kingston Class4 4GB as still in the kit now. But I got the green Ormerod from about May 2014.

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

Quote
Treito
I would really like to compare your upload speed with my Toshiba card. For me it is very strange and it does not make any sense if I look at my tests with CrystalDiskMark. Normally the sequential write speed shozld be the only thing that matters while uploading and the random read speed when reading the card content.
My original ard was a Kingston Class4 4GB as still in the kit now. But I got the green Ormerod from about May 2014.

PC based benchmark tests will not necessarily reflect the speed the Duet can write at, because PC drivers use optimizing methods that the Duet probably does not have (e.g. caching the FAT instead of updating on the SD card after every cluster write).

David, how does your firmware behave using a FAT formatted SD card if your print file has a ".gcode" suffix such as S3D creates? IIRC the FAT format only supports 8.3 file names and so the suffix will presumably be changed. Will that affect the way the firmware treats the file?

Dave

Quote
Treito
I would really like to compare your upload speed with my Toshiba card. For me it is very strange and it does not make any sense if I look at my tests with CrystalDiskMark. Normally the sequential write speed shozld be the only thing that matters while uploading and the random read speed when reading the card content.

The reason is that the sequential writes are interrupted by reads and writes to the FAT allocate new clusters. These writes can be extremely slow, e.g. 40ms on a Class 10 card according to my measurements. I suspect that the processor on the card has decided that sequential streaming mode has finished, and it has decided to do some wear-levelling. Or maybe it actually cached the data it received at high speed into RAM and is busy writing it to flash.

One of the things I want to try is pre-allocating all the clusters at the start of the upload. This will require a change to the web upload protocol, to pass the file size as well as the filename at the start.

That Toshiba card looks a good choice to me, based on the random 4K speed.

Quote
dmould
David, how does your firmware behave using a FAT formatted SD card if your print file has a ".gcode" suffix such as S3D creates? IIRC the FAT format only supports 8.3 file names and so the suffix will presumably be changed. Will that affect the way the firmware treats the file?

The FAT filing system has supported long filenames for a very long time. Microsoft has a patent on it, and extracts royalties from some companies.

Although my old 2Gb card appeared to support high upload speeds, it had turned out to be not very reliable. It may be that the Duet is driving it at too high a clock speed, or something like that.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Dave that is why I am curiuos how the Toshiba behaves with the optimized firmware for reference as this is the fastest card that I had in my Duet yet. The Intenso Class10 and especially the Kingston were much slower. The Intenso reached 200kb/s, the Kingston 150kb/s and the Toshiba now 240kb/s.
I know that you cannot compare PC based card reading with the Duet card reading, but the link David gave there were poor write 4k-speed of class10-cards with only 0.004 kbit/s and a class4-card reached 1.4 kbit/s. My tests showed the complete opposite.

Edited 1 time(s). Last edit at 09/14/2015 07:50AM by Treito.

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

Quote
dc42

Quote
dmould
David, how does your firmware behave using a FAT formatted SD card if your print file has a ".gcode" suffix such as S3D creates? IIRC the FAT format only supports 8.3 file names and so the suffix will presumably be changed. Will that affect the way the firmware treats the file?

The FAT filing system has supported long filenames for a very long time. Microsoft has a patent on it, and extracts royalties from some companies.

OK, in which case it is more correctly called a VFAT file system, though I've just read that MS stopped making the distinction (and also no longer makes any distinction between FAT8, FAT12 and FAT16 either).

Dave

As I formatted the Kingston card I could only choose between FAT, FAT32, exFAT and NTFS.

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.

Quote
dc42
I just got my best ever upload speed: 840kb/sec on a 6.3Mb file.

The way I got this speed was by using the original unbranded 2Gb card that came with my red Ormerod 1 kit! It's the same card that can't always be read when the Duet boots up.

So thats why i have to reset my duet so meny times to get a connection.... god damn card xD

I just tried a Samsung 16Gb UHS-1 card. The speed seems to be very variable, it's anything between 180 and 355kb/sec even on the same file..

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Are you timing the transfer and calculating the average speed, or talking about reported speed from the OS? In my days working with large video files over "fast" SD cards, which are usually formatted in systems other than fat32, the speeds seemed to be very inconsistently reported to the OS. I think the true speed needs to be calculated by timing the transfer. I think it has to do with the the way the file system verifies the writes, and waits to close the tables... or something. (I'm not knowledgeable in this, I just pretend to be)

n8bot is right. The OS maybe caches some transfer for example. I can see this using FileZilla. At the end of a big transfer I only get stable transfer rates. At the beginnjing my Toshiba also achieves >800 kb/s but the OS simple has filled its cache for example. You have to use exactly the same file size. Transfer it and measuring the transfer time. Linux may give some good tools. FileZilla reports in fully seconds so the file has to be large enough and then this would be okay.

---

Slicer: Simplify3D 4.0; sometimes CraftWare 1.14 or Cura 2.7
Delta with Duet-WiFi, FW: 1.20.1RC2; mini-sensor board by dc42 for auto-leveling
Ormerod common modifications: Mini-sensor board by dc42, aluminum X-arm, 0.4 mm nozzle E3D like, 2nd fan, Z stepper nut M5 x 15, Herringbone gears, Z-axis bearing at top, spring loaded extruder with pneumatic fitting, Y belt axis tensioner
Ormerod 2: FW: 1.19-dc42 on Duet-WiFi. own build, modifications: GT2-belts, silicone heat-bed, different motors and so on. Printed parts: bed support, (PSU holder) and Y-feet.
Ormerod 1: FW: 1.15c-dc42 on 1k Duet-Board. Modifications: Aluminium bed-support, (nearly) all parts reprinted in PLA/ ABS, and so on.