Unaligned write command (SATA errors) on Linux (with specific hardware)
I was trying to use a (Chinese) T-BAO R3 Pro AMD Ryzen 7 5700U Mini PC for in-house NAS/server, has plenty of RAM, decent processor, takes only a little space and not loud either. Put two Seagate EXOS disks in it. Comes with Windows preinstalled. Ordered mine from BangGood.
Links (in case):
https://www.techradar.com/pro/t-bao-minipc-nas-r3-review
https://geekbuying.com/item/T-BAO-R3-Pro-Mini-PC-32-1TB-523101.html
https://banggood.com/T-BAO-R3-Pro-AMD-Ryzen-7-5700U-Mini-PC-16GB-RAM-512GB-SSD-WIFI6-Windows-11-4K-Triple-Display-Mini-Computer-Desktop-PC-p-2004961.html
https://forum.openmediavault.org/index.php?thread/44547-topton-nas-thoughts
When installing Ubuntu on it and starting my RAID (with the EXOS disks), I was greeted by the following errors:
ata2.00: exception Emask 0x10 SAct 0x4000 SErr 0x2c0101 action 0x6 frozen
ata2.00: irq_stat 0x08000000, interface fatal error
ata2: SError: { RecovData UnrecovData CommWake 10B8B BadCRC }
ata2.00: failed command: READ FPDMA QUEUED
ata2.00: cmd 60/20:70:c0:91:00/00:00:00:00:00/40 tag 14 ncq dma 131072 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
sd 1:0:0:0: [sda] tag#14 Sense Key : Illegal Request [current]
sd 1:0:0:0: [sda] tag#14 Add. Sense: Unaligned write command
sd 1:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 00 00 91 c0 00 00 20 00
I/O error, dev sda, sector 298496 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2
ata2: EH complete
ata2.00: exception Emask 0x10 SAct 0x30 SErr 0x80100 action 0x6
ata2.00: irq_stat 0x40000008
ata2: SError: { UnrecovData 10B8B }
ata2.00: failed command: READ FPDMA QUEUED
ata2.00: cmd 60/20:20:40:cb:00/00:00:00:00:00/40 tag 4 ncq dma 131072 in
res 43/84:20:40:cb:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
ata2.00: status: { DRDY SENSE ERR }
ata2.00: error: { ICRC ABRT }
ata2: hard resetting link
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sda] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
sd 1:0:0:0: [sda] tag#4 Sense Key : Aborted Command [current]
sd 1:0:0:0: [sda] tag#4 Add. Sense: Scsi parity error
sd 1:0:0:0: [sda] tag#4 CDB: Read(10) 28 00 00 00 cb 40 00 00 20 00
I/O error, dev sda, sector 416256 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
ata2: EH complete
ata2.00: exception Emask 0x10 SAct 0x20000 SErr 0x280100 action 0x6 frozen
ata2.00: irq_stat 0x08000000, interface fatal error
ata2: SError: { UnrecovData 10B8B BadCRC }
ata2.00: failed command: READ FPDMA QUEUED
ata2.00: cmd 60/20:88:80:66:01/00:00:00:00:00/40 tag 17 ncq dma 131072 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
sd 1:0:0:0: [sda] tag#17 Sense Key : Illegal Request [current]
sd 1:0:0:0: [sda] tag#17 Add. Sense: Unaligned write command
sd 1:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 00 01 66 80 00 00 20 00
I/O error, dev sda, sector 734208 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
ata2: EH complete
ata2: limiting SATA link speed to 3.0 Gbps
ata2.00: exception Emask 0x10 SAct 0x20000000 SErr 0x280100 action 0x6 frozen
ata2.00: irq_stat 0x08000000, interface fatal error
ata2: SError: { UnrecovData 10B8B BadCRC }
ata2.00: failed command: READ FPDMA QUEUED
ata2.00: cmd 60/20:e8:00:07:02/00:00:00:00:00/40 tag 29 ncq dma 131072 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
ata2.00: status: { DRDY }
ata2: hard resetting link
Only happened with the disk in the first slot (did not matter which disk that was, also tried another Seagate model), no luck, then came across the following long thread on GitHub: https://github.com/openzfs/zfs/issues/10094
There's s decent summary of the possible causes/workarounds somewhere in the middle, but let me paste it here:
1.) Faulty SATA cables (I replaced them all, no change, but I admit this could be the problem in some cases)
2.) Faulty disks (Mine were known to be good, and also, errors were randomly distributed among them)
3.) Power saving in the SATA link or the PCI bus (disabling this did not help)
4.) Problematic controllers (Both the FCH and the ASM1166 chips as well as a JMB585 showed the same behaviour)
5.) Limiting SATA speed to SATA 3.0 Gbps or even to 1.5 Gbps (3.0 Gbps did not help, and was not even possible with the ASM1166 as the speed was always reset to 6.0 Gbps, but I could check with FCH and JMB585 controllers)
6.) Disabling NCQ (guess what, this helped!)
7.) Replacing the SATA controllers with an LSI 9211-8i (I guess this would have helped, as others have reported, because it probably does not use NCQ)
I will have a try to set link_power_management_policy to max_performance and see the result. And by the way, following changes it not permanent, it will disappear after reboot
ykun92 was very kind to elaborate on the SATA/PCI link power saving matter (which sounds very weird to me; how power saving can play a role in the middle of data transfers - but let's not think about that):
echo max_performance | sudo tee /sys/class/scsi_host/host*/link_power_management_policy
for permanent change
add a file to /etc/udev/rules.d and name it like 60-scsi.rules and edit the content like followingKERNEL=="host[0-2]", SUBSYSTEM=="scsi_host", ATTR{link_power_management_policy}="max_performance"
Some weeks later he came back: And after a month of monitoring, I am now sure set link_power_management_policy to max_performance solved the Unaligned write command issue completely, at least in my environment. I also confirmed this issue come back when turn link_power_management_policy back to the default value.
So that's exactly what I did and it worked well. Kudos c0d3z3r0, ykun92!
My LSPCI in case someone finds it interesting:
06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0])
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
Flags: bus master, fast devsel, latency 0, IRQ 48
Memory at fcd01000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/2 Maskable- 64bit+
Capabilities: [d0] SATA HBA v1.0
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [270] Secondary PCI Express
Capabilities: [2a0] Access Control Services
Capabilities: [400] Data Link Feature <?>
Capabilities: [410] Physical Layer 16.0 GT/s <?>
Capabilities: [440] Lane Margining at the Receiver <?>
Kernel driver in use: ahci
Kernel modules: ahci
06:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0])
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
Flags: bus master, fast devsel, latency 0, IRQ 50
Memory at fcd00000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/2 Maskable- 64bit+
Capabilities: [d0] SATA HBA v1.0
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [2a0] Access Control Services
Kernel driver in use: ahci
Kernel modules: ahci
Don't know what will come here, let's wait and see... But in the first instance it will be a login box (for myself):
Leave a comment
Comments (if any)