Mdadm Raid 5 startet nicht nach Neustart (IO Error & Dirty Degraded)

[davidberryman]

Hallo,
Ich habe gestern eine neue 3TB HDD zu meinem funktionierenden Raid5 Array hinzugefügt und es übernacht rebuilden lassen. Heute hatte ich dann folgenden Fehler im Journal:

Jan 16 07:49:42 iHugo kernel: INFO: task md0_resync:854 blocked for more than 120 seconds.
Jan 16 07:49:42 iHugo kernel: task:md0_resync      state:D stack:    0 pid:  854 ppid:     2 flags:0x00004000
Jan 16 07:49:42 iHugo kernel: INFO: task jbd2/md0p1-8:1006 blocked for more than 120 seconds.
Jan 16 07:49:42 iHugo kernel: task:jbd2/md0p1-8    state:D stack:    0 pid: 1006 ppid:     2 flags:0x00004000
Jan 16 07:51:43 iHugo kernel: INFO: task md0_resync:854 blocked for more than 241 seconds.
Jan 16 07:51:43 iHugo kernel: task:md0_resync      state:D stack:    0 pid:  854 ppid:     2 flags:0x00004000
Jan 16 07:51:43 iHugo kernel: INFO: task jbd2/md0p1-8:1006 blocked for more than 241 seconds.
Jan 16 07:51:43 iHugo kernel: task:jbd2/md0p1-8    state:D stack:    0 pid: 1006 ppid:     2 flags:0x00004000
Jan 16 07:53:44 iHugo kernel: INFO: task md0_resync:854 blocked for more than 362 seconds.
Jan 16 07:53:44 iHugo kernel: task:md0_resync      state:D stack:    0 pid:  854 ppid:     2 flags:0x00004000
Jan 16 07:53:44 iHugo kernel: INFO: task jbd2/md0p1-8:1006 blocked for more than 362 seconds.
Jan 16 07:53:44 iHugo kernel: task:jbd2/md0p1-8    state:D stack:    0 pid: 1006 ppid:     2 flags:0x00004000
Jan 16 07:55:45 iHugo kernel: INFO: task md0_resync:854 blocked for more than 483 seconds.
Jan 16 07:55:45 iHugo kernel: task:md0_resync      state:D stack:    0 pid:  854 ppid:     2 flags:0x00004000
Jan 16 07:55:45 iHugo kernel: INFO: task jbd2/md0p1-8:1006 blocked for more than 483 seconds.
Jan 16 07:55:45 iHugo kernel: task:jbd2/md0p1-8    state:D stack:    0 pid: 1006 ppid:     2 flags:0x00004000
Jan 16 07:57:45 iHugo kernel: INFO: task md0_resync:854 blocked for more than 604 seconds.
Jan 16 07:57:45 iHugo kernel: task:md0_resync      state:D stack:    0 pid:  854 ppid:     2 flags:0x00004000
Jan 16 07:57:45 iHugo kernel: INFO: task jbd2/md0p1-8:1006 blocked for more than 604 seconds.
Jan 16 07:57:45 iHugo kernel: task:jbd2/md0p1-8    state:D stack:    0 pid: 1006 ppid:     2 flags:0x00004000

Dann habe ich versucht den Server zu rebooten, was zu folgendem Fehler geführt hat:

Jan 16 09:17:26 iHugo blkdeactivate[82348]:   [MD]: deactivating part device md0p1...
Jan 16 09:17:26 iHugo blkdeactivate[82359]: cat: /sys/block/md0p1/md/sync_action: No such file or directory
Jan 16 09:35:58 iHugo kernel: md/raid:md0: not clean -- starting background reconstruction
Jan 16 09:35:58 iHugo kernel: md/raid:md0: device sdd operational as raid disk 1
Jan 16 09:35:58 iHugo kernel: md/raid:md0: device sdf1 operational as raid disk 3
Jan 16 09:35:58 iHugo kernel: md/raid:md0: device sdb operational as raid disk 0
Jan 16 09:35:58 iHugo kernel: md/raid:md0: force stripe size 512 for reshape
Jan 16 09:35:58 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 09:35:58 iHugo kernel: md/raid:md0: failed to run raid set.

Hier sind ein paar Details:

mdstat

root@iHugo:~# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : inactive sdf1[4] sdd[1] sde[3] sdb[0]
8790276327 blocks super 1.2

unused devices: <none>

mdadm -D

root@iHugo:~# mdadm -D /dev/md0
mdadm: Unknown keyword INACTIVE-ARRAY /dev/md0:
           Version : 1.2
     Creation Time : Thu Jan 13 18:57:19 2022
        Raid Level : raid5
     Used Dev Size : 1953378304 (1862.89 GiB 2000.26 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Sun Jan 16 07:47:20 2022
             State : active, FAILED, Not Started
    Active Devices : 3    Working Devices : 4
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : unknown

     Delta Devices : 1, (3->4)

              Name : iHugo:0  (local to host iHugo)
              UUID : dc5e662f:4f32bd91:95ee7139:7ef94601
            Events : 59708

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
       -       0        0        2      removed
       -       0        0        3      removed

       -       8       64        2      spare rebuilding   /dev/sde
       -       8       48        1      sync   /dev/sdd
       -       8       16        0      sync   /dev/sdb
       -       8       81        3      sync   /dev/sdf1

fdisk -l

root@iHugo:~# fdisk -l
Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 840
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1B771FD2-DAB6-428D-88A6-0A1F1D35671E

Device         Start       End   Sectors   Size Type
/dev/sda1       2048   1050623   1048576   512M EFI System
/dev/sda2    1050624 486395903 485345280 231.4G Linux filesystem
/dev/sda3  486395904 488396799   2000896   977M Linux swap


Disk /dev/sdf: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000DM001-1CH1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DCF1340A-3DA8-42EE-B7B1-7F439E571148

Device     Start        End    Sectors  Size Type
/dev/sdf1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: SAMSUNG HD103UJ
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EFBD098F-4397-4EC9-B242-1712149A75C9

Device     Start        End    Sectors   Size Type
/dev/sdc1   2048 1953525134 1953523087 931.5G Linux filesystem


Disk /dev/sde: 1.82 TiB, 2000394706432 bytes, 3907020911 sectors
Disk model: WDC WD20EARS-00J
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EARS-00M
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

blkid

root@iHugo:~# blkid
/dev/sdb: UUID="dc5e662f-4f32-bd91-95ee-71397ef94601" UUID_SUB="2a63392a-2f36-5e40-509a-8a968c132b66" LABEL="iHugo:0" TYPE="linux_raid_member"
/dev/sda1: UUID="CC0A-CBAA" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="a8c63d30-5ddd-4a1f-b9d5-faed36434457"
/dev/sda2: UUID="7fe4974d-d6ae-4c09-a8b7-8cb46ab978b8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="39f49ee3-8061-4da8-9802-a77beaec158a"
/dev/sda3: UUID="e668eb26-e954-48d9-9fba-f92f6437c49f" TYPE="swap" PARTUUID="95f3627d-1e7f-4fa8-a39b-d91b2b2b7012"
/dev/sdf1: UUID="dc5e662f-4f32-bd91-95ee-71397ef94601" UUID_SUB="336f8167-c619-4553-8ab2-0b4516106ae1" LABEL="iHugo:0" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="26c1353a-0796-48ba-92db-d4fdae4f7f98"
/dev/sdc1: LABEL="Leer" BLOCK_SIZE="512" UUID="01D437C7F0FED880" TYPE="ntfs" PARTUUID="975d1223-be02-471d-a291-e3433048e0ee"
/dev/sde: UUID="dc5e662f-4f32-bd91-95ee-71397ef94601" UUID_SUB="b66d96aa-5e60-83f2-8a8c-9f8c3d1caf65" LABEL="iHugo:0" TYPE="linux_raid_member"
/dev/sdd: UUID="dc5e662f-4f32-bd91-95ee-71397ef94601" UUID_SUB="6d33f5c9-e81f-3965-da7b-37a2366ed1d1" LABEL="iHugo:0" TYPE="linux_raid_member"

Die Platten scheinen vom System erkannt zu werden, aber irgendwie startet das Raid nicht.
Bisher habe ich folgendes probiert:

Raid neustarten mit mdadm --run

root@iHugo:~# mdadm --run /dev/md0
mdadm: Unknown keyword INACTIVE-ARRAY
mdadm: failed to start array /dev/md/iHugo:0: Input/output error

force reassemble

root@iHugo:~# mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdd /dev/sde /dev/sdf1
mdadm: Unknown keyword INACTIVE-ARRAY
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error

Vielen Dank für jegliche Hilfe im Vorhinein

 

[Piktogramm]

Bevor du mdadm ein --force mitgibst, solltest du (nach Möglichkeit) Images aller betroffenen Laufwerke ziehen!

Was interessant wäre, wäre ob Fehler in den Logs zu finden sind. Der erste Ansatz wäre journalctl -p 0..2 wobei der Bereich 0..2 die Fehlerlevel angeht (siehe manpage). Dummerweise sind die Voreinstellungen von journald mitunter ungünstig und es wäre ne Maßnahme /etc/systemd/journald.conf zu ändern.

Storage=persistent
SystemMaxUse=64M

Und dann wäre es auch einen Blick wert, was die SMART-Werte der Laufwerke andeuten mittels smartctl -a /dev/sdX

 

[der-graph]

warum eig. sdf1 und nicht sdf? Ist nicht da was schiefgelaufen und f1 müsste erst mal weg und f rein

 

[davidberryman]

Bevor du mdadm ein --force mitgibst, solltest du (nach Möglichkeit) Images aller betroffenen Laufwerke ziehen!

Bringt das denn bei nem Raid5 was? Könnte ich damit im Fehlerfall ein ganzes Laufwerk austauschen?

Journalctl -p 0..2:

-- Boot a2b3aa9bd597471d9e9d2b6f3d3617c2 --
Jan 16 09:35:58 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 10:05:46 iHugo openmediavault-webgui[1781]: Authorized login from xxx.xxx.xxx.xxx [...]
Jan 16 10:12:04 iHugo openmediavault-webgui[2542]: Authorized login from xxx.xxx.xxx.xxx [...]
Jan 16 10:26:00 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 10:26:01 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 10:31:21 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 10:31:22 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 11:19:19 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 11:37:50 iHugo openmediavault-webgui[4909]: xxx.xxx.xxx.xxx [...]
Jan 16 11:43:51 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:16:51 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:16:51 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:37:18 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:38:24 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:38:24 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 12:41:46 iHugo kernel: md/raid:md0: cannot start dirty degraded array.
Jan 16 13:24:52 iHugo openmediavault-webgui[7080]:  xxx.xxx.xxx.xxx [...]

Smartctl /dev/sdb:

smartctl -a /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-10-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M1VPNX7Z
LU WWN Device Id: 5 0014ee 2b838193b
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Jan 16 15:27:15 2022 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (25500) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 258) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   171   021    Pre-fail  Always       -       4133
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       82
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       6639
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       82
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       55
193 Load_Cycle_Count        0x0032   193   193   000    Old_age   Always       -       22654
194 Temperature_Celsius     0x0022   110   081   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Smartctl /dev/sdd

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF)
Device Model:     WDC WD20EARS-00MVWB0
Serial Number:    WD-WCAZA0761345
LU WWN Device Id: 5 0014ee 204f31e7c
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Sun Jan 16 15:28:07 2022 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (38880) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 375) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   191   191   051    Pre-fail  Always       -       8181
  3 Spin_Up_Time            0x0027   169   168   021    Pre-fail  Always       -       6533
  4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3446
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   012   012   000    Old_age   Always       -       64747
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       366
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       267
193 Load_Cycle_Count        0x0032   132   132   000    Old_age   Always       -       206080
194 Temperature_Celsius     0x0022   109   084   000    Old_age   Always       -       41
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   197   196   000    Old_age   Always       -       1255
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 1
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 64687 hours (2695 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 81 4f c2 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d4 00 81 4f c2 00 08      00:53:15.260  SMART EXECUTE OFF-LINE IMMEDIATE
  c8 00 88 80 a6 08 e8 08      00:53:15.260  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short captive       Completed: read failure       90%     64687         1864964898
# 2  Short offline       Aborted by host               10%     64687         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Smartctl /dev/sde

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (AF)
Device Model:     WDC WD20EARS-00J2GB0
Serial Number:    WD-WCAYY0198110
LU WWN Device Id: 5 0014ee 25a254b56
Firmware Version: 80.00A80
User Capacity:    2,000,394,706,432 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Sun Jan 16 15:28:58 2022 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40260) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 459) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   164   157   021    Pre-fail  Always       -       8783
  4 Start_Stop_Count        0x0032   094   094   000    Old_age   Always       -       6058
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   011   011   000    Old_age   Always       -       65312
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       318
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       218
193 Load_Cycle_Count        0x0032   108   108   000    Old_age   Always       -       277176
194 Temperature_Celsius     0x0022   112   080   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Smartctl /dev/sdf

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  592) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 339) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   117   099   006    Pre-fail  Always       -       153237464
  3 Spin_Up_Time            0x0003   092   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   095   095   020    Old_age   Always       -       6090
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   058   058   030    Pre-fail  Always       -       25773758696
  9 Power_On_Hours          0x0032   047   047   000    Old_age   Always       -       47292
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       325
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   082   000    Old_age   Always       -       7 7 36
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   062   042   045    Old_age   Always   In_the_past 38 (Min/Max 37/41 #67)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       175
193 Load_Cycle_Count        0x0032   094   094   000    Old_age   Always       -       13462
194 Temperature_Celsius     0x0022   038   058   000    Old_age   Always       -       38 (0 15 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       3
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       555h+46m+09.463s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       14793642707
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       20278376213

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Bei der /dev/sdb scheint ein Fehler zu sein, könnte der hiermit was zutun haben?

warum eig. sdf1 und nicht sdf? Ist nicht da was schiefgelaufen und f1 müsste erst mal weg und f rein

Stimmt. Das hatte ich nach beginn des Rebuilds auch festgestellt, dass ich da eine extra Partition erstellt hatte und bei den anderen Platten nicht. Kann das auch zu einem fehlerhaftem Rebuild führen?

Und falls ja, kann ich jetzt ohne weiteres die sdf platte aushängen und den Rebuild so ausführen lassen? Schließlich war ja bevor ich sdf angefügt hatte das Raid auch nur auf sdb sdd und sde.

 

[Inzersdorfer]

sdb: "No Errors Logged"

sdd: 8181 Lesefehler, 1255 schwebende Sektoren (die nicht gelesen werden können, ob die defekt sind zeigt sich erst nach dem nächsten Beschreiben Selbiger).

sde: 1 Schreibfehler, der ist aber nicht aktuell.

sdf: 6 Suchfehler, 24 command timeouts und älter 3 CRC Fehler.

sdd ersetzten angesichts der Fehler und der 65.000 Betriebsstunden, auch die sdf dürfte für ein Raid nicht mehr im optimalen Zustend sein.

 

[davidberryman]

Alles klar. Das Problem ist aber, dass da natürlich noch Daten drauf sind. Und jetzt ohne dass der Rebuild fertig gelaufen war würde ich ungerne irgendeine Platte austauschen. Oder sollte das gehen? Bzw. was wäre jetzt die sinnvollste Variante die Daten noch irgendwie zu retten?

 

[Inzersdorfer]

Wenn sdf beim rebuild das Problemkind wär, sollte der rebuild ohne sie klappen, wenn nicht sdd dzwischen funkt.

Das Problem ist aber, dass da natürlich noch Daten drauf sind

Wo ist die Datensicherung?

 

[Piktogramm]

Images ziehen von Laufwerken hilft auch bei Raids. mdadm schreibt ja auch nur Verwaltungsdaten auf die Laufwerke in Form von Bits und Bytes. Wie Inzersdorfer bereits schreibt, zwei HDDs sind mindestens fragwürdig. Anstatt denen ein Rebuild zuzutrauen würde ich mindestens von allten alten HDDs 1zu1 Kopien erstellen.

Beim Journal, vom letzten Boot an bringt es fast nichts. Logs aus dem Bereich wo mdadm die ersten Fehler gemeldet hat wären sinnvoll. Wenn du die nicht hast gewöhne dir an in Zukunft besagte journald.conf wie vorgeschlagen auf jedem System abzuändern.

So weiter mit mdadm --examine auf jedes betreffende Laufwerk bzw. Partition

~$ sudo mdadm --examine /dev/sda1 
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 64f57547:155c4810:b94a8508:0366e184
           Name : hostname-server:0
  Creation Time : Sun Dec 17 02:21:49 2017
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5860260785 (2794.39 GiB 3000.45 GB)
     Array Size : 14650649600 (13971.95 GiB 15002.27 GB)
  Used Dev Size : 5860259840 (2794.39 GiB 3000.45 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=8192 sectors
          State : clean
    Device UUID : dfbdb488:a8de0d60:5cae6889:418108ed

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jan 16 20:02:01 2022
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : f7ee8f3c - correct
         Events : 32330

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

Mit Abstand am Interessantesten ist hier Events: .. Auf allen Laufwerken, die regulär teil des Raids sind, sollte der Event counter exakt gleich sein. Jede Abweichung bedeutet defakto Datenverlust und mdadm weigert sich. Wenn ich mich richtig entsinne hilft da dann nur noch, dass Raid aufzulösen und ein neues Rais anzulegen, bei dem man jede Einheit mit abweichendem Eventcounter in den Verbund zwingt.
TU DIES NICHT ohne Images von den Platten gezogen zu haben und kontrolliere VORHER wie es um deine Backups etwaiger wichtiger Daten aussieht, die auf dem Raid gelegen haben!

Und für das nächste mal bei einem neuen Raid. mdadm kann seit einer Weile journaling, was das sogenannte "Write-Hole" beseitigt. Tendenziell hilft das gegen Laufwerkausfälle während eines Rebuilds und ist dringend anzuraten. BTRFS kann dass mittlerweile auch um das selbe Problem zu bekämpfen. Fummlig aber lohnend. Oder direkt auf ZFS umsteigen.
Egal welche der drei Varianten du nutzt, der Aufwand lohnt