I have a drive currently resilvering that is taking a LONG time.
The TL;DR version:
The pool is a 8x6TB RAIDZ2, and is 81% full. I am trying to replace a drive in the array with a shucked WD Blue (WD60EZAZ). The resilver has been running for 2 days already. The write speed is dropping (29MB/s and falling), and the time to complete is increasing. The rate of % complete has stayed steady, at .03% per 10 minutes. At this rate, the resilver will take about 23 days to complete. I recently completed 2 separate resilvers on this array (see below for details) using WD 8TB Reds (WD80EFAX); each took about 20 hours (avg speed was 530MB/s). I’ve never had a resilver run this slowly (even on a Blue). What am I missing?
The long version:
The pool at the beginning of this saga was comprised of:
4x 6TB WD Red WD60EFRX
2x 6TB WD Blue WD60EZRZ
1x 6TB Seagate ST6000DX000 (7200rpm)
1x 6TB Mediamax WL6000GSA6457 (WD OEM drive, “white label” rebrand)
I wanted to replace the Seagate, as it runs hot (obviously). I recently picked up a 6TB WD Elements (Blue WD60EZAZ) to shuck and replace the Seagate. After starting, I tried to cancel the replace (to change location of the 2 drives), but I screwed it up, and was left with a degraded array. No problem; this is what RAIDZ2 is for. I set up the new drive in the system, and set about resilvering. After a couple of hours, one of the other drives in the array (the Mediamax) started throwing errors, and ultimately died. Now I’m scared: I have no redundancy left. After running the resilver with the Blue for a couple of days, the speed is insanely slow (sub 30MB/s), gstat shows the Busy% at 100, and I’m terrified of losing another drive. I have another system in development with a bunch of WD 8TB drives (Red WD80EFAX), and figured that they might be faster. They were; a lot. I ran the 2 resilvers sequentially (not simultaneously); each took about 20 hours at 530MB/s (gstat Busy% was around 40).
Now that the pool is healthy, I need my 8TB reds back for the dev machine, and I go to replace one with the Blue I bought. I figured it would be faster with the array healthy. Nope. Same as before. As stated above, the replacement is on track to take over 23 DAYS to complete.
I know that Blue drives are crappier than Reds, and expected it to be slower, but not by this much. I did this exact replacement a couple years ago (minus my screw-ups), replacing a Seagate with the Blue EZRZ. I don’t have exact times, but it didn’t take more than a day or two. WD’s spec sheet shows the only difference between the EZRZ and EZAZ to be the cache; 64 vs 256. It almost seems like this is an SMR drive, but I’m not aware of WD having any SMR drives.
Does anybody have any insight or suggestions? I’ve tried to include everything I can think of, and will attach output from zpool status and gstat in the next post. Please let me know if there’s any other info that would be helpful.
I greatly appreciate your help.
System specs:
FreeNAS-11.1-U7
Supermicro A1SAi-2750F
4x Kingston 8GB ECC DDR3-1600
IBM M1015 flashed to IT mode
(boot) Supermicro SSD-DM016-PHI SATA DOM
(jail & system, mirror) 2x Sandisk SDSSDP-128G-G25
(aux, mirror) 2x 4TB Seagate ST4000LM016
(main pool, RAIDZ2)
4x 6TB WD Red WD60EFRX
2x 6TB WD Blue WD60EZRZ
2x 8TB WD Red WD80EFAX (temporary replacements)
Silverstone DS380B w/ 3x Noctua NF-F12 iPPC-3000
Silverstone ST45SF-G 450W PSU
APC SUA1500 UPS