Comments on: Recovering from pvmove failure http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/ Thu, 22 Dec 2011 18:53:23 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Turbo Fredriksson http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/comment-page-1/#comment-403 Turbo Fredriksson Sat, 18 Sep 2010 08:29:45 +0000 http://www.dmi.me.uk/blog/?p=203#comment-403 I had almost the same problem, with the exeption that the pvmove failed, removing the temporary LV, leaving the PV 'empty'. The cfg restore wasn't enough for me. My PV was still showing up as empty. But I did the vgcfgrestore, vgscan et al. The PV still showed as empty... But I also did a 'pvmove --abort' (even though it seems that there WAS no move in progress). Then I want to bed, didn't want to risk doing something worse due to fatigue :). When I woke up, I checked the PV. It was full again!! So give it some time... Don't panic! This is actually the most important part!! DO. NOT. PANIC. Oh, and don't reboot if you can avoid it (I didn't). This is what my (failed) pvmove gave me: <cite> celia:~# pvmove -v /dev/md6 /dev/md9 Finding volume group "movies" Archiving volume group "movies" metadata (seqno 86). Creating logical volume pvmove0 Moving 119234 extents of logical volume movies/movies Found volume group "movies" Updating volume group metadata Creating volume group backup "/etc/lvm/backup/movies" (seqno 87). Found volume group "movies" Found volume group "movies" Suspending movies-movies (253:1) with device flush Found volume group "movies" Found volume group "movies" Creating movies-pvmove0 device-mapper: create ioctl failed: Enhet eller resurs upptagen Loading movies-movies table device-mapper: reload ioctl failed: Ogiltigt argument Checking progress every 15 seconds WARNING: dev_open(/dev/md6) called while suspended WARNING: dev_open(/dev/md2) called while suspended WARNING: dev_open(/dev/md3) called while suspended WARNING: dev_open(/dev/md4) called while suspended WARNING: dev_open(/dev/md5) called while suspended WARNING: dev_open(/dev/md8) called while suspended WARNING: dev_open(/dev/md9) called while suspended WARNING: dev_open(/dev/md6) called while suspended WARNING: dev_open(/dev/md2) called while suspended WARNING: dev_open(/dev/md3) called while suspended WARNING: dev_open(/dev/md4) called while suspended WARNING: dev_open(/dev/md5) called while suspended WARNING: dev_open(/dev/md8) called while suspended WARNING: dev_open(/dev/md9) called while suspended WARNING: dev_open(/dev/md6) called while suspended WARNING: dev_open(/dev/md2) called while suspended WARNING: dev_open(/dev/md3) called while suspended WARNING: dev_open(/dev/md4) called while suspended WARNING: dev_open(/dev/md5) called while suspended WARNING: dev_open(/dev/md8) called while suspended WARNING: dev_open(/dev/md9) called while suspended /dev/md6: Moved: 100,0% Found volume group "movies" Found volume group "movies" Found volume group "movies" Found volume group "movies" Suspending movies-pvmove0 (253:0) with device flush Found volume group "movies" Creating movies-pvmove0 device-mapper: create ioctl failed: Enhet eller resurs upptagen Unable to reactivate logical volume "pvmove0" Found volume group "movies" Loading movies-movies table Resuming movies-movies (253:1) Found volume group "movies" Removing movies-pvmove0 (253:0) Found volume group "movies" Removing temporary pvmove LV Writing out final volume group after pvmove Creating volume group backup "/etc/lvm/backup/movies" (seqno 89). celia:~# pvmove -v /dev/md6 /dev/md9 Finding volume group "movies" Archiving volume group "movies" metadata (seqno 89). Creating logical volume pvmove0 No data to move for movies celia:~# pvdisplay /dev/md6 --- Physical volume --- PV Name /dev/md6 VG Name movies PV Size 465,76 GB / not usable 1,44 MB Allocatable yes PE Size (KByte) 4096 Total PE 119234 Free PE 119234 Allocated PE 0 PV UUID O4Stdr-Izas-1rwv-aPoF-2aNu-15ZR-kfxi2Y celia:~# less pvdisplay.txt [...] --- Physical volume --- PV Name /dev/md6 VG Name movies PV Size 465,76 GB / not usable 1,44 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 119234 Free PE 0 Allocated PE 119234 PV UUID O4Stdr-Izas-1rwv-aPoF-2aNu-15ZR-kfxi2Y </cite> The pvdisplay.txt text was before I did the pvmove... I had almost the same problem, with the exeption that the pvmove failed, removing the temporary LV, leaving the PV ‘empty’.

The cfg restore wasn’t enough for me. My PV was still showing up as empty. But I did the vgcfgrestore, vgscan et al. The PV still showed as empty… But I also did a ‘pvmove –abort’ (even though it seems that there WAS no move in progress). Then I want to bed, didn’t want to risk doing something worse due to fatigue :) . When I woke up, I checked the PV. It was full again!!

So give it some time… Don’t panic! This is actually the most important part!! DO. NOT. PANIC.

Oh, and don’t reboot if you can avoid it (I didn’t).

This is what my (failed) pvmove gave me:

celia:~# pvmove -v /dev/md6 /dev/md9
Finding volume group “movies”
Archiving volume group “movies” metadata (seqno 86).
Creating logical volume pvmove0
Moving 119234 extents of logical volume movies/movies
Found volume group “movies”
Updating volume group metadata
Creating volume group backup “/etc/lvm/backup/movies” (seqno 87).
Found volume group “movies”
Found volume group “movies”
Suspending movies-movies (253:1) with device flush
Found volume group “movies”
Found volume group “movies”
Creating movies-pvmove0
device-mapper: create ioctl failed: Enhet eller resurs upptagen
Loading movies-movies table
device-mapper: reload ioctl failed: Ogiltigt argument
Checking progress every 15 seconds
WARNING: dev_open(/dev/md6) called while suspended
WARNING: dev_open(/dev/md2) called while suspended
WARNING: dev_open(/dev/md3) called while suspended
WARNING: dev_open(/dev/md4) called while suspended
WARNING: dev_open(/dev/md5) called while suspended
WARNING: dev_open(/dev/md8) called while suspended
WARNING: dev_open(/dev/md9) called while suspended
WARNING: dev_open(/dev/md6) called while suspended
WARNING: dev_open(/dev/md2) called while suspended
WARNING: dev_open(/dev/md3) called while suspended
WARNING: dev_open(/dev/md4) called while suspended
WARNING: dev_open(/dev/md5) called while suspended
WARNING: dev_open(/dev/md8) called while suspended
WARNING: dev_open(/dev/md9) called while suspended
WARNING: dev_open(/dev/md6) called while suspended
WARNING: dev_open(/dev/md2) called while suspended
WARNING: dev_open(/dev/md3) called while suspended
WARNING: dev_open(/dev/md4) called while suspended
WARNING: dev_open(/dev/md5) called while suspended
WARNING: dev_open(/dev/md8) called while suspended
WARNING: dev_open(/dev/md9) called while suspended
/dev/md6: Moved: 100,0%
Found volume group “movies”
Found volume group “movies”
Found volume group “movies”
Found volume group “movies”
Suspending movies-pvmove0 (253:0) with device flush
Found volume group “movies”
Creating movies-pvmove0
device-mapper: create ioctl failed: Enhet eller resurs upptagen
Unable to reactivate logical volume “pvmove0″
Found volume group “movies”
Loading movies-movies table
Resuming movies-movies (253:1)
Found volume group “movies”
Removing movies-pvmove0 (253:0)
Found volume group “movies”
Removing temporary pvmove LV
Writing out final volume group after pvmove
Creating volume group backup “/etc/lvm/backup/movies” (seqno 89).
celia:~# pvmove -v /dev/md6 /dev/md9
Finding volume group “movies”
Archiving volume group “movies” metadata (seqno 89).
Creating logical volume pvmove0
No data to move for movies
celia:~# pvdisplay /dev/md6
— Physical volume —
PV Name /dev/md6
VG Name movies
PV Size 465,76 GB / not usable 1,44 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 119234
Free PE 119234
Allocated PE 0
PV UUID O4Stdr-Izas-1rwv-aPoF-2aNu-15ZR-kfxi2Y
celia:~# less pvdisplay.txt
[...]
— Physical volume —
PV Name /dev/md6
VG Name movies
PV Size 465,76 GB / not usable 1,44 MB
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 119234
Free PE 0
Allocated PE 119234
PV UUID O4Stdr-Izas-1rwv-aPoF-2aNu-15ZR-kfxi2Y

The pvdisplay.txt text was before I did the pvmove…

]]> By: Dave http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/comment-page-1/#comment-335 Dave Fri, 08 Jan 2010 18:46:26 +0000 http://www.dmi.me.uk/blog/?p=203#comment-335 Indeed it would. I believe S3 do accept hard drives in some situations (http://aws.amazon.com/importexport/) but that depends on cost- and time-effectiveness. I've just prodded Katie about the email, and she's looking into it now. Her excuse is that she's been ill (very true) and that nothing requiring intellect has been done. Indeed it would. I believe S3 do accept hard drives in some situations (http://aws.amazon.com/importexport/) but that depends on cost- and time-effectiveness.

I’ve just prodded Katie about the email, and she’s looking into it now. Her excuse is that she’s been ill (very true) and that nothing requiring intellect has been done.

]]> By: David Durant http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/comment-page-1/#comment-334 David Durant Fri, 08 Jan 2010 15:11:11 +0000 http://www.dmi.me.uk/blog/?p=203#comment-334 *nods* - My initial upload with JD took circa 4-5 days. For people with serious amounts of data (I'm only syncing ~400 Meg probably) it would be quicker if the service took DVDs by post ("never underestimate the bandwidth of a Volvo full of DVDs" :-) ). Oh - and Katie still hasn't replied to David P. *nods* – My initial upload with JD took circa 4-5 days. For people with serious amounts of data (I’m only syncing ~400 Meg probably) it would be quicker if the service took DVDs by post (“never underestimate the bandwidth of a Volvo full of DVDs” :-) ).

Oh – and Katie still hasn’t replied to David P.

]]> By: Dave http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/comment-page-1/#comment-333 Dave Fri, 08 Jan 2010 13:14:13 +0000 http://www.dmi.me.uk/blog/?p=203#comment-333 I've got a sync system (s3cmd) -- it's just that the upload speed here (~5KiB/s gusting up to 20KiB/s) sucks enough that the initial upload would take forever. Just uploading 5400 photos (~4GiB) has so far been running continuously since shortly after I finished recovery on the evening of the 4th... with 800 still to go as of this moment. I’ve got a sync system (s3cmd) — it’s just that the upload speed here (~5KiB/s gusting up to 20KiB/s) sucks enough that the initial upload would take forever. Just uploading 5400 photos (~4GiB) has so far been running continuously since shortly after I finished recovery on the evening of the 4th… with 800 still to go as of this moment.

]]> By: David Durant http://www.dmi.me.uk/blog/2010/01/07/recovering-from-pvmove-failure/comment-page-1/#comment-332 David Durant Fri, 08 Jan 2010 00:42:36 +0000 http://www.dmi.me.uk/blog/?p=203#comment-332 This is why JungleDisk (or equivalent) is your friend. We used to cross-backup between Annie's and my PC overnight but now we both use JD to push the diffs up to EC2. Seemless, works on Windows and Linux, zero effort and about $5 a month. This is why JungleDisk (or equivalent) is your friend. We used to cross-backup between Annie’s and my PC overnight but now we both use JD to push the diffs up to EC2. Seemless, works on Windows and Linux, zero effort and about $5 a month.

]]>