Shulou(Shulou.com)06/02 Report--

AIX rootvg error-- erroneous deletion of PV fault

System environment:

Operating system: AIX 5300-08

When doing the operation and maintenance of AIX system two days ago, the customer encountered the following cases:

Error phenomenon:

1. When viewing rootvg, a PV missing

[root@aix199 /] # lsvg-p rootvg

Rootvg:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

Hdisk0 active 524 0 00..00..00..00..00

0516-304: Unable to find device id 000681aa29d5ceff in the Device

Configuration Database.

000681aa29d5ceff missing 480 184 70..00..00..18..96

[root@aix199 /] #

2. Create a directory under / home, and there is an iUnip O error (you can create a normal file)

[root@aix199 /] # mkdir / home/aaa

Mkdir: cannot create / home/aaa.

/ home/aaa: I/O error

[root@aix199 /] # df-m

Filesystem MB blocks Free% Used Iused% Iused Mounted on/dev/hd4 10880.00 10012.31 8% 5890 1% / / dev/hd2 10560.00 8354.95 21% 45691 2% / usr/dev/hd9var 5120.00 4769.67 7% 1145 1% / var/dev/hd3 9856.00 9404.13 5% 440 1% / Tmp/dev/hd10opt 5120.00 4693.67 9% 4800 1% / opt/dev/lv00 5120.00 4959.23 4% 18 1% / var/adm/csd/dev/lv_soft 9600.00 8029.12 17% 2522 1% / soft/dev/hd1 5120.00 4877.34% 213 1% / home

It's not because of the lack of space.

Preliminary analysis, there should be two PV under rootvg, and a PV is deleted incorrectly, resulting in the above error!

Solution:

1. Delete the PV with missing rootvg normally.

[root@aix199 /] # reducevg rootvg 000681aa29d5ceff

0516-016 ldeletepv: Cannot delete physical volume with allocated partitions. Use either migratepv to move the partitions or reducevg with the-d option to delete the partitions.0516-884reducevg: Unable to remove physical volume 000681aa29d5ceff.

[root@aix199 /] # reducevg-d rootvg 000681aa29d5ceff

0516-914 rmlv: Warning, all data belonging to logical volume lv00 on physical volume 000681aa29d5ceff will be destroyed.rmlv: Do you wish to continue? Y (es) n (o)? Y0516-1008 rmlv: Logical volume lv00 must be closed. If the logical volume contains a filesystem, the umount command will close the LV device.0516-1008 rmlv: Logical volume hd9var must be closed. If the logical volume contains a filesystem, the umount command will close the LV device.0516-1008 rmlv: Logical volume hd10opt must be closed. If the logical volume contains a filesystem, the umount command will close the LV device.0516-884reducevg: Unable to remove physical volume 000681aa29d5ceff.

Part of the PP of LV in rootvg is allocated in the lost PV, including the logical volumes of hd9var and hd10opt. If the information of these PP is not cleared, the lost PV cannot be deleted.

[root@aix199 /] # lsvg-l rootvg

Rootvg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINThd5 boot 1 1 1 closed/syncd N/Ahd6 paging 8 8 1 open/syncd N/Ahd8 jfslog 1 1 1 open/syncd N/Ahd4 jfs 170 170 1 open/syncd / hd2 jfs 165 1 open/syncd / usr0516-1147: Warning-logical volume hd9var may be partially mirrored.hd9var jfs 80 81 3 open/stale / varhd3 jfs 1541 open/syncd / tmphd1 jfs 80 80 2 open/syncd / homehd10opt jfs 80 80 2 open/syncd / optlv00 jfs 80 80 2 open/syncd / var/adm/csd

[root@aix199 /] #

2. View the allocation information of PP on hd9var

[root@aix199 /] # lslv-m hd9var

Hd9var:/varLP PP1 PV1 PP2 PV2 PP3 PV30001 0215 hdisk0 0002 0504 hdisk0 0003 0509 hdisk0 0004 0510 hdisk0 0005 0513 hdisk0 0006 0514 hdisk0 0007 0521 hdisk0 0008 0522 hdisk0 0516304: Unable to find device id 000681aa29d5ceff in the Device Configuration Database.0009 0217 000681aa29d5ceff 0516304: Unable to find device id 000681aa29d5ceff In the Device Configuration Database.0010 0218 000681aa29d5ceff.

As can be seen from the above, on hdisk0, hd9var allocates a total of 8 PP, and the rest is allocated on the lost PV

3. Write PP allocation information to a temporary file

[root@aix199 /] # lquerylv-L `getlvodm-l hd9var`-r > / tmp/mapfile

Note: inverted quotation marks are used

View the PP allocation table and modify

[root@aix199 /] # cat / tmp/mapfile

0009affa94970f34 215 1 0009affa94970f34 504 2 0009affa94970f34 509 3 0009affa94970f34 510 4. 000681aa29d5ceff 286 78 000681aa29d5ceff 287 79 000681aa29d5ceff 288 80

[root@aix199 /] #

Note: total 80 PP

Keep the PP to be deleted in the file (the first 8 PP are in hdisk0):

[root@aix199 /] # cat / tmp/mapfile

000681aa29d5ceff 217 9 000681aa29d5ceff 218 10. 000681aa29d5ceff 287 79 000681aa29d5ceff 288 80

[root@aix199 /] #

4. Delete the PP assigned on the missing PV

[root@aix199 /] # wc-l / tmp/mapfile

73 / tmp/mapfile

[root@aix199 /] # lreducelv-l `getlvodm-l hd9var`-s 73 / tmp/mapfile

[root@aix199 /] #

5. View the information of hd9var

[root@aix199 /] # lslv hd9var

LOGICAL VOLUME: hd9var VOLUME GROUP: rootvgLV IDENTIFIER: 0008570c00004c0000000144684ecb4c.6 PERMISSION: read/writeVG STATE: active/complete LV STATE: opened/syncdTYPE: jfs WRITE VERIFY: offMAX LPs: 512 PP SIZE: 64 megabyte (s) COPIES: 1 SCHED POLICY: parallelLPs: 8 PPs: 8STALE PPs: 0 BB POLICY: relocatableINTER-POLICY: minimum RELOCATABLE: yesINTRA-POLICY: center UPPER BOUND: 32MOUNT POINT: / var LABEL: / varMIRROR WRITE CONSISTENCY: on/ACTIVE EACH LP COPY ON A SEPARATE PV?: yes Serialize IO?: NO

[root@aix199 /] # getlvcb-AT hd9var

AIX LVCB intrapolicy = c copies = 1 interpolicy = m lvid = 0008570c00004c0000000144684ecb4c.6 lvname = hd9var label = / var machine id = 8570C4C00 number lps = 8 relocatable = y strict = y stripe width = 0 stripe size in exponent = 0 type = jfs upperbound = 32 fs = Time created = Tue Feb 25 09:10:14 2014 time modified = Thu Mar 6 12:21:22 2014

Save configuration information:

[root@aix199 /] # savebase

[root@aix199 /] # lsvg-l rootvg

Rootvg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINThd5 boot 1 1 1 closed/syncd N/Ahd6 paging 8 8 1 open/syncd N/Ahd8 jfslog 1 1 1 open/syncd N/Ahd4 jfs 170 170 1 open/syncd / hd2 jfs 165 165 1 open/syncd / usrhd9var jfs 8 8 1 open/syncd / varhd3 jfs 154 154 1 open/syncd / tmphd1 jfs 80 80 2 closed / syncd / homehd10opt jfs 80 80 2 open/syncd / optlv00 jfs 3 3 1 closed/syncd / var/adm/csd

Deal with hd10opt in the same way:

[root@aix199 /] # lslv-m hd10opt

Hd10opt:/optLP PP1 PV1 PP2 PV2 PP3 PV30001 0218 hdisk0 0002 0229 hdisk0 0003 0505 hdisk0 0004 0506 hdisk0 0005 0507 hdisk0 0006 0511 hdisk0 0007 0512 hdisk0 0008 0517 hdisk0 0009 0518 hdisk0 0010 0523 hdisk0 0011 0524 hdisk0 0516-0005: Unable to find device Id 000681aa29d5ceff in the Device Configuration Database.0012 0193 000681aa29d5ceff 0516-304Unable to find device id 000681aa29d5ceff in the Device Configuration Database.0013 0194 000681aa29d5ceff 0516304: Unable to find device id 000681aa29d5ceff in the Device Configuration Database.0014 0195000681aa29d5ceff.

[root@aix199 /] # cat / tmp/mapfile

0009affa94970f34 218 1 0009affa94970f34 229 2 0009affa94970f34 505 3. 000681aa29d5ceff 190 78 000681aa29d5ceff 191 79 000681aa29d5ceff 192 80

[root@aix199 /] #

Keep the PP to delete in the temporary file:

[root@aix199 /] # cat / tmp/mapfile

000681aa29d5ceff 193 12 000681aa29d5ceff 194 13 000681aa29d5ceff 195 14. 000681aa29d5ceff 190 78 000681aa29d5ceff 191 79 000681aa29d5ceff 192 80

[root@aix199 /] # wc-l / tmp/mapfile

69 / tmp/mapfile

[root@aix199 /] #

[root@aix199 /] # lreducelv-l `getlvodm-l hd10opt`-s 69 / tmp/mapfile

[root@aix199 /] # lslv-m hd10opt

Hd10opt:/optLP PP1 PV1 PP2 PV2 PP3 PV30001 0218 hdisk0 0002 0229 hdisk0 0003 0505 hdisk0 0004 0506 hdisk0 0005 0507 hdisk0 0006 0511 hdisk0 0007 0512 hdisk0 0008 0517 hdisk0 0009 0518 hdisk0 0010 0523 hdisk0 0011 0524 hdisk0

6. Delete the missing PV in rootvg again

[root@aix199 /] # lsvg-p rootvg

Rootvg:PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTIONhdisk0 active 524 00.. 00.. 00.. 000516-304: Unable to find device id 000681aa29d5ceff in the Device Configuration Database.000681aa29d5ceff missing 480 480 96. 96. 96 [root@aix199 /] # reducevg-d rootvg 000681aa29d5ceff0516- 304putlvodm: Unable to find device id 000681aa29d5ceff0000000000000000 in the Device Configuration Database.0516-896reducevg: Warning Cannot remove physical volume 000681aa29d5ceff from Device Configuration Database. [root@aix199 /] # lsvg-p rootvgrootvg:PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTIONhdisk0 active 524000. 00. 00. 00.

Note: the missing PV has been deleted

7. Solve the error problem of Imax O

[root@aix199 /] # mkdir / home/aaa

Mkdir: cannot create / home/aaa.

/ home/aaa: I/O error

Note: there is still an error creating the directory under / home/

[root@aix199 /] # lslv-m hd1

Hd1:/homeLP PP1 PV1 PP2 PV2 PP3 PV30001 0217 hdisk0 0002 0515 hdisk0 0003 0516 hdisk0 [root@aix199 /] # lsvg rootvgVOLUME GROUP: rootvg VG IDENTIFIER: 0008570c00004c0000000144684ecb4cVG STATE: active PP SIZE: 64 megabyte (s) VG PERMISSION: read/write TOTAL PPs: 33536 megabytes) MAX LVs: 256 FREE PPs: 0 (0 megabytes) LVs: 10 USED PPs: 524 (33536 megabytes) OPEN LVs: 8 QUORUM: 2 (Enabled) TOTAL PVs: 1 VG DESCRIPTORS: 2STALE PVs: 0 STALE PPs: 0ACTIVE PVs: 1 AUTO ON: yesMAX PPs per VG: 32512 MAX PPs per PV: 1016 MAX PVs: 32LTG size (Dynamic): 2048 kilobyte (s) AUTO SYNC: noHOT SPARE: no BB POLICY: relocatable

[root@aix199 /] # lsvg-l rootvg

Rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINThd5 boot 1 1 1 closed/syncd N/Ahd6 paging 8 8 1 open/syncd N/Ahd8 jfslog 1 1 1 open/syncd N/Ahd4 jfs 170 170 1 open/syncd / hd2 jfs 165 165 1 open/syncd / usrhd9var jfs 8 8 1 open/syncd / varhd3 jfs 154 154 1 open/syncd / tmphd1 jfs 3 3 1 open / syncd / homehd10opt jfs 11 11 1 open/syncd / optlv00 jfs 3 3 1 closed/syncd / var/adm/csd

[root@aix199 /] # df-m

Filesystem MB blocks Free% Used Iused% Iused Mounted on/dev/hd4 10880.00 10012.24 8% 5890 1% / / dev/hd2 10560.00 8354.95 21% 45691 2% / usr/dev/hd9var 5120.00 4769.67 7% 1145 1% / var/dev/hd3 9856.00 9404.13 5% 441% / Tmp/dev/hd10opt 5120.00 4693.67 9% 4800 1% / opt/dev/lv_soft 9600.00 8029.12 17% 2522 1% / soft/dev/hd1 5120.00 4877.34 5% 213.1% / home above The LV corresponding to / home only allocates 3 PP, but the space shown is 5120m, so the space of / home should still be using the lost PV.

8. Back up / home and delete / home file system for re-establishment

[root@aix199 /] # umount / home

[root@aix199 /] # smit rmfs

After the / home file system is deleted, the hd1 logical volume is also deleted

[root@aix199 /] # lsvg-l rootvg

Rootvg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINThd5 boot 1 1 1 closed/syncd N/Ahd6 paging 8 8 1 open/syncd N/Ahd8 jfslog 1 1 1 open/syncd N/Ahd4 jfs 170 170 1 open/syncd / hd2 jfs 165 165 1 open/syncd / usrhd9var jfs 8 8 1 open/syncd / varhd3 jfs 154 154 1 open/syncd / tmphd10opt jfs 11 11 1 open/syncd / optlv00 jfs 3 3 1 open/syncd / var/adm/csd

9. Re-establish the logical volume of hd1 and mount to / home

[root@aix199 /] # lsvg rootvg

VOLUME GROUP: rootvg VG IDENTIFIER: 0008570c00004c0000000144684ecb4cVG STATE: active PP SIZE: 64 megabyte (s) VG PERMISSION: read/write TOTAL PPs: 524 (33536 megabytes) MAX LVs: 256 FREE PPs: 3 (192 megabytes) LVs: 9 USED PPs: 521 (33344 megabytes) OPEN LVs: 8 QUORUM: 2 (Enabled) TOTAL PVs: 1 VG DESCRIPTORS: 2STALE PVs: 0 STALE PPs: 0ACTIVE PVs: 1 AUTO ON: yesMAX PPs per VG: 32512 MAX PPs per PV: 1016 MAX PVs: 32LTG size (Dynamic): 2048 kilobyte (s) AUTO SYNC: noHOT SPARE: no BB POLICY: relocatable

[root@aix199 /] # lsvg-l rootvg

Rootvg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINThd5 boot 1 1 1 closed/syncd N/Ahd6 paging 8 8 1 open/syncd N/Ahd8 jfslog 1 1 1 open/syncd N/Ahd4 jfs 170 170 1 open/syncd / hd2 jfs 165 165 1 open/syncd / usrhd9var jfs 8 8 1 open/syncd / varhd3 jfs 154 154 1 open/syncd / tmphd1 jfs 2 2 1 Closed/syncd N/Ahd10opt jfs 11 11 1 open/syncd / optlv00 jfs 3 3 1 open/syncd / var/adm/csd

[root@aix199 /] # mount / home

Verify:

[root@aix199 /] # ls / homelost+found [root@aix199 /] # mkdir / home/aa [root@aix199 /] # ls-l / hometotal 16drwxr-sr-x 2 root sys 512 May 4 17:54 aadrwxrwx--- 2 root system 512 May 4 17:53 lost+found [root@aix199 /] # df-mFilesystem MB blocks Free% Used Iused% Iused Mounted on/dev/hd4 10880.00 10012.22 8 5893 1% / / dev/hd2 10560.00 8354.95 21% 45691 2% / usr/dev/hd9var 5120.00 4769.67 7% 1145 1% / var/dev/hd3 9856.00 9404.12 5% 4411% / tmp/dev/hd10opt 5120.00 4693.67% 4800 1% / opt/dev/lv00 5120.00 4959.23 4% 18 1% / var/adm/csd/dev/lv_soft 9600.00 8029.12 17% 2522 1% / soft/dev/hd1 128.00 123.94 4% 18 1% / home

@ at this point, the problem is solved. When adding PV to rootvg, be sure to use local disks instead of disks on the array, and choose the right steps when deleting PV!

Documentation reference:

Http://www.doc88.com/p-687602374822.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.