Shulou(Shulou.com)06/03 Report--

AIX system failure-- extended file system failure

System environment:

Operating system: AIX5300-08

Oracle: Oracle 10gR2 (64)

Failure phenomenon:

When smit chfs extends "/", it is prompted that "device" cannot be found; after rootvg extends pv, pv is destroyed, resulting in ODM error

1. View rootvg

[root@aix216 /] # lsvg-p rootvg

Rootvg:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

Hdisk0 active 546 161 107..00..00..12..42

0516-304: Unable to find device id 0000c908116d1325 in the Device

Configuration Database.

0000c908116d1325 missing 320 320 64..64..64..64..64

[root@aix216 /] #

-try to solve the problem

[root@aix216 /] # redefinevg-d hdisk0 rootvg

[root@aix216 /] # lsvg-p rootvg

Rootvg:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

Hdisk0 active 546 161 107..00..00..12..42

0516-304: Unable to find device id 0000c908116d1325 in the Device

Configuration Database.

0000c908116d1325 missing 320 320 64..64..64..64..64

[root@aix216 /] # lspv

Hdisk0 0027713e1b5646e5 rootvg active

[root@aix216 /] # reducevg

0516-606 reducevg: Volume group name not entered.

Usage: reducevg [- d] [- f] VGname PVname...

Reduces volume group size by removing a physical volume.

2. Delete hdisk0 from rootvg

-reducevg after deleting pv, hd5 is deleted

[root@aix216 /] # reducevg rootvg 0027713e1b5646e5

0516-016 ldeletepv: Cannot delete physical volume with allocated

Partitions. Use either migratepv to move the partitions or

Reducevg with the-d option to delete the partitions.

0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.

[root@aix216 /] # reducevg rootvg 0027713e1b5646e5-d

0516-306 getlvodm: Unable to find physical volume-d in the Device

Configuration Database.

0516-016 ldeletepv: Cannot delete physical volume with allocated

Partitions. Use either migratepv to move the partitions or

Reducevg with the-d option to delete the partitions.

0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.

0516-306 getlvodm: Unable to find physical volume-d in the Device

Configuration Database.

0516-884reducevg: Unable to remove physical volume-d.

[root@aix216 /] # reducevg-d rootvg 0027713e1b5646e5

0516-914 rmlv: Warning, all data belonging to logical volume

Hd5 on physical volume 0027713e1b5646e5 will be destroyed.

Rmlv: Do you wish to continue? Y (es) n (o)?

Y

0516-1246 rmlv: If hd5 is the boot logical volume, please run 'chpv-c'

As root user to clear the boot record and avoid a potential boot

Off an old boot p_w_picpath that may reside on the disk from which this

Logical volume is moved/removed.

Rmlv: Logical volume hd5 is removed.

0516-1734 rmlv: Warning, savebase failed. Please manually run 'savebase' before rebooting.

0516-914 rmlv: Warning, all data belonging to logical volume

Fwdump on physical volume 0027713e1b5646e5 will be destroyed.

Rmlv: Do you wish to continue? Y (es) n (o)?

Yes

0516-1008 rmlv: Logical volume fwdump must be closed. If the logical volume

Contains a filesystem, the umount command will close the LV device.

0516-1734 rmlv: Warning, savebase failed. Please manually run 'savebase' before rebooting.

0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.

0516-1734 reducevg: Warning, savebase failed. Please manually run 'savebase' before rebooting.

[root@aix216 /] # savebase

[root@aix216 /] #

[root@aix216 / tmp] # lsvg-l rootvg

Rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

Hd6 paging 32 32 1 open/syncd N/A

Hd8 jfs2log 1 1 1 open/syncd N/A

Hd4 jfs2 136 136 1 open/syncd /

Hd2 jfs2 24 24 1 open/syncd / usr

Hd9var jfs2 12 12 1 open/syncd / var

Hd3 jfs2 16 16 1 open/syncd / tmp

Hd1 jfs2 16 16 1 open/syncd / home

Hd10opt jfs2 24 24 1 open/syncd / opt

Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform

Fs001 jfs2 120 120 1 open/syncd / u01

Loglv00 jfslog 1 1 1 open/syncd N/A

Lv00 jfs 1 1 1 open/syncd / var/adm/csd

3. Re-establish hd5

Rebuild hd5, error

[root@aix216 / tmp] # mklv-y hd5-t boot-an e rootvg 1

Mklv-y hd5-t boot-an e rootvg 1

0516-304lquerypv: Unable to find device id 0000c908116d13250000000000000000 in the Device

Configuration Database.

0516-634 mklv: / tmp directory does not have enough space

Delete some files and try again.

0516-1734 mklv: Warning, savebase failed. Please manually run 'savebase' before rebooting.

[root@aix216 / tmp] # bosboot-ad / dev/ipldevice

0516-306 lslv: Unable to find hd5 in the Device

Configuration Database.

0301-168 bosboot: The current boot logical volume, / dev/hd5

Does not exist on / dev/hdisk0.

-Tip: / tmp space is insufficient, the actual / tmp space is very large, try to delete all files under the tmp

[root@aix216 / tmp] # df-m

Filesystem MB blocks Free Used Iused Iused Mounted on

/ dev/hd4 7168.00 6032.57 16% 4899 1

/ dev/hd2 3072.00 1207.03 61% 46571 15% / usr

/ dev/hd9var 1536.00 696.15 55% 1170 1% / var

/ dev/hd3 2048.00 2039.82 1% 25 1% / tmp

/ dev/fwdump 128.00 127.66 41% / var/adm/ras/platform

/ dev/hd1 2048.00 2003.45 3% 233 1% / home

/ proc-/ proc

/ dev/hd10opt 1024.00 749.93 27% 4799 3% / opt

/ dev/lv00 128.00 123.94 4% 18 1% / var/adm/csd

/ dev/fs001 15360.00 15357.19 1 7 1 per cent / U01

[root@aix216 / tmp] # rm-rf / tmp/*

4. Delete the hdisk0 under rootvg again

[root@aix216 / tmp] # lsvg-p rootvg

Rootvg:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

Hdisk0 active 546 162 108..00..00..12..42

0516-304: Unable to find device id 0000c908116d1325 in the Device

Configuration Database.

0000c908116d1325 missing 320 320 64..64..64..64..64

[root@aix216 / tmp] # reducevg-d rootvg "0000c908116d1325"

0516-304putlvodm: Unable to find device id 0000c908116d13250000000000000000 in the Device

Configuration Database.

0516-896 reducevg: Warning, cannot remove physical volume 0000c908116d1325 from

Device Configuration Database.

0516-1734 reducevg: Warning, savebase failed. Please manually run 'savebase' before rebooting.

[root@aix216 / tmp] # synclvodm-v rootvg

Synclvodm: Physical volume data updated.

Synclvodm: Logical volume hd6 updated.

Synclvodm: Logical volume hd8 updated.

Synclvodm: Logical volume hd4 updated.

Synclvodm: Logical volume hd2 updated.

Synclvodm: Logical volume hd9var updated.

Synclvodm: Logical volume hd3 updated.

Synclvodm: Logical volume hd1 updated.

Synclvodm: Logical volume hd10opt updated.

Synclvodm: Logical volume fwdump updated.

Synclvodm: Logical volume fs001 updated.

Synclvodm: Logical volume loglv00 updated.

Synclvodm: Logical volume lv00 updated.

[root@aix216 / tmp] # lsvg-l rootvg

Rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

Hd6 paging 32 32 1 open/syncd N/A

Hd8 jfs2log 1 1 1 open/syncd N/A

Hd4 jfs2 136 136 1 open/syncd /

Hd2 jfs2 24 24 1 open/syncd / usr

Hd9var jfs2 12 12 1 open/syncd / var

Hd3 jfs2 16 16 1 open/syncd / tmp

Hd1 jfs2 16 16 1 open/syncd / home

Hd10opt jfs2 24 24 1 open/syncd / opt

Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform

Fs001 jfs2 120 120 1 open/syncd / u01

Loglv00 jfslog 1 1 1 open/syncd N/A

Lv00 jfs 1 1 1 open/syncd / var/adm/csd

5. Re-establish hd5

Continue to rebuild hd5 and rebuild successfully

[root@aix216 / tmp] # mklv-y hd5-t boot-an e rootvg 1

Mklv-y hd5-t boot-an e rootvg 1

Hd5

[root@aix216 / tmp] # bosboot-ad / dev/ipldevice

Bosboot: Boot p_w_picpath is 37908 512 byte blocks.

[root@aix216 / tmp] #

6. Troubleshooting

View rootvg pv for problem resolution

[root@aix216 / tmp] # lsvg-p rootvg

Rootvg:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

Hdisk0 active 546 161 107..00..00..12..42

[root@aix216 / tmp] #

[root@aix216 / tmp] # lsvg-l rootvg

Rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

Hd5 boot 1 1 1 closed/syncd N/A

Hd6 paging 32 32 1 open/syncd N/A

Hd8 jfs2log 1 1 1 open/syncd N/A

Hd4 jfs2 136 136 1 open/syncd /

Hd2 jfs2 24 24 1 open/syncd / usr

Hd9var jfs2 12 12 1 open/syncd / var

Hd3 jfs2 16 16 1 open/syncd / tmp

Hd1 jfs2 16 16 1 open/syncd / home

Hd10opt jfs2 24 24 1 open/syncd / opt

Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform

Fs001 jfs2 120 120 1 open/syncd / u01

Loglv00 jfslog 1 1 1 open/syncd N/A

Lv00 jfs 1 1 1 open/syncd / var/adm/csd

[root@aix216 / tmp] #

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.