AIX system failure-- extended file system failure
AIX system failure-- extended file system failure
System environment:
Operating system: AIX5300-08
Oracle: Oracle 10gR2 (64)
Failure phenomenon:
When smit chfs extends "/", it is prompted that "device" cannot be found; after rootvg extends pv, pv is destroyed, resulting in ODM error
1. View rootvg
[root@aix216 /] # lsvg-p rootvg
Rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
Hdisk0 active 546 161 107..00..00..12..42
0516-304: Unable to find device id 0000c908116d1325 in the Device
Configuration Database.
0000c908116d1325 missing 320 320 64..64..64..64..64
[root@aix216 /] #
-try to solve the problem
[root@aix216 /] # redefinevg-d hdisk0 rootvg
[root@aix216 /] # lsvg-p rootvg
Rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
Hdisk0 active 546 161 107..00..00..12..42
0516-304: Unable to find device id 0000c908116d1325 in the Device
Configuration Database.
0000c908116d1325 missing 320 320 64..64..64..64..64
[root@aix216 /] # lspv
Hdisk0 0027713e1b5646e5 rootvg active
[root@aix216 /] # reducevg
0516-606 reducevg: Volume group name not entered.
Usage: reducevg [- d] [- f] VGname PVname...
Reduces volume group size by removing a physical volume.
2. Delete hdisk0 from rootvg
-reducevg after deleting pv, hd5 is deleted
[root@aix216 /] # reducevg rootvg 0027713e1b5646e5
0516-016 ldeletepv: Cannot delete physical volume with allocated
Partitions. Use either migratepv to move the partitions or
Reducevg with the-d option to delete the partitions.
0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.
[root@aix216 /] # reducevg rootvg 0027713e1b5646e5-d
0516-306 getlvodm: Unable to find physical volume-d in the Device
Configuration Database.
0516-016 ldeletepv: Cannot delete physical volume with allocated
Partitions. Use either migratepv to move the partitions or
Reducevg with the-d option to delete the partitions.
0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.
0516-306 getlvodm: Unable to find physical volume-d in the Device
Configuration Database.
0516-884reducevg: Unable to remove physical volume-d.
[root@aix216 /] # reducevg-d rootvg 0027713e1b5646e5
0516-914 rmlv: Warning, all data belonging to logical volume
Hd5 on physical volume 0027713e1b5646e5 will be destroyed.
Rmlv: Do you wish to continue? Y (es) n (o)?
Y
0516-1246 rmlv: If hd5 is the boot logical volume, please run 'chpv-c'
As root user to clear the boot record and avoid a potential boot
Off an old boot p_w_picpath that may reside on the disk from which this
Logical volume is moved/removed.
Rmlv: Logical volume hd5 is removed.
0516-1734 rmlv: Warning, savebase failed. Please manually run 'savebase' before rebooting.
0516-914 rmlv: Warning, all data belonging to logical volume
Fwdump on physical volume 0027713e1b5646e5 will be destroyed.
Rmlv: Do you wish to continue? Y (es) n (o)?
Yes
0516-1008 rmlv: Logical volume fwdump must be closed. If the logical volume
Contains a filesystem, the umount command will close the LV device.
0516-1734 rmlv: Warning, savebase failed. Please manually run 'savebase' before rebooting.
0516-884reducevg: Unable to remove physical volume 0027713e1b5646e5.
0516-1734 reducevg: Warning, savebase failed. Please manually run 'savebase' before rebooting.
[root@aix216 /] # savebase
[root@aix216 /] #
[root@aix216 / tmp] # lsvg-l rootvg
Rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
Hd6 paging 32 32 1 open/syncd N/A
Hd8 jfs2log 1 1 1 open/syncd N/A
Hd4 jfs2 136 136 1 open/syncd /
Hd2 jfs2 24 24 1 open/syncd / usr
Hd9var jfs2 12 12 1 open/syncd / var
Hd3 jfs2 16 16 1 open/syncd / tmp
Hd1 jfs2 16 16 1 open/syncd / home
Hd10opt jfs2 24 24 1 open/syncd / opt
Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform
Fs001 jfs2 120 120 1 open/syncd / u01
Loglv00 jfslog 1 1 1 open/syncd N/A
Lv00 jfs 1 1 1 open/syncd / var/adm/csd
3. Re-establish hd5
Rebuild hd5, error
[root@aix216 / tmp] # mklv-y hd5-t boot-an e rootvg 1
Mklv-y hd5-t boot-an e rootvg 1
0516-304lquerypv: Unable to find device id 0000c908116d13250000000000000000 in the Device
Configuration Database.
0516-634 mklv: / tmp directory does not have enough space
Delete some files and try again.
0516-1734 mklv: Warning, savebase failed. Please manually run 'savebase' before rebooting.
[root@aix216 / tmp] # bosboot-ad / dev/ipldevice
0516-306 lslv: Unable to find hd5 in the Device
Configuration Database.
0301-168 bosboot: The current boot logical volume, / dev/hd5
Does not exist on / dev/hdisk0.
-Tip: / tmp space is insufficient, the actual / tmp space is very large, try to delete all files under the tmp
[root@aix216 / tmp] # df-m
Filesystem MB blocks Free Used Iused Iused Mounted on
/ dev/hd4 7168.00 6032.57 16% 4899 1
/ dev/hd2 3072.00 1207.03 61% 46571 15% / usr
/ dev/hd9var 1536.00 696.15 55% 1170 1% / var
/ dev/hd3 2048.00 2039.82 1% 25 1% / tmp
/ dev/fwdump 128.00 127.66 41% / var/adm/ras/platform
/ dev/hd1 2048.00 2003.45 3% 233 1% / home
/ proc-/ proc
/ dev/hd10opt 1024.00 749.93 27% 4799 3% / opt
/ dev/lv00 128.00 123.94 4% 18 1% / var/adm/csd
/ dev/fs001 15360.00 15357.19 1 7 1 per cent / U01
[root@aix216 / tmp] # rm-rf / tmp/*
4. Delete the hdisk0 under rootvg again
[root@aix216 / tmp] # lsvg-p rootvg
Rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
Hdisk0 active 546 162 108..00..00..12..42
0516-304: Unable to find device id 0000c908116d1325 in the Device
Configuration Database.
0000c908116d1325 missing 320 320 64..64..64..64..64
[root@aix216 / tmp] # reducevg-d rootvg "0000c908116d1325"
0516-304putlvodm: Unable to find device id 0000c908116d13250000000000000000 in the Device
Configuration Database.
0516-896 reducevg: Warning, cannot remove physical volume 0000c908116d1325 from
Device Configuration Database.
0516-1734 reducevg: Warning, savebase failed. Please manually run 'savebase' before rebooting.
[root@aix216 / tmp] # synclvodm-v rootvg
Synclvodm: Physical volume data updated.
Synclvodm: Logical volume hd6 updated.
Synclvodm: Logical volume hd8 updated.
Synclvodm: Logical volume hd4 updated.
Synclvodm: Logical volume hd2 updated.
Synclvodm: Logical volume hd9var updated.
Synclvodm: Logical volume hd3 updated.
Synclvodm: Logical volume hd1 updated.
Synclvodm: Logical volume hd10opt updated.
Synclvodm: Logical volume fwdump updated.
Synclvodm: Logical volume fs001 updated.
Synclvodm: Logical volume loglv00 updated.
Synclvodm: Logical volume lv00 updated.
[root@aix216 / tmp] # lsvg-l rootvg
Rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
Hd6 paging 32 32 1 open/syncd N/A
Hd8 jfs2log 1 1 1 open/syncd N/A
Hd4 jfs2 136 136 1 open/syncd /
Hd2 jfs2 24 24 1 open/syncd / usr
Hd9var jfs2 12 12 1 open/syncd / var
Hd3 jfs2 16 16 1 open/syncd / tmp
Hd1 jfs2 16 16 1 open/syncd / home
Hd10opt jfs2 24 24 1 open/syncd / opt
Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform
Fs001 jfs2 120 120 1 open/syncd / u01
Loglv00 jfslog 1 1 1 open/syncd N/A
Lv00 jfs 1 1 1 open/syncd / var/adm/csd
5. Re-establish hd5
Continue to rebuild hd5 and rebuild successfully
[root@aix216 / tmp] # mklv-y hd5-t boot-an e rootvg 1
Mklv-y hd5-t boot-an e rootvg 1
Hd5
[root@aix216 / tmp] # bosboot-ad / dev/ipldevice
Bosboot: Boot p_w_picpath is 37908 512 byte blocks.
[root@aix216 / tmp] #
6. Troubleshooting
View rootvg pv for problem resolution
[root@aix216 / tmp] # lsvg-p rootvg
Rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
Hdisk0 active 546 161 107..00..00..12..42
[root@aix216 / tmp] #
[root@aix216 / tmp] # lsvg-l rootvg
Rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
Hd5 boot 1 1 1 closed/syncd N/A
Hd6 paging 32 32 1 open/syncd N/A
Hd8 jfs2log 1 1 1 open/syncd N/A
Hd4 jfs2 136 136 1 open/syncd /
Hd2 jfs2 24 24 1 open/syncd / usr
Hd9var jfs2 12 12 1 open/syncd / var
Hd3 jfs2 16 16 1 open/syncd / tmp
Hd1 jfs2 16 16 1 open/syncd / home
Hd10opt jfs2 24 24 1 open/syncd / opt
Fwdump jfs2 1 1 1 open/syncd / var/adm/ras/platform
Fs001 jfs2 120 120 1 open/syncd / u01
Loglv00 jfslog 1 1 1 open/syncd N/A
Lv00 jfs 1 1 1 open/syncd / var/adm/csd
[root@aix216 / tmp] #
