AIX LPAR missing hdisk after vios reboot

In case that you are doing routine checkup of your LPAR’s on IBM pSeries, you probably are checking status of your LPAR OS disks or volume group from time to time.
To check status of your volume grouphdisks use thisroot@aix-server> [/]  lsvg -p roottvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0                   missing                    546               4                 00..00..00..00..04
hdisk1                   active                       546               0                 00..00..00..00..00

As you can see one of hdisk is missing! And you start to panic! “OMG, hdisk is missing, where, how, when?!?!”

There is no place for panic. You will see that one of your disks is missing only after you have restarted one of your VIOS. In are case there is two VIOS. hdisk0 is from first VIOS, hdisk1 is from second VIOS. These two hdisk is creating volume group called rootvg.

How to fix this missing hdisk state?
All you need to do is to activate.

root@aix-server> [/] varyon rootvg

This will activate your volume group rootvg. After this you will see both of your hdisk as active!
Why this is important? Because of this:

When a volume group is activated, physical partitions are synchronized if they are not current.

But there is one case when you can’t make your hdisk active without making additional changes! In this case, after you execute varyon command, error will be prompted and you won’t be able to make your hdisk active!

root@aix-server> [/] varyon rootvg
varyonvg: Cannot varyon volume group with an active dump device on a missing physical volume. Use sysdumpdev to temporarily replace the dump device with /dev/sysdumpnull and try again.

So, as error said active dump device is on missing physical volume hdisk0.(I will not explaind here what system dump device is) How to change this? First we will list status of sysdump devices.

root@aix-server> [/]  sysdumpdev -l
primary              /dev/lg_dumplv
secondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     TRUE
always allow dump    FALSE
dump compression     ON

From here we can see, that primary device is located on /dev/lg_dumplv and secondary device is /dev/sysdumpnull. In error message, active dump device is actually primary dump device in sysdumpdev -l. So we need to change that.

root@aix-server> [/] sysdumpdev -p /dev/sysdupmnull

List again sysdump devices.

root@aix-server> [/]  sysdumpdev -l
primary             
/dev/sysdumpnullsecondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     TRUE
always allow dump    FALSE
dump compression     ON

Now execute activation of volume group. 

root@aix-server> [/] varyon rootvg

root@aix-server> [/] 
root@aix-server> [/]  lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0                         active            546                    4             00..00..00..00..04
hdisk1                         active            546                    0             00..00..00..00..00

As you can see now, both hdisk are active now.
Now, change back you primary dump device

root@aix-server> [/] sysdumpdev -p /dev/lg_dumplv

Visualize the Physical Layout of an AIX Volume Group

Here is a script I’ve written to visualize the physical layout of an AIX volume group.   The script visually shows the location of every Physical Partition (PP) on each hdisk (AKA Physical Volume).   The output shows which Logical Volume (LV) is on each of the PP’s (or if it is free space).   The output is color coded so each LV has its own color so that it is very easy to see where each LV physically is across the entire Volume Group.  You can specify the number of columns of output depending on the size of your screen.

The intended use of the script is to show a visual representation of the Volume Group to make using commands which move around LP’s/PP’s such as migratelp easier to use, to make LVM/disk maintenance easier, and also as a learning tool.

Here are a few screenshots:

image

image

 image

  When running the script you specify 2 parameters:  The volume group name, and the number of columns you would like displayed (or it will default to 3 columns if not specified).

Here is the script:

 #!/bin/ksh

#vvg - visualize physical layout of AIX volume group
#Copyright Brian Smith, 2013

index=0
set -A colors 41m 42m 43m 44m 45m 46m 47m 100m 101m 102m 103m 104m 105m 106m
tempfile=”/tmp/`basename $0`_$$”
tempfile2=”/tmp/`basename $0`_2$$”
> $tempfile
> $tempfile2

if [ -n “$1” ]; then
vg=$1
else
echo “Specify VG name as first parameter”
exit 1
fi

if ! lsvg $vg >/dev/null 2>&1; then
echo “Error: VG name not correct or VG not varried on”
exit 2
fi

[ -n “$2” ] && col=$2 || col=3
if ! echo $col | grep “^[0-9]*$” >/dev/null || [ “$col” -eq 0 ]; then
echo “Error: second parameter should be number of columns”
exit 3
fi

count=0
columns=””
while [ “$count” -lt “$col” ]; do
columns=”$columns -”
count=`expr $count + 1`
done

showdisk()
{
. $tempfile
. $tempfile2
[ “$index” -gt 0 ] && index=`expr $index + 1`
pv=$1
lspv -M $pv | while read line; do
if echo $line |  awk ‘NF==1 {print}’ | grep ‘-‘ >/dev/null; then
                    beg=`echo $line | awk -F: ‘{print $2}’ | awk -F ‘-‘ ‘{print $1}’`
                    end=`echo $line | awk -F: ‘{print $2}’ | awk -F ‘-‘ ‘{print $2}’`
                    while [ “$beg” -le “$end” ]; do
                            echo “${pv}:$beg Free”
                            beg=`expr $beg + 1`
                    done
elif echo $line | awk -F: ‘{print $2}’ | grep “^[0-9]*$” >/dev/null ; then
                    echo “$line Free”
else
                    echo “$line”
fi
done | while read line2; do
pp=`echo “$line2” | awk ‘{print $1}’ | awk -F: ‘{print $2}’`
lv=`echo “$line2” | awk ‘{print $2}’ | awk -F: ‘{print $1}’`
lp=`echo “$line2″ | awk ‘{print $2}’ | awk -F: ‘{print $2}’`

eval if ! [ -n \”\$${lv}\” ]\; then \
                    ${lv}=$index\;        \
                    echo ${lv}=$index \>\> $tempfile  \; \
                    index=`expr $index + 1`\;   \
                    [ \”\$index\” -gt \”13\” ] \&\& index=0 \; \
                    echo index=$index \> $tempfile2  \; \
fi
eval printf  \\\\033[\${colors[\$${lv}]}
if [ -n “$lp” ]; then
                    printf “%-7s %-15s %+7s\033[0m ” “PP$pp” “$lv” “LP$lp”
else
                    printf “%-7s %-23s\033[0m ” “PP$pp” “$lv”
fi
echo
done | paste -d ” ” $columns
}

for pv in `lspv | grep ” $vg ” | awk ‘{print $1}’`; do
ppsize=`lspv $pv | grep “^PP SIZE” | awk ‘{print $3 ” ” $4}’`
echo “\033[1;36m******************************* \033[0m”
printf “\033[1;36m* %-8s                    * \n\033[0m” $pv
printf “\033[1;36m* Size   : %-10s         * \n\033[0m” “`getconf DISK_SIZE /dev/$pv` MB”
printf “\033[1;36m* PP Size: %-19s* \n\033[0m” “$ppsize”
echo “\033[1;36m******************************* \033[0m”
showdisk $pv
done
rm $tempfile
rm $tempfile2

AIX Tips & tricks

1. To list machines configured in a NIM Server,
# lsnim -c machines

2. To list networks configured in a NIM Server,
# lsnim -c networks

3. To reset a machine (return to ready state)
# nim -Fo reset MachineName

4. To list core file settings for a user,
# lscore user1

The output will look like:
compression: on
path specification: default
corefile location: default
naming specification: off

5. To list the default settings for the system,

# lscore -d

The output will look like:
compression: off
path specification: on
corefile location: /corefiles
naming specification: off

6. To make any process run by root dump compressed core files and restore the location of the core files to the system default,

# chcore -c on -p default root
Note: If no default is specified, cores will dump in the current directory.

7. To enable a default core path for the system, type:

# chcore -p on -l /corefiles -d

8. To scan logical volume lv01, report the status of each partition, and have every block of each partition read to determine whether it is capableof performing I/O operations, type:

# mirscan -l lv01

9. To do the above operation in a PV,

# mirscan -p hdisk1

10. To do the above operation in a VG,

# mirscan -v vg01

11. To determine if the 64-bit kernel extension is loaded,

# genkex grep 64

12. To list all JFS file systems,

# lsjfs

13. To list all JFS2 file systems

# lsjfs2

14. To mirror a terminal1 on terminal2
a. Open terminal 1 and find the pts value (ps -ef grep pts)

b. Open terminal 2 and enter ‘portmir -t pts/1’
c. Now you will see commands and outputs from terminal 1 in terminal 2.
This is basically monitor a terminal.
d. Say “portmir -o” to end the mirroring after the use

15. To identify the current run level,

# cat /etc/.init.state

16. To list the available CD ROM drives,

# lsdev -Cc cdrom

17. To find out the speed of your network adapter,

# entstat -d ent0 grep “Media Speed”

18. To find out when your system was last installed/updated

# lslpp -f bos.rte

19. To list the status of your tape drive,

# tctl -f /dev/rmt0 status

20. How to setup anonymous ftp in AIX

Run the below script to setup anon ftp,
# /usr/lpp/tcpip/samples/anon.ftp

21. If telnet takes more time to produce a prompt, do the below checks

a. do nslookup of the client ip from the aix serverb.
b. Check the nameservers in /etc/resolv.confc.
c. Check the ‘hosts’ entry in /etc/netsvc.conf or NSORDER variable

This issue might be due to the DNS configuration issue. Pointing to a good nameserver should solve the problem.

22. How to shutdown the system to maintenance mode ?

# shutdown -Fm

23. How to log ftp accesses to a file

a. Add the below line in /etc/syslog.confdaemon.debug /tmp/daemon.log
b. # touch /tmp/daemon.log
c. # refresh syslogd
d. Modify your inetd.conf so that ftpd is called with the “-l” flag.

24. How to find a file name from inode number ?

# ncheck -i xxxx /mountpoint
where xxxx -> inode number of the file

25. How to redirect the system console to a file or tty temporarily

# swcons /tmp/console.out

or

# swcons /dev/tty5

26. How to recreate a deleted /dev/null file ?

# /bin/mknod /dev/null c 2 2

27. How to add commands that should get executed during every system shutdown ?

Add them to /etc/rc.shutdown

28. How to reduce the size or do cleanup of /var/adm/wtmp ?

# > /var/adm/wtmp

29. How to find out the fileset a file belongs to ?

# which_fileset command_name

30. In which file, the mapping of file Vs fileset stored ?

# /usr/lpp/bos/AIX_file_list

31. How to set maximum logins for a user in a system ?

Change the value of “maxlogins” under “usw” stanza in /etc/security/login.cfg

32. How to change the initial message that prints while logging in ?

Change the value of “herald” in /etc/security/login.cfg

33. How to set the # of seconds the user is given to enter their password ?

Change the value of “logintimeout” under “usw” stanza in /etc/security/login.cfg

AIX Shell Scripting

I explained the basics of shell scripting and why it’s so useful. However, it’s not only simple, but easy! Delving further, we’ll see how easy and powerful this tool can be. Additionally, we’ll explore building more advanced borne shell scripts as well as perl “one-liners” for certain situations.

As you may know, IBM releases security vulnerability information for AIX. The site usually gives a description of the vulnerability, fileset levels it effects and where to find the PTF/EFIX. So the script, if built right, will go out to all your servers and detect whether they’re vulnerable.

The Shell Script

The mockup script shown here assumes you have keys set up to SSH out to each server. It does not have to be run as root. The entire shell script, without comments, can be found here.

#!/usr/bin/env sh
# Checks for vulnerability in AIX RPC first issued May 8th.     

#############
# Variables #
#############
# These are all the levels vulnerable to this exploit.
_vulnerable_levels=(  "5.3.12.0" "5.3.12.1" "5.3.12.2" "5.3.12.3"
 "5.3.12.4" "5.3.12.5" "6.1.5.0" "6.1.5.1" "6.1.5.2"
 "6.1.5.3" "6.1.5.4" "6.1.5.5" "6.1.5.6" "6.1.5.7"
 "6.1.6.0" "6.1.6.1" "6.1.6.2" "6.1.6.3" "6.1.6.4"
 "6.1.6.5" "6.1.6.6" "6.1.6.7" "6.1.6.8" "6.1.6.9"
 "6.1.6.10" "6.1.6.11" "6.1.6.12" "6.1.6.13" "6.1.6.14"
 "6.1.6.15" "6.1.6.16"  "6.1.7.0" "6.1.7.1" "7.1.0.0"
  "7.1.0.1"  "7.1.0.2"  "7.1.0.3"  "7.1.0.4"  "7.1.0.5"
  "7.1.0.6"  "7.1.0.7"  "7.1.0.8"  "7.1.0.9"  "7.1.0.10"
  "7.1.0.11"  "7.1.0.12"  "7.1.0.13"  "7.1.0.14"  "7.1.0.15"
  "7.1.0.16" "7.1.0.17" "7.1.1.0" "7.1.1.1" )

This produces an array with the filesets affected by this vulnerability, which is important because we need to define all of the vulnerable filesets. You can find these values on the site where the vulnerability is defined.

_ssh_opts='-q -o BatchMode=yes -o ConnectTimeout=20
 -o ConnectionAttempts=1 -o ClearAllForwardings=yes'

This indicates the options we’re going to use to SSH to each servers. You can change these without affecting the script output.

# Makes text bold.
_b=`tput smso`

# Unset text bold.
_nb=`tput sgr0`

_date=`date +"%m%d%y_%H%M%S"`

_hosts="host1 host2 host3"

This variable specifies which servers you want to use. If you don’t set it, it won’t work.

for _host in ${_hosts}
do
        # Go out to the server and make sure it is AIX.
 If it's not, skip it.
        _uname=`ssh ${_ssh_opts} ${_host} "uname"`
        if [ "${_uname}" != "AIX" ]; then
                continue
        fi

        # Get the fileset level for bos.net.tcp.client
        _actual_host_level=`ssh ${_ssh_opts} ${_host} "lslpp -L bos.net.tcp.client | grep
 bos.net.tcp.client | awk '{ print \\$2 }'"`

The line above SSHs out to the server, ${_host}, and gets the value for the affected fileset—in this case bos.net.tcp.client.

        # Failure counter.
        _fail=0

        # Main loop.
        for _vulnerable_level in ${_vulnerable_levels[@]} #

(Notice the [@], which represents the entire array.)

        do
                # If the values from our array match actuals, then say so.
                if [ "${_actual_host_level}" = "${_vulnerable_level}" ]; then
                        printf "${_host} is vulnerable with bos.net.tcp.client of 
${_actual_host_level}.\n"     | tee -a rpc.scan.${_date}
                        _fail=`expr ${_fail} + 1`
                fi
        done

Here is the loop where we take the value of the fileset from our server and compare it to the filesets in our array.

        # If our failure counter hasn't gone off, then declare us not vulnerable.
        if [ ${_fail} -eq 0 ]; then
                printf "${_host} is not vulnerable.\n" | tee -a rpc.scan.${_date}
        fi
done

Perl One-Liners

Perl one-liners are extremely versatile tools. While probably not needed every day, they’re great to write down and keep handy, when needed. Having that perfect one-liner you need is often impressive to those around you. While, not critical, it’s kind of cool:

1. Convert Epoch (UNIX time; since January 1st 1970) to human readable time.

perl -e 'print scalar(localtime(EPOCHTIME)), "\n"'

Example:

[root@aix61-03]/root> perl -e 'print
 scalar(localtime(1342885345)), "\n"'
Sat Jul 21 11:42:25 2012

This one is very useful on AIX for determining the last time a user changed his or her password. To do so, look in the /etc/security/passwd file. It will look like this:

root:
        password = encryptedpassword
        lastupdate = 1311019884

So take the lastupdate epoch time number, and place it inside the perl parentheses to get the last time the password was changed.

2. Remove all blank lines from a file.

perl -ne 'print unless /^$/' FILE

Example:

This is our test file:

[root@gvicaix61-03]/root> cat test
This is a line

This is a space ^

This is two spaces ^

This is using our one-liner:

[root@aix61-03]/root> perl -ne 'print unless /^$/' test
This is a line
This is a space ^
This is two spaces ^

3. Insert line numbers on a file.

perl -ne 'print "$. $_"' FILE

Example:

[root@aix61-03]/root> perl -ne 'print "$. $_"' test
1 This is a line
2
3 This is a space ^
4
5
6 This is two spaces ^

As we moved beyond the basics of shell scripting, creating arrays, comparing values and understanding more advanced logic are important lessons to learn. The perl one-liners are an excellent resource to keep in your back pocket as well. In my next shell scripting article, we’ll dive into more advanced borne scripting, and introduce other languages well-suited for sysadmin tasks.