Script to monitor CPU, Memory, Load threshold and LDAP Status on Solaris and send notification E-Mail

#! /bin/ksh
vmstat 5 5 > /tmp/vmstat.temp
freemem=`sed -n '5p' /tmp/vmstat.temp | awk '{print $5}'`
freecpu=`sed -n '5p' /tmp/vmstat.temp | awk '{print $22}'`
load=`uptime | awk '{print $10}'| cut -c1-4`
ldapstatus=`zlogin lpr-srv-prd-02 /usr/lib/ldap/ldap_cachemgr -g |  grep DOWN| wc -l`
date >> /var/tmp/performacestats ; uptime >> /var/tmp/performacestats; prstat -Z 1 1 >> /var/tmp/performacestats; top -b 1 1 >> /var/tmp/performacestats; /usr/lib/ldap/ldap_cachemgr -g >> /var/tmp/performacestats; vmstat 5 5 >> /var/tmp/performacestats ; echo "===================================" >> /var/tmp/performacestats

#Alert on less than 40% free CPU
thresholcpu="40"
#Memory in KB (Alert on less than 25gb free)
thresholdmem="26214400"
#Alert on higher than 40% LOAD
thresholdload="20.00"

if [ "$freemem" -lt "$thresholdmem"  ] || [ "$freecpu" -lt "$thresholdcpu" ] || [ "$load" -gt "$thresholdload" ] || [ "$ldapstatus" -gt 0 ]
then
echo "Check performance on <Server Name>. Log file is /var/tmp/performacestats" | mailx -s "ALERT: Performace Test" <E-Mail>
fi

script to check Veritas dgname, available space in each dg and zone name of all vxfs mount points in Sun Servers

This script can be run from a centralized server on all the servers and will list the DG name, zone name and space available in each DG correspondig to all Veritas Mount Points available in server.


#!/usr/bin/ksh
#########################################################################################
#       Description:    Check space available to extend mount point and its             #
#                       corresponding DG name                                           #
#                       This Script is designed to work on Solaris 10 Global Zones      #
#                       Solaris 8 and Solaris 9 Only                                    #
#       Author:         Yogesh Aggarwal                                   #
#       Created:        2013-Aug-6                                                      #
#       Last Modified:  2013-Aug-6                                                      #
#       Last Mod by:    Yogesh Aggarwal                                                 #
#       Version:        1.1                                                             #
#                                                                                       #
#########################################################################################


PATH=/sbin:/usr/sbin:/usr/bin:/opt/VRTS/bin:/usr/local/bin ; export PATH

zn=`/sbin/zonename 2>/dev/null`
if [[ $zn == "global" ]] || [[ `uname -r` == 5.9 ]] || [[ `uname -r` == 5.8 ]]
then
{

rm -rf /var/tmp/DGSPACECHECK
mkdir -p /var/tmp/DGSPACECHECK

cat /etc/mnttab | grep vxfs > /var/tmp/DGSPACECHECK/vxfsfslist
cat /var/tmp/DGSPACECHECK/vxfsfslist | awk '{print $1}' > /var/tmp/DGSPACECHECK/vxfsdevicelist
cat /var/tmp/DGSPACECHECK/vxfsdevicelist | cut -d / -f5 > /var/tmp/DGSPACECHECK/vxfsdglist
for i in `cat /var/tmp/DGSPACECHECK/vxfsdglist`
do
size=`vxassist -g $i maxsize  2>/dev/null`
if [ $? -eq 0 ]
then echo "$size" | awk '{print $5}'>> /var/tmp/DGSPACECHECK/vxfsdgspacelist
else echo "No Space Left in DG" >> /var/tmp/DGSPACECHECK/vxfsdgspacelist
fi
done
cat /var/tmp/DGSPACECHECK/vxfsfslist | awk '{print $2}' > /var/tmp/DGSPACECHECK/vxfsmountlist

dgs=$(wc -l /var/tmp/DGSPACECHECK/vxfsdglist )
mounts=$(wc -l /var/tmp/DGSPACECHECK/vxfsmountlist)

for i in `cat /var/tmp/DGSPACECHECK/vxfsmountlist`; do echo $i| grep root > /dev/null
if [ $? -eq 0 ]
then echo $i |sed -e 's/root/:/g' | awk -F':' '{print $1}' | sed -e 's/\//\:/g' | awk -F':' '{print $3}'>> /var/tmp/DGSPACECHECK/zonename
echo $i |sed -e 's/root/:/g' | awk -F':' '{print $2}' >>  /var/tmp/DGSPACECHECK/mpname
else echo $i |sed -e 's/root/:/g' | awk -F':' '{print $1}' >> /var/tmp/DGSPACECHECK/mpname
echo "NA" >> /var/tmp/DGSPACECHECK/zonename
fi
done

###Final to get requried output in required format.

a=1 b=1 c=1 d=1
while [ $a -le $dgs -a $b -le $mounts ]; do
zone=`sed -n -e "${a}p" /var/tmp/DGSPACECHECK/zonename`
mount=`sed -n -e "${b}p" /var/tmp/DGSPACECHECK/mpname`
dg=`sed -n -e "${c}p" /var/tmp/DGSPACECHECK/vxfsdglist`
space=`sed -n -e "${d}p" /var/tmp/DGSPACECHECK/vxfsdgspacelist`
echo "`uname -n` \t $zone \t $mount \t $dg \t $space " >> /var/tmp/DGSPACECHECK/final
a=$(( a + 1 )); b=$(( b + 1)); c=$(( c + 1)); d=$(( d + 1));
done

### Print the final Output to screen.

}
else echo " This is neither a Global Zone, nor a Soalris 8 or 9 Server"
fi
cat /var/tmp/DGSPACECHECK/final

lpsched dying automatically again and again in Solaris

In case lpsched is dying again and again in Solaris, please check the logs in /var/spoo/lp/logs using below command.
# tail lpsched
received unexpected signal 11: terminating

If lpsched logs shows the above mentioned logs, please follow the below procedure to fix the issue:

## lpshut
Print services stopped.
## cd /var/spool/lp
## ps -ef | grep "lp -c" | awk '{print $2}' > /tmp/testy
## for i in `cat /tmp/testy`
> do
> kill -9 $i
> done
bash: kill: (21483) - No such process
##
## cd /var/spool/lp/tmp
## mv -r localhost/ /var/tmp/localhost.lp.tmp
## cd /var/spool/lp/requests
## mv -r localhost/ /var/tmp/localhost.lp.request 
## /usr/lib/lp/local/lpsched
Print services started.
##
## svcadm enable -r /application/print/server
##
## svcs -a | grep -i print
online 2012 svc:/application/print/ppd-cache-update:default
online 4:03:42 svc:/application/cde-printinfo:default
online 11:30:55 svc:/application/print/ipp-listener:default
online 11:30:55 svc:/application/print/server:default
online 11:30:55 svc:/application/print/rfc1179:default
##

## lpstat -lp lpd_chatpa-prt22
printer lpd_chatpa-prt22 now printing lpd_chatpa-prt22-15. enabled since Thu Apr 25 16:59:32 2013. available.
Form mounted:
Content types: application/octet-stream
Description:
Printer types: unknown
Connection: direct
Interface: /usr/lib/lp/model/netstandard
On fault: write root
After fault: continue
Users allowed:
(all)
Forms allowed:
(none)
Media supported:
(none)
Banner page never printed
Character sets:
(none)
Default pitch:
Default page size:
Default port setting:
Options: dest=3.8.90.37:9100, protocol=tcp
##
##
 

Script to test IPMP in Solaris Servers

Below script is to test the IPMP running status as well as configuration status on a Solaris Server. It will test below failure conditions
1. If each the NIC is under IPMP or not.
2. Running config and File config is same.
3. NIC Flag is running or not
4. Minimum two NICs are there in IPMP.
5. It will list the output in below columns:

Server Name > Zone Name > NIC NAME > IP Address > Netmask > Group >  Ether >  Success/Failure

#!/usr/bin/ksh
#########################################################################################
#       Description:    IPMP Auditing Script                                            #
#                       This Script is designed to work on Solaris 10 Global Zones      #
#                       Solaris 8 and Solaris 9 Only                                    #
#       Author:         Yogesh Aggarwal                                      #
#       For:            Solaris Admins                                          #
#       Created:        2012-June-27                                                    #
#       Last Modified:  23 July 2012                                                    #
#       Last Mod by:    Yogesh Aggarwal                                                 #
#       Version:        1.3                                                             #
#                                                                                       #
#########################################################################################


PATH=/sbin:/usr/sbin:/usr/bin:/opt/VRTS/bin:/usr/local/bin ; export PATH

zn=`/sbin/zonename 2>/dev/null`
if [[ $zn == "global" ]] || [[ `uname -r` == 5.9 ]] || [[ `uname -r` == 5.8 ]]
then
{

rm -rf /var/tmp/IPMPTEST
mkdir -p /var/tmp/IPMPTEST
/sbin/ifconfig -a | grep -v inet | grep -v zone | grep -v groupname | grep -v ether | awk '{print $1}' | grep -v lo0 > /var/tmp/IPMPTEST/tmp_nic
sed -e 's/:$//' /var/tmp/IPMPTEST/tmp_nic > /var/tmp/IPMPTEST/ifnics
cut -f1 -d : /var/tmp/IPMPTEST/ifnics > /var/tmp/IPMPTEST/nics

### To find out IPMP Group Information

for i in `cat /var/tmp/IPMPTEST/nics`
do
grp=`/sbin/ifconfig $i | grep groupname | awk '{print $2}'` ; count=`/sbin/ifconfig $i | grep groupname | awk '{print $2}' | wc -l`
if [ $count  -ne  0 ]
then echo "$grp" >> /var/tmp/IPMPTEST/ipmp_group
else echo "Null" >> /var/tmp/IPMPTEST/ipmp_group
fi
done

### To find out MAC Information

for i in `cat /var/tmp/IPMPTEST/nics`
do
ether=`/sbin/ifconfig $i | grep ether | awk '{print $2}'` ; count=`/sbin/ifconfig $i | grep ether | awk '{print $2}' | wc -l`
if [ $count  -ne  0 ]
then echo "$ether" >> /var/tmp/IPMPTEST/ether
else echo "Null" >> /var/tmp/IPMPTEST/ether
fi
done

for i in `cat /var/tmp/IPMPTEST/ifnics`
do
grptemp=`/sbin/ifconfig $i | grep groupname | awk '{print $2}'` ; count2=`/sbin/ifconfig $i | grep groupname | awk '{print $2}' | wc -l`
if [ $count2  -ne  0 ]
then echo "$grptemp" >> /var/tmp/IPMPTEST/ipmp_group_temp
else echo "Null" >> /var/tmp/IPMPTEST/ipmp_group_temp
fi
done

### To find out Zones Information

for ifnics in `cat /var/tmp/IPMPTEST/ifnics`
do
zone=`/sbin/ifconfig $ifnics | grep zone | awk '{print $2}'` count1=`/sbin/ifconfig $ifnics | grep zone | awk '{print $2}'| wc -l`
if [ $count1  -ne  0 ]
then echo "$zone" >> /var/tmp/IPMPTEST/zone
else echo "Null" >> /var/tmp/IPMPTEST/zone
fi
done

### To find out if NIC Flag is Running and IPMP Config file has same group name as Running Configuration.

groups=$(wc -l /var/tmp/IPMPTEST/ipmp_group )
NICS=$(wc -l /var/tmp/IPMPTEST/nics  )
k=1 m=1
while [ $k -le $groups -a $m -le $NICS ]; do
group=`sed -n -e "${k}p" /var/tmp/IPMPTEST/ipmp_group`
nic=`sed -n -e "${m}p" /var/tmp/IPMPTEST/nics`
count=`grep $group /etc/hostname.$nic 2>/dev/null | wc -l`
count1=`/sbin/ifconfig $nic | grep RUNNING | wc -l`
count3=`grep -w $group /var/tmp/IPMPTEST/ipmp_group_temp | grep -v Null | wc -l`
if [ $count1 -ge 1 ] && [ $count -ge 1 ] && [ $count3 -eq 2 ]
then
echo "Success" >> /var/tmp/IPMPTEST/check1
else
echo "Failure" >> /var/tmp/IPMPTEST/check1
fi
k=$(( k + 1 )); m=$(( m + 1));
done

### To find out IP Addresses against each NIC.

for ifnics in `cat /var/tmp/IPMPTEST/ifnics`
do
echo "$ifnics \t `/sbin/ifconfig $ifnics | grep inet | awk '{print $2}'` \t `/sbin/ifconfig $ifnics | grep inet | awk '{print $4}'`" >> /var/tmp/IPMPTEST/first
done

###Final to get requried output in required format.

a=1 b=1 c=1 d=1 e=1
while [ $a -le $groups -a $b -le $NICS ]; do
group1=`sed -n -e "${a}p" /var/tmp/IPMPTEST/ipmp_group`
check1=`sed -n -e "${b}p" /var/tmp/IPMPTEST/check1`
first=`sed -n -e "${c}p" /var/tmp/IPMPTEST/first`
zone=`sed -n -e "${d}p" /var/tmp/IPMPTEST/zone`
ether=`sed -n -e "${e}p" /var/tmp/IPMPTEST/ether`
echo "`uname -n` \t $zone \t $first \t $group1 \t $ether \t $check1 " >> /var/tmp/IPMPTEST/final
a=$(( a + 1 )); b=$(( b + 1)); c=$(( c + 1)); d=$(( d + 1)); e=$(( e + 1));
done

### Print the final Output to screen.

}
else echo " This is neither a Global Zone, nor a Soalris 8 or 9 Server"
fi
cat /var/tmp/IPMPTEST/final

Server not booting after luactivate to Patch ID Generic_147440-02 and above on T2000

Problem:

Rebooting with command: boot
Boot device: disk1:a  File and args:
SunOS Release 5.10 Version Generic_147440-10 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
sorry, variable 'tcp_conn_hash_size' is not defined in the 'tcp' module
Hostname: sq......104
NOTICE: VxVM vxdmp V-5-0-34 added disk array 604033, datype = EMC
NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk
NOTICE: VxVM vxdmp V-5-3-1700 dmpnode 287/0x0 has migrated from enclosure FAKE_ENCLR_SNO to enclosure DISKS
Cross trap sync timeout:  at cpu_sync.xword[1]: 0x1010panic: failed to stop cpu12
panic: failed to stop cpu13
panic: failed to stop cpu14
panic: failed to stop cpu15
panic[cpu20]/thread=2a1020cdca0: xt_sync: timeout
000002a1020cce90 unix:xt_sync+370 (10bcc00, 1, 2a1020ccfa8, 626475a3f8, 1, 1913c08)
  %l0-3: 0000000000000008 00000062647575c4 00000062abfc3028 00000062abfc2ff8
  %l4-7: 000002a1020ccfb0 0000000000000020 0000000000000000 00000000010bcd28
000002a1020cd1b0 unix:hat_unload_callback+7d4 (7fc00, 2a1020cd348, 0, 2a1020cd448, 0, 30002c77b40)
  %l0-3: 00000300090fe000 fffffffffffffff8 0000000000000001 0000000000000001
  %l4-7: 0000000000000000 000003000907b010 ffffffffffffffff 0000030002c77b48
000002a1020cd590 swrand:physmem_ent_gen+210 (1a25348, 700000ccd80, 0, 0, 0, 1000)
  %l0-3: 0000000000001d9b 0000000000000000 000002a1020cd68c 0000000000001fff
  %l4-7: 0000000000002000 0000000000001000 0000000003b36000 000000000000000d
000002a1020cd6f0 swrand:rnd_handler+14 (0, 2a1020cdca0, 512c90a521, 60035591bf2, 1a25338, 1a25000)
  %l0-3: 00000600371d1b80 8000000000000000 0000000000000001 0000000000080000
  %l4-7: 0000060035591be0 0000000000010000 00000000fffeffff 000000007bfedea4
000002a1020cd7a0 genunix:callout_list_expire+5c (60033bb0940, 60035a4be80, 80000000, 0, bfffffffffffffff, 4000000000000000)
  %l0-3: 00000600371d1b80 8000000000000000 0000000000000001 0000000000080000
  %l4-7: 0000060035591be0 0000000000010000 00000000fffeffff 000000007bfedea4
000002a1020cd850 genunix:callout_expire+1c (60033bb0940, 60033bb09c0, 512c90a521, 60035591bf2, 0, 60035591bf4)
  %l0-3: 0000060035a4be80 0000060035591bf0 0000000000000001 0000000000080000
  %l4-7: 0000060035591be0 0000000000010000 00000000fffeffff 0000060035591be8
000002a1020cd900 genunix:callout_execute+c (60033bb0940, 60034a84708, 10e5138, 0, 10d5400, 0)
  %l0-3: 0000060034a84708 0000060035591bf0 0000000000000001 0000000000080000
  %l4-7: 0000060035591be0 0000000000010000 00000000fffeffff 0000060035591be8
000002a1020cd9b0 genunix:taskq_thread+3b8 (60035591c28, 60035591bc0, 512c90a521, 60035591bf2, 51fe5f7609, 60035591bf4)
  %l0-3: 0000060034a84708 0000060035591bf0 0000000000000001 0000000000080000
  %l4-7: 0000060035591be0 0000000000010000 00000000fffeffff 0000060035591be8
syncing file systems... done
skipping system dump - no dump device configured
rebooting...
SC Alert: Host System has Reset
SC Alert: Indicator SYS/ACT is now SLOW BLINK
SC Alert: Indicator SYS/ACT is now STANDBY BLINK
SC Alert: Host system has shut down.
SC Alert: Indicator SYS/ACT is now SLOW BLINK
|
SC Alert: Indicator SYS/ACT is now ON

Sun Fire T200, No Keyboard
Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.25.0, 32760 MB memory available, Serial #75381276.
Ethernet address 0:14:4f:7e:3a:1c, Host ID: 847e3a1c.

Rebooting with command: boot

Solution :

Hi Friends,

I am back on Blogging after a gap of couple of years. If you are trying to upgrade a T2000 running Solaris 10 from u8 to u9 and install patch 147440-02 or above and your current firmware level is ver old, your server will never boot from the new BE. This is a Solaris Bug and the only solution to resolve it is to upgrade the Firware to  139434-09

Here in my server below was the current firmware version:

sc> showsc version -v
Advanced Lights Out Manager CMT v1.3.2
SC Firmware version: CMT 1.3.2
SC Bootmon version: CMT 1.3.2
VBSC 1.3.1
VBSC firmware built Feb  5 2007, 21:11:38
SC Bootmon Build Release: 01
SC bootmon checksum: F44BD111
SC Bootmon built Feb  5 2007, 21:18:50
SC Build Release: 01
SC firmware checksum: 48C51947
SC firmware built Feb  5 2007, 21:19:05
SC firmware flashupdate THU MAY 10 19:00:22 2007
SC System Memory Size: 32 MB
SC NVRAM Version = 12
SC hardware type: 4
FPGA Version: 4.2.4.7


I upgraded it to below:

sc> showsc version -v

Advanced Lights Out Manager CMT v1.7.11

SC Firmware version: CMT 1.7.11
SC Bootmon version: CMT 1.7.11

VBSC 1.7.3.d

VBSC firmware built Jul  6 2011, 19:27:17

SC Bootmon Build Release: 01

SC bootmon checksum: 4CB78FC8
SC Bootmon built Jul  6 2011, 19:37:05

SC Build Release: 01

SC firmware checksum: C41F3325

SC firmware built Jul  6 2011, 19:37:18

SC firmware flashupdate THU AUG 30 09:42:18 2012

SC System Memory Size: 32 MB

SC NVRAM Version = 14
SC hardware type: 4
FPGA Version: 4.2.4.7

After Firmware upgrade when i tried booted from the new BE. It worked absoluely fine.

Good Luck.

Yogesh

 

VxVM vxassist ERROR V-5-1-436 Cannot allocate space to grow volume to 1958928768 blocks

Hi All,

# vxdg -g oradgh free
DISK DEVICE TAG OFFSET LENGTH FLAGS
emcpower67 emcpower67s2 emcpower67 0 230395648 -

# df -h /fh01 /bfh01
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/oradgh/fh01vol 824G 771G 50G 94% /fh01
/dev/vx/dsk/shadowdg/fh01vol 824G 228G 560G 29% /bfh01root on cidcshlspora01:

# /etc/vx/bin/vxresize -g oradgh fh01vol +230395648 emcpower67 &
9080
root on cidcshlspora01:
# VxVM vxassist ERROR V-5-1-436 Cannot allocate space to grow volume to 1958928768 blocks
VxVM vxresize ERROR V-5-1-4703 Problem running vxassist command for volume fh01vol, in diskgroup oradgh
vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
root on cidcshlspora01:

In this kind of case it may happen the block size selected for the FS while its creation is different, so you can grow the FS by taking the size from( vxassist -g dgname maxsize diskname) command as mentioned below:

# vxassist -g oradgh maxsize emcpower67
Maximum volume size: 230391808 (112496Mb)

# /etc/vx/bin/vxresize -g oradgh fh01vol +230391808 emcpower67
root on cidcshlspora01:
# df -h /fh01
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/oradgh/fh01vol 934G 771G 153G 84% /fh01
root on cidcshlspora01:

So the FS grown as above.. Please let meknow if any doubts..

Thanks,
Yogesh

metastat: host: data1: must be owner of the set for this command

Hi All,

For this error, Please run teh below command. don't worry if it is not a cluster set as well. But if it is a cluster set then confirm that it is not imported on other server.

#metaset -s data1 -C take
It would import the diskset and problem would be resolved.

Thanks,
Yogesh