Weblogic Stuck Thread Monitor
前段时间Weblogic发生了Stuck Thread的情况，运维Dump出来的东西好像都没啥可以看的 接到个写监控程序的任务，发现Stuck Thread到一个指标的话，就开始进行ThreadDump OK，开撸
WLST 与 环境设置
本地安装Windows的Weblogic，安装完成再弄个测试的Domain
（看到下面的图让我想到了2012年刚毕业的那个暑假…
安装完成后，会有个叫Weblogic Scripting Tool的东西出现在开始菜单
点开，就是一堆的命令行，设置了PATH和CLASSPATH，关联Weblogic中的一些东西，关闭窗口后，这些设置就都回滚了，所以需要去环境变量里面再设置好PATH和CLASSPATH，方便后面直接使用weblogic.WLST启动程序
在Linux环境下的话，也需要设置一下Profile
这个脚本在Weblogic安装目录下的\server\bin，脚本为setWLSEnv.cmd或setWLSEnv.sh
Jython
启动WLST，说是“WebLogic Server Administration Scripting Shell”，实际上就是Jython的Shell啦~ Jython在较新的Windows系统下会认不出系统，解决方法在这 –> 点基我 Jython是我目前写得最别扭的代码，Python的语法，Java的方法和库 由于与Weblogic通信交互的方法全都在Weblogic的这个Shell里面，摸了好长时间cd、pwd跳来跳去，还需要在各种config和runtime中跳来跳去，摸清楚后，写了下面的代码，其中还包含了Coherence的监控
from java.io import FileInputStream
from javax.management import MBeanServerConnection
from javax.management import ObjectName
from javax.management.remote import JMXConnector
from javax.management.remote import JMXConnectorFactory
from javax.management.remote import JMXServiceURL
from java.util import Date
import java.lang
import os
propInputStream = FileInputStream("domains.properties")
configProps = Properties()
configProps.load(propInputStream)
\#\# Common Properties
adminUrl = configProps.get("server.url")
adminUser = configProps.get("admin.username")
adminPassword = configProps.get("admin.password")
checkInterval_in_Milliseconds = configProps.get("checkInterval_in_Milliseconds")
checkTimes_Number = configProps.get("checkTimes_Number")
\#\# Stuck Thread Properties
ExecuteThread_HoggerThread_Ratio = configProps.get("ExecuteThread_HoggerThread_Ratio")
threadDumpTimes_Number = configProps.get("threadDumpTimes_Number")
threadDumpInterval_in_Milliseconds = configProps.get("threadDumpInterval_in_Milliseconds")
\#\# Coherence Monitor Properties
jmxhost = configProps.get("coherence.jmx.host")
jmxport = configProps.get("coherence.jmx.port")
\#
i = 0
y = int(checkTimes_Number)
\#\# Check Hogger Threads Ratio
def alertHoggerThreads(executeTTC , hoggerTC, monitoringServerName):
\# print 'Execute Threads : ', executeTTC
\# print 'Hogger Thread Count : ', hoggerTC
\# print 'ExecuteThread_HoggerThread_Ratio :', ExecuteThread_HoggerThread_Ratio
if hoggerTC != 0:
ratio=(executeTTC/hoggerTC)
\# print 'Ratio : ' , ratio
\# print ''
if (int(ratio) <= int(ExecuteThread_HoggerThread_Ratio)):
\# print ' !!!! ALERT !!!! Stuck Threads are on its way.....'
\# print ''
message = 'ExecuteThreads Count= ' + str(executeTTC) + ' HoggingThreads= '+ str(hoggerTC) +' ExecuteThreads/HoggingThreads Ratio= '+ str(ratio)
genrateThreadDump(monitoringServerName)
\# else:
\# print '++++++++++++++++++++++++++++++++++++'
\# print monitoringServerName + ' is working fine till now'
\# print '++++++++++++++++++++++++++++++++++++'
\# else:
\# print '++++++++++++++++++++++++++++++++++++'
\# print monitoringServerName + ' is working fine till now'
\# print '++++++++++++++++++++++++++++++++++++'
\#\# Take Thread Dumps
def genrateThreadDump(monitoringServerName):
p = 0
q = int(threadDumpTimes_Number)
serverConfig()
cd ('Servers/'+ monitoringServerName)
while (p < q):
print 'Taking Thread Dump : ', p
threadDump()
cmd = "cat Thread_Dump_" + monitoringServerName + ".txt >> All_ThreadDump.txt"
os.system(cmd)
print 'Thread Dump Collected : ', p ,' now Sleeping for ', int(threadDumpInterval_in_Milliseconds) , ' Seconds ...'
print ''
Thread.sleep(int(checkInterval_in_Milliseconds))
p = p + 1
\#cmd = "rm -f All_ThreadDump.txt"
\#os.system(cmd)
serverRuntime()
\#\# Prepare Weblogic Monitor
connect(adminUser,adminPassword,adminUrl)
servers=cmo.getServers()
domainRuntime()
\#\# Prepare Coherence Monitor
url = JMXServiceURL("service:jmx:rmi:///jndi/rmi://"+str(jmxhost)+":"+str(jmxport)+"/jmxrmi")
jmxc = JMXConnectorFactory.connect(url)
mbsc = jmxc.getMBeanServerConnection()
ClusterMBean = ObjectName('Coherence:type=Cluster')
MemberIds = list(mbsc.getAttribute(ClusterMBean, 'MemberIds'))
ClusterSize = int(mbsc.getAttribute(ClusterMBean, 'ClusterSize'))
while (i < y or y == 0):
\#\# Weblogic
for monitorserver in servers:
monitoringServerName = monitorserver.getName()
cd('/ServerRuntimes/'+monitoringServerName)
serverName = cmo.getName()
cd('ThreadPoolRuntime/ThreadPoolRuntime')
executeTTC=cmo.getExecuteThreadTotalCount();
hoggerTC=cmo.getHoggingThreadCount();
alertHoggerThreads(executeTTC , hoggerTC, monitoringServerName)
\#\# Coherence
for mid in range(ClusterSize):
Services = ObjectName('Coherence:type=Service,name=DistributedCache,nodeId=' + str(MemberIds[mid]))
try:
\#\# MACHINE-SAFE means Coherence is OK, remove 'MACHINE-SAFE' can make it clear to see
if str(mbsc.getAttribute(Services, "StatusHA")) not in ['MACHINE-SAFE','n/a','']:
message = str(Date()) + "\#\#\#\#node[" + str(MemberIds[mid]) + "] = " + str(mbsc.getAttribute(Services, 'StatusHA'))
cmd = "echo " + message + " >> coherence_status.txt"
os.system(cmd)
except:
pass
\# print 'Sleeping for ', int(checkInterval_in_Milliseconds) , ' ...'
\# print ''
Thread.sleep(int(checkInterval_in_Milliseconds))
i = i + 1
脚本会遍历目标主机上Weblogic中的服务器，检查每台服务器中的线程数状态 当Stuck Thread的数量达到Execute Thread的一半（Ratio = 2）时，则会触发threadDump()，在执行脚本的目录下产生Dump日志 Coherence的监控，则是在Coherence中的Node发生异常，如非“MACHINE-SAFE”的状态，或诸如“ENDANGERED”这类的状态时，会立即写入日志 出现上面的情况，外围再触发消息通知，发送警报通知 脚本里面也可以集成mailx这类的工具，触发某条件后直接发送邮件通知
一些释义：
Active Execute Threads: 在活动的线程池内处理请求的线程个数
Execute Thread Total Count: 线程池内线程的总数
Execute Thread Idle Count：池内的空闲线程数，不包含Stuck和Standby的线程数，是指等待接受新请求到来并处理的线程个数
Queue Length：请求队列的长度，及队列中等待线程处理的请求的个数
Hogging Thread Count：线程处理一个请求时间超过一定值被视为hogging状态，如果继续处理请求超过一定时间将被视为Stuck，或处理完请求后被放回线程池
Standby Thread Count：统计在Standby（备用）线程池内的线程数。这些线程不需要处理当前请求被放入Standby池内，当活动的线程池内需要更多线程时，这些线程将被激活
Execute Thread Total Count = Active Execute Threads + Standby Thread Count
执行
将这两个文件放在同一目录，在该目录下执行
java weblogic.WLST weblogic_monitor.py
在目录下会生成Thread_Dump_ServerName.txt的Dump文件，还有Coherence监控发生异常时的coherence_status.txt文件