Wednesday, October 10, 2012

Tuning Weblogic - Case study of performance issue resolution

Case Study of how to analyze a slower performance or frequent out of memory exceptions of a web applications containing ejb deployed on oracle weblogic 

Enable jvm heap dump on app server

For e.g : export JAVA_OPTS=-verbose:gc -XX:+PrintClassHistogram -XX:+PrintGCDetails -Xloggc:/tmp/jvm_loggc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp

Using gcviewer monitor the heap usage and note the trend of increasing memory consumption even after the garbage collection is executed.


Take several snapshots/heap dumps and identify where the bottle neck is using tools like Eclipse MAT or visualvm
the heap dump can be taken manually using following command, can be analyzed using EclipseMAT or visualVM :
jmap -dump:format=b,file=  

Sort by object size in Dominator Tree or in histogram in EclipseMAT. Look for the top class/object and drill down to the classe name that look more familiar. If the identified class belongs to an ejb then there is a possibility that the ejb is retained in the in memory cache of jvm. To confirm this, check the cached beans current count in weblogic console. (refer to the how to monitor ejb cache section below for details)

One possible solution to above problem is described below :

Configure max-beans-in-cache and idle-timeout-seconds appropriate according to your need in weblogic-ejb-jar.xml as below use stateful/stateless as per your ejb:

<weblogic-enterprise-bean>
   <ejb-name>BeanName</ejb-name>
      <stateful-session-descriptor>
 <stateful-session-cache>
   <max-beans-in-cache>200</max-beans-in-cache>
   <idle-timeout-seconds>1200</idle-timeout-seconds>
 </stateful-session-cache>
      </stateful-session-descriptor>
</weblogic-enterprise-bean>

For testing purpose you can set the max-beans to 3-5 and idle-timeout to 60 secs and do some testing and notice the changes in weblogic console, like the ejb passivation count is increasing that means the ejbs are cleared from the cache once they are used.
Another important parameter is cache-type. Refer to the references for detailed explanations.

How to monitor the ejb cache on weblogic console :

1. Go to your deployed application
2. Expand it and go the the ejbs
3. Select the Monitoring tab.
4. Click and navigate through the pages to view monitoring statistics for deployed stateless, stateful, entity, and message-driven EJBs.
5. If you don't see the activation count or passivation count, click on customize table and add these columns.

References

1. http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/EJBTuning.html
2. http://middlewaremagic.com/weblogic/?p=5665