performance / tuning tips. to the point.                
About Us | Site Map | Privacy
Disclaimer | Feedback
About RSS Feed | Useful Links
Search Partner Links
Original Blog
Add to My Yahoo!
Google Reader or Homepage
del.icio.us performancewiki.com Latest Items


© 2005-2012 PerformanceWiki.com
All Rights Reserved.


Memory Dump Diagnostic for Java



If you think Java application never leaks memory, just run the following java code for 5 minutes and see:

public class MyClass {
  static HashSet myContainer = new HashSet();
  public void leak(int numObjects) {
    for (int i = 0; i < numObjects; ++i) {
      String leakingUnit = new String("this is leaking object: " + i);
      myContainer.add(leakingUnit);
    }
  }
  public static void main(String[] args) throws Exception {
    System.out.print("May I start leaking? (hit enter)");  
    System.in.read(new byte[4]);
    {
      MyClass myObj = new MyClass();
      myObj.leak(1000000000);
    }
    System.out.println("oops!!! I have leaked " +
                        myContainer.size() + " objects!!!!!");
    System.out.print("hit enter to exit"); System.in.read();
  }
}

The above example shows how it is possible to unintentionally hold references to objects (Strings) causing these objects to live beyond the lifecycle of the creating object (MyClass), and preventing them from getting garbage collected.

The only effective way to debug a memory leak problem in java is to diagnose the heap dumps taken over the time interval when the Java application is experience high garbage collecting activities (usually reflected in 100% CPU usage on a cpu), or the application appears to hang.

Steps for getting heap jumps for WebSphere Application Server or non-WebSphere JVMs.

WebSphere Application Server V4.0, on an AIX system

   1. In the administrative console, navigate to and select the default application server.  
      In the file tree, click WebSphere Administrative Domain > Nodes > localhost > 
      Application Servers > Default Server.
   2. On the JVM Settings tab, click Advanced JVM Settings and enable verbose garbage collection.
   3. On the General tab, click Environment, and set both 
      IBM_HEAPDUMP = true and IBM_HEAPDUMP_OUTOFMEMORY = true.
   4. To direct the destination of the heap dump to an alternate directory, 
      click Environment, and set IBM_HEAPDUMPDIR to the appropriate directory 
      (that is, \heapdumps).By default, the memory dumps are created in 
      the \Websphere4\AppServer\bin\ directory. 
   5. Go to the /usr/WebSphere40/AppServer/bin/ directory and issue the command: 
      ulimit -f unlimited
      so file size problems do not occur when writing the memory dumps.
   6. Take a heap dump by issuing this command: kill -3 XXXXX where XXXXX is the process ID.

      The heap dump and an accompanying Java core file, which contains information about the 
      heap dump are created in the ~/WebSphere40/AppServer/bin/ directory, unless specified 
      otherwise.  The heap dump file name looks similar to heapdump28176.1089822616.txt.  
      The first group of numbers is the process ID, and the second group of numbers is the 
      system timestamp.

WebSphere Application Server V4.0, on a Windows system

   1. In the administrative console, navigate to and select the default application server.  
      In the file tree click WebSphere Administrative Domain > Nodes > localhost > 
      Application Servers > Default Server.
   2. On the General tab, click Environment and input both 
      IBM_HEAPDUMP = true and IBM_HEAPDUMP_OUTOFMEMORY = true.
   3. By default, the memory dumps is created in the ~\Websphere4\AppServer\bin\. directory.  
      To direct the destination of the heap dump to an alternate directory, click Environment, 
      and set IBM_HEAPDUMPDIR to the appropriate directory, that is\heapdumps).
   4. On the JVM Settings tab, click Advanced JVM Settings and enable garbage collection 
      verbose mode.
   5. Start the application server (Upon starting the application server, 
      tail the >Default_Server_stdout.log file and note the port number given to the 
      application server in the DrAdmin line).
   6. Invoke a heap dump by opening a command prompt and going to 
      the ~\Websphere4\AppServer\bin\ directory.  Issue the command:
      "DrAdmin -serverport XXXX -dumpThreads"
      where XXXX is the port number.

      The heap dump and an accompanying Java core file that contains information about 
      the heap dump       are created in the \Websphere4\AppServer\bin\ directory, 
      unless specified as otherwise.  The heap dump file looks similar to 
      "heapdump.20040709.105232.1140.txt".  The first group of numbers is the date, 
      the second group is the time, and the third group is the process ID.

WebSphere Application Server V5.x, V6.x on an AIX system

   1. In an Internet browser, access the administrative console at 
      http://hostname:9091/admin.
   2. Navigate to: Servers > Application Servers > Server1 (or the name of 
      the server to get a heap dump) > Process Definition > Environment Entries.
   3. Click New.
   4. In the name field, input IBM_HEAPDUMP.  In the value field, input true".
   5. Click OK.
   6. Repeat Steps 3 through 5, except set IBM_HEAPDUMP_OUTOFMEMORY to true.
   7. Memory dumps are created in the ~/WebSphere/AppServer/ directory by default 
      (for WebSphere Application Server V6.x, the default directory is: 
      ~/WebSphere/AppServer/profiles/default). To direct the destination of the 
      heap dump to an alternate directory, go to Environment Entries, click New, 
      set IBM_HEAPDUMPDIR to the appropriate directory, that is, /heapdumps, 
      and click OK.
   8. Click Save, and in the next screen click Save again.
   9. Navigate to: Servers > Application Servers > server1 (or the name of 
      the server to get a heap dump) > Process Definition > Java Virtual Machine.
  10. Select Verbose garbage collection.
  11. Click Save, and on the next screen click Save again.
  12. Restart the server.
  13. Open a command prompt and go to the /WebSphere/AppServer/bin directory.
  14. Invoke a heap dump by issuing a kill -3 XXXXX command, where XXXXX is the process ID.

      The heap dump and an accompanying Java core file that contains information about 
      the memory dump are created in the ~/WebSphere40/AppServer/directory, unless an 
      alternate directory is specified. The heap dump file name looks similar to: 
      heapdump57128.1090349140.txt. The first group of numbers is the process ID, 
      and the second group of numbers is the system timestamp.

WebSphere Application Server V5.x, V 6.x, run on a Windows system

   1. In an Internet browser, access the administrative console at 
      http://hostname:9091/admin.
   2. Navigate to: Servers > Application Servers > Server1 (or the name of 
      the server to get a heap dump) > Process Definition > Environment Entries
   3. Click New.
   4. In the name field, input IBM_HEAPDUMP. In the value field, input true.
   5. Click OK.
   6. Repeat Steps 3 through 5, except set IBM_HEAPDUMP_OUTOFMEMORY to true.
   7. To direct the destination of the heap dump to an alternate directory, go to 
      Environment Entries, click New, set IBM_HEAPDUMPDIR to the appropriate directory, 
      that is, /heapdumps, and click OK. Memory dumps are created in the 
      ~/WebSphere/AppServer/ directory by default (for WebSphere Application 
      Server V6.x, the default directory is: ~/WebSphere/AppServer/profiles/default.
   8. Click Save, and in the next screen click Save again.
   9. Navigate to Servers > Application Servers > Server1 (or the name of 
      the server to get a heap dump) > Process Definition > Java Virtual Machine
  10. Select Verbose garbage collection.
  11. Click Save, and on the next screen click Save again.
  12. Restart the server.
  13. Open a command prompt and go to the ~\WebSphere\AppServer\bin directory.
  14. Issue the command: wsadmin. The command creates an interactive session to server1.
  15. When you see "wsadmin>, issue the command: 
      set jvm [$AdminControl completeObjectName type=JVM,*], 
      which sets the variable jvm with the application server server1.
  16. Issue another command: $AdminControl invoke $jvm dumpThreads, which 
      tells the JVM to start a heap dump.

      The heap dump and an accompanying Java core file that contains information 
      about the memory dump are created in the ~/WebSphere40/AppServer/ directory, 
      unless an alternate directory is specified. The heap dump file name looks 
      similar to heapdump.20040630.140939.3944.txt. The first group of numbers is 
      the date, the second group is the time, and the third group is the process ID.


WebSphere Application Server V5.x,V6.x on a Solaris system

   1. In an Internet browser, access the administrative console at 
      http://hostname:9091/admin.
   2. Navigate to: Servers > Application Servers > Server1 (or the name of 
      the server to get  a heap dump) > Process Definition > Environment Entries.
   3. In General arguments type: -Xrunhprof:depth=0,heap=dump,format=a,thread=n,doe=n
      By default, the memory dumps are created in the ~/Websphere/AppServer/directory.  
      To direct the destination of the heap dump to an alternate directory, add 
      the HProf argument: file=/heapdumpdir/hprof.txt, where heapdumpdir is the 
      appropiate directory and hprof.txt is the appropriate file name.  If multiple 
      memory dumps are taken, each is appended to the one hprof.txt file.
   4. Select Enable garbage collection verbose mode.
   5. Restart the server.
   6. Take a heap dump by issuing this command: kill -3 XXXXX, where XXXXX is 
      the process ID.

      The hprof dump is created in the ~/WebSphere/AppServer/ directory and the 
      file name looks similar to java.hprof.txt, unless specified otherwise.
   7. Shut down the application server and then move the hprof dump file.  
      The hprof dump file is not complete until the application server is 
      shut down properly.

      Attention: Check each hprof dump for two sets of HEAP DUMP BEGIN and 
      HEAP DUMP END markers.  If both markers do not exist for each dump, 
      then the hprof dump is not complete and cannot be used for analyzing.

      During the run of a leaking application, it is necessary to take multiple 
      memory dumps at separate intervals. 

WebSphere Application Server V5.1, V6.x, on a z/OS system

Before WebSphere Application Server for z/OS Version 510.200, no easy means 
of gathering a memory dump from the WebSphere Application Server address 
spaces exists. If you have a version of the JVM 1.4.1 with a build date 
later than July, 2004, you might be able to code a servlet that invokes 
the heapdump method.

As a result, the procedure to analyze the heap on z/OS systems involves 
gathering a SVCDUMP of the address space with the heap that you want to analyze.

Prepare to get a SVCDUMP on a z/OS system.

Before you take the dump:

    * Try to run the application with just one SR/Servant.
    * It is best to take the memory dump when the bad behavior is occuring, 
      for example, stop, slow response, and so on.

      When done, the basic instructions for taking the dump are as follows:
         1. Bring up the application server with GC tracing running only.
         2. Type the console command: CD SET,SDUMP,MAXSPACE=2048M.
         3. Start your workload.
         4. When the workload is running steadily and assuming that the 
            performance issue is occurring, type the console command: ,999K,BR=ON.
         5. After 30 seconds, type the console command: DUMP COMM=([some comment])
         6. Answer the WTOR.  If you use SDSF, type a slash (/) on the command 
            line to get to a four-line entry screen: 
            [xx],ASID=([yy]),SDATA=(RGN,TRT,CSA,NUC,PSA,GRSQ,LPA,SQA,SUM) 
            where xx is the WTOR reply number and zz is the ASIDX of the address 
            spaces to dump. In V4.x with the plug-in, either dump the plug-in 
            address space or the server and control region address spaces, 
            depending on where the current investigation is focused.
         7. Wait 30 seconds after replying to the dump WTOR, and type the 
            console command: TRACE ST,OFF

      Move the dump to your workstation for processing:
         1. Use File Transfer Protocol to the host from your workstation
         2. Change directory to the name of the dump in quotes, for example, 
            cd 'my.dump.dataset' 
         3. Change to binary mode ex. bin .
         4. Type the command: get 'my.dump.dataset'. This action can take 
            30 minutes or so; the minimum dump size noticed is 800M.
 

Windows systems without WebSphere Application Server

   1. In the shell where the JVM is going to run, type: 
      set IBM_HEAPDUMP=true
   2. In the shell where the JVM is going to run, type: 
      set IBM_HEAPDUMP_OUTOFMEMORY=true
   3. To direct the destination of the heap dump to an alternate directory, type: 
      set IBM_HEAPDUMPDIR=/heapdumps, 
      or the appropriate directory. 
      Memory dumps are created in the current working directory by default.
   4. Enable Verbose GC:
          * For IBM Software Developer Kits, add this line to the 
            JVM command line: -verbose:gc -Xverbosegclog:verbosegc.log
          * For the Sun JDK, add this line to the JVM 
            command line: -Xloggc:verbosegc.log
   5. Start the JVM process and do not run the JVM as a background process.
   6. In the same shell where the JVM process is started, press Cntrl-Break.
      This action produces a memory dump and can also stop the current process.
   7. If the current process is stopped, restart the JVM and take the 
      second memory dump later.

AIX systems without WebSphere Application Server

   1. In the shell where the JVM is going to run, for the Bourne shell type: 
      export IBM_HEAPDUMP=true or for CSH:setenv IBM_HEAPDUMP true
   2. In the shell where the JVM is going to run, type: 
      export IBM_HEAPDUMP_OUTOFMEMORY=true
   3. To direct the destination of the heap dump to an alternate directory, 
      type: set IBM_HEAPDUMPDIR=/heapdumps, or the appropriate directory. 
      Memory dumps are created in the current working directory by default.
   4. Enable verbose GC by adding this line to the JVM command line: 
      -verbose:gc -Xverbosegclog:verbosegc.log
   5. Start the JVM process.
   6. Issue the command: kill -3 XXXXX, where XXXX is the process ID.

Solaris systems without WebSphere Application Server

   1. On a Java command line, add the following parameters to enable HProf: 
      -Xrunhprof:depth=0,heap=dump,format=a,thread=n,doe=n
   2. Enable verbose GC to the verbosegc.log file by adding this line to 
      the JVM command line: -Xloggc:verbosegc.log
   3. Start the JVM process.
   4. Issue the command: kill -3 XXXXX, where XXXX is the process ID.

ISeries IBM SDK (Version V5 R3) without WebSphere Application Server

1. In an x5250 session, type strsst and sign in.
2. Select option 1 - start a service tool.
3. Select option 4 - display/alter/dump.
4. Select option 2 - dump to printer.
5. Select option 2 - licensed internal code (lic) data.
6. Select option 14 - advanced analysis.
7. Scroll down and find JAVAGCINFO, type a 1 next to it, and press Enter.
8. Press Enter to run the javagcinfo macro with no parameters.
9. Press Enter again.
10. Type wrksplf, for which it is useful for a second x5250 session.
11. Use F18 to get to the bottom of the list.
12. Type 5 by your spool file and press Enter to display the file. The spool 
    file contains a short summary of each of the JVMs on the system.
13. Page down until you find your JVM (The job number, user, and name for each 
    JVM is located at the beginning of the data for each JVM).  Select the hex 
    string following VM*= on the line with the job number, and so on.
14. Return to the screen to run the javagcinfo macro through sst. If you use separate 
    x5250 sessions, the first session is already at this screen after step 9.
15. Type -heap  on the options line, where  is the string copied in 
    step 13, and press Enter.
16. Type 100000 for your Through page value to ensure you have enough space, and 
    press Enter. This action creates a spool file with the heapdump. 
17. To retrieve the heap dump, follow these steps to download the spool file 
    with iSeries Navigator.
   1. Double click "Add a connection" under the Environment Tasks work area.
   2. Type in the name of the server and enter.
   3.  Type in your userid on the server and enter.
   4.  Mouse click on Verify Connection.
   5.  Once the connection is verified, click on OK and then click on Finish.
   6.  In the My Connections work area, double click on the server you added
   7.  Type in your password on the server and enter.
   8. Select Basic Operations.
   9. Select Printer Output.
  10. Drag and drop the spool file from the list to a folder on your client system. 
      If you retrieve more than one spool file, the resulting text file names will 
      include spaces; you may find it convenient to change the file names before 
      proceeding to the next step.
  11. If you prefer to use Convert and the other tools on the iSeries, FTP the 
      resulting file to an ifs directory on the iSeries sever.

18. After getting the iSeries dump in a unicode text file, use the Convert command 
    from the updated svcdump.jar file to convert the text file to PHD format. 
    The ordinary syntax for this action, which works from QSHELL is: 
    java -classpath _path_to_svcdump.jar file_ com.ibm.jvm.ras.findroots.Convert 
    _file_name_. Note that the svcdump.jar file is available with the Memory Dump 
    Diagnostic for Java in the IBM Support Assistant installation folder in the 
    directory: plugins/com.ibm.websphere.mdd4j_1.0.0/WEB-INF/lib/svcdump.jar.

After you get the heap dump, use the Memory Dump Diagnostic for Java from IBM to analyze them.

    

Couldn't find what you need? Check out our blog entries.