In order to find memory leaks within a java application it is sometimes not enough looking at thread dumps. With tools like visualgc, a tool the see the different memory pools inside a running java app, you can tune the memory setup concering heap size, perm size, survivor ratio, etc., but you can't see which type of objects are causing the heap to fill up.

Enable Heap Dumps

There are several possibilities to enable heap dumps.

One way is to start to application with -XX:+HeapDumpOnCtrlBreak (Introduced since JDK 1.4.2 update 12 and 5.0 update 7), see for additional information.
Sending signal SIGQUIT (3) to the running process causes the jvm to write a heap dump into its working directory. Unfortunately the -XX:HeapDumpPath option, to set the path to write the heap dump file to, currently doesn't work. See Sun Bug report.

Another way to get a heap dump of any running java application is using jmap, which is now part of the jdk.

jmap -heap:format=b pid

writes heap.bin to current directory. Since JDK 1.6 jmap you can use -heap:format=b,file=<heap_dump.out>

Analyse Heap Dumps

The best tool start with is probably hat, which is part of the jdk since JDK1.6 and called jhat.

If you don't have java 6 yet, get hat from here. Good examples and Documentation are available as well.
Prior to JDK 1.6 call: (attune heap size if hat to the size of your heap dumps)

java -jar hat.jar -Xmx512M heap_dump.out

After hat finished analysing the heap dump, it starts a HTTP server accessible on port 7000.