Managing JVM heap size in Talend Open Studio

on Wednesday, 18 September 2013. Posted in Talend OS/IS

Talend Open Studio (DI / Big Data and other versions as well) allows you to manage your machine's JVM heap size (allocated memory) in a number of places. Not having the right amount of memory can result in a number of errors including the following:

demo

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx2048M
The specified size exceeds the maximum representable size.

The JVM heap size can be modified as follows:
1 - Talend .ini File: You can modify the memory allocated to Talend Studio by modifying the Studio .ini configuration file, such as TOS_DI-win32-x86.ini for 32-bit Windows systems. This configuration file is located in your Studio installation folder.
By default the ini file includes the following JVM parameters:

-vmargs
-Xms64m
-Xmx768m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

The memory that you can allocate to your Talend Studio depends mostly on your system memory availability. The following are common recommendations:
With 2 GB of memory available on a 32-bit system, bounds can be changed as follows:

-vmargs
-Xms256m
-Xmx1024m
-XX:MaxPermSize=256m
-Dfile.encoding=UTF-8

With 8 GB of memory available on 64-bit system, the optimal settings can be

-vmargs
-Xms1024m
-Xmx4096m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

To test how much memory can be allocated to your JVM, from a command line, run the following:
java -Xmx????m -version where ???? are valid memory assignments - 512, 1024, 2048, 4096. Use the amount that your system accepts (when it returns your default java environment info) - see screenshot below.

demo

2 - The studio settings: Go to Windows > Preferences

demo

3 - Job Properties: JVM heap size can be set on the job itself: Open the job, go to the Run tab, click on Advanced Settings:

demo

Comments (6)

  • sanvaibhav

    sanvaibhav

    05 March 2015 at 10:38 |
    Hi Will,

    Whether following error

    "-Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded"

    could be resolved using above workarounds? or we need special attention or method to solve this?

    reply

    • Will

      Will

      05 March 2015 at 10:40 |
      Hello Sanvaibhav - this error indicates that the Talend job runs, however it fails because of it's spending too much time in garbage collection.

      I've seen jobs where the GC (garbage collection) does not happen fast enough, when running high-memory usage jobs - such as processing heavy XML files. In that case, increasing the heap size (Xmx) would be recommended. It may be that your JVM is struggling on memory and GC is not freeing up memory fast enough.

      Other alternatives would be to include design considerations that leverage disk over RAM (for example storing data on disk with the tMap component), break up the job into smaller pieces (by looping through a large dataset instead of processing it all at once), kicking off an independent process (jvm thread) when using child jobs. Your current job may be trying to do too much in the single thread within the JVM...

      Let me know what option you try and what works.

      Thanks..

      reply

  • Andrew

    Andrew

    05 March 2015 at 10:55 |
    Hi there,

    If you have a 64-bit machine then you will be able to optimize memory usage better.
    You should run the Talend 64-bit application instead of the 32-bit one. i.e. TOS_DI-win-x86_64.exe

    But first thing first, you need to install java 64 bit version ( www.java.com). Please note that you can have both 32-bit and 64-bit Java Virtual Machine running on the same machine.
    Please add the following parameter to your JVM Runtime/Configuration parameters:
    -XX:+UseConcMarkSweepGC
    This garbage collector parameter will free up/ release memory as soon as its no longer in use, so it can be used by other parts of your program.

    As an example, I am running a server with a total memory capacity of 96GB.

    In my configuration file or the Java run-time parameters I use the following settings:

    -Xms32768M
    -Xmx65536M
    -XX:+UseConcMarkSweepGC

    This works wonders for me. No more memory error issues.

    I hope this helps.

    Andrew
    (agapare@hotmail.com)

    reply

    • Will Munji

      Will Munji

      05 March 2015 at 11:00 |
      Thank you, Andrew. Very useful tips...
      Will

      reply

  • rajesh

    rajesh

    05 March 2015 at 11:05 |
    Thanks Andrew. Its working like wonder


    Rajesh A

    reply

Leave a comment

You are commenting as guest.

Contact Us

Kindle Consulting, Inc

6595 Roswell Road 
Suite G2025 
Atlanta, GA 30328

Phone: 404.551.5494
Fax: 404.551.5452
Email: info@kindleconsulting.com

Talend Gold Partner