Wednesday, 18 September 2013 09:45

Managing JVM heap size in Talend Open Studio

Written by
Rate this item
(1 Vote)

Talend Open Studio (DI / Big Data and other versions as well) allows you to manage your machine's JVM heap size (allocated memory) in a number of places. Not having the right amount of memory can result in a number of errors including the following:

demo

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx2048M
The specified size exceeds the maximum representable size.

The JVM heap size can be modified as follows:
1 - Talend .ini File: You can modify the memory allocated to Talend Studio by modifying the Studio .ini configuration file, such as TOS_DI-win32-x86.ini for 32-bit Windows systems. This configuration file is located in your Studio installation folder.
By default the ini file includes the following JVM parameters:

-vmargs
-Xms64m
-Xmx768m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

The memory that you can allocate to your Talend Studio depends mostly on your system memory availability. The following are common recommendations:
With 2 GB of memory available on a 32-bit system, bounds can be changed as follows:

-vmargs
-Xms256m
-Xmx1024m
-XX:MaxPermSize=256m
-Dfile.encoding=UTF-8

With 8 GB of memory available on 64-bit system, the optimal settings can be

-vmargs
-Xms1024m
-Xmx4096m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

To test how much memory can be allocated to your JVM, from a command line, run the following:
java -Xmx????m -version where ???? are valid memory assignments - 512, 1024, 2048, 4096. Use the amount that your system accepts (when it returns your default java environment info) - see screenshot below.

demo

2 - The studio settings: Go to Windows > Preferences

demo

3 - Job Properties: JVM heap size can be set on the job itself: Open the job, go to the Run tab, click on Advanced Settings:

demo
Read 8433 times Last modified on Wednesday, 03 June 2015 20:48
Will Munji

Will Munji is a seasoned data integration, data warehousing and business intelligence (BI) architect & developer who has been working in the DW/BI space for a while. He got his start in BI working on Brio SQR (later Hyperion SQR) and the Crystal Decisions stack (Reports, Analysis & Enterprise) and SAP BusinessObjects / Microsoft BI stacks. He currently focuses on Talend Data Management Suite, Hadoop, SAP BusinessObjects BI stack as well as Jaspersoft and Tableau. He has consulted for many organizations across a variety of industries including healthcare, manufacturing, retail, insurance and banking. At Kindle Consulting, Will delivers DW/BI/Data Integration solutions that range from front-end BI development (dashboards, reports, cube development, T-SQL/ PL/SQL ...) to data services (ETL/ DI development), data warehouse architecture and development, data integration to BI Architecture design and deployment.

Leave a comment

Make sure you enter all the required information, indicated by an asterisk (*). HTML code is not allowed.

6 comments

  • Comment Link rajesh Saturday, 10 January 2015 14:24 posted by rajesh

    Thanks Andrew. Its working like wonder


    Rajesh A

  • Comment Link Will Munji Monday, 27 October 2014 16:19 posted by Will Munji

    Thank you, Andrew. Very useful tips...
    Will

  • Comment Link Andrew Wednesday, 03 September 2014 10:22 posted by Andrew

    Hi there,

    If you have a 64-bit machine then you will be able to optimize memory usage better.
    You should run the Talend 64-bit application instead of the 32-bit one. i.e. TOS_DI-win-x86_64.exe

    But first thing first, you need to install java 64 bit version (http://www.java.com). Please note that you can have both 32-bit and 64-bit Java Virtual Machine running on the same machine.
    Please add the following parameter to your JVM Runtime/Configuration parameters:
    -XX:+UseConcMarkSweepGC
    This garbage collector parameter will free up/ release memory as soon as its no longer in use, so it can be used by other parts of your program.

    As an example, I am running a server with a total memory capacity of 96GB.

    In my configuration file or the Java run-time parameters I use the following settings:

    -Xms32768M
    -Xmx65536M
    -XX:+UseConcMarkSweepGC

    This works wonders for me. No more memory error issues.

    I hope this helps.

    Andrew
    (agapare@hotmail.com)

  • Comment Link sanvaibhav Friday, 22 August 2014 09:18 posted by sanvaibhav

    Adding an option in Talend environment helped to solved issue.

    Windows-->Preferences-->Talend-->Run/Debug - XX:-UseGCOverheadLimit

    http://www.talendforge.org/forum/viewtopic.php?pid=143444#p143444

    Further more, at safer side later I also increased Xmx to 10240m to avoid further complications

  • Comment Link Will Munji Monday, 18 August 2014 15:42 posted by Will Munji

    Hello Sanvaibhav - this error indicates that the Talend job runs, however it fails because of it's spending too much time in garbage collection.

    I've seen jobs where the GC (garbage collection) does not happen fast enough, when running high-memory usage jobs - such as processing heavy XML files. In that case, increasing the heap size (Xmx) would be recommended. It may be that your JVM is struggling on memory and GC is not freeing up memory fast enough.

    Other alternatives would be to include design considerations that leverage disk over RAM (for example storing data on disk with the tMap component), break up the job into smaller pieces (by looping through a large dataset instead of processing it all at once), kicking off an independent process (jvm thread) when using child jobs. Your current job may be trying to do too much in the single thread within the JVM...

    Let me know what option you try and what works.

    Thanks..

  • Comment Link sanvaibhav Monday, 18 August 2014 11:25 posted by sanvaibhav

    Hi Will,

    Whether following error

    "-Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded"

    could be resolved using above workarounds? or we need special attention or method to solve this?

Contact Us

Kindle Consulting, Inc

6595 Roswell Road 
Suite G2025 
Atlanta, GA 30328

Phone: 404.551.5494
Fax: 404.551.5452
Email: info@kindleconsulting.com

Talend Gold Partner