Managing JVM heap size in Talend Open Studio

Wednesday, 18 September 2013 09:45

Managing JVM heap size in Talend Open Studio

Written by Will Munji

font size decrease font size increase font size
6 comments

Rate this item

(1 Vote)

Talend Open Studio (DI / Big Data and other versions as well) allows you to manage your machine's JVM heap size (allocated memory) in a number of places. Not having the right amount of memory can result in a number of errors including the following:

demo

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx2048M
The specified size exceeds the maximum representable size.

The JVM heap size can be modified as follows:
1 - Talend .ini File: You can modify the memory allocated to Talend Studio by modifying the Studio .ini configuration file, such as TOS_DI-win32-x86.ini for 32-bit Windows systems. This configuration file is located in your Studio installation folder.
By default the ini file includes the following JVM parameters:

-vmargs
-Xms64m
-Xmx768m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

The memory that you can allocate to your Talend Studio depends mostly on your system memory availability. The following are common recommendations:
With 2 GB of memory available on a 32-bit system, bounds can be changed as follows:

-vmargs
-Xms256m
-Xmx1024m
-XX:MaxPermSize=256m
-Dfile.encoding=UTF-8

With 8 GB of memory available on 64-bit system, the optimal settings can be

-vmargs
-Xms1024m
-Xmx4096m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

To test how much memory can be allocated to your JVM, from a command line, run the following:
java -Xmx????m -version where ???? are valid memory assignments - 512, 1024, 2048, 4096. Use the amount that your system accepts (when it returns your default java environment info) - see screenshot below.

2 - The studio settings: Go to Windows > Preferences

3 - Job Properties: JVM heap size can be set on the job itself: Open the job, go to the Run tab, click on Advanced Settings:

Read 14498 times Last modified on Wednesday, 03 June 2015 20:48

Published in Talend DI

Tagged under

Will Munji

Will Munji is a seasoned data integration, data warehousing and business intelligence (BI) architect & developer who has been working in the DW/BI space for a while. He got his start in BI working on Brio SQR (later Hyperion SQR) and the Crystal Decisions stack (Reports, Analysis & Enterprise) and SAP BusinessObjects / Microsoft BI stacks. He currently focuses on Talend Data Management Suite, Hadoop, SAP BusinessObjects BI stack as well as Jaspersoft and Tableau. He has consulted for many organizations across a variety of industries including healthcare, manufacturing, retail, insurance and banking. At Kindle Consulting, Will delivers DW/BI/Data Integration solutions that range from front-end BI development (dashboards, reports, cube development, T-SQL/ PL/SQL ...) to data services (ETL/ DI development), data warehouse architecture and development, data integration to BI Architecture design and deployment.

Latest from Will Munji

6 comments

Comment Link Saturday, 10 January 2015 14:24 posted by rajesh
Thanks Andrew. Its working like wonder

Rajesh A
Comment Link Monday, 27 October 2014 16:19 posted by Will Munji
Thank you, Andrew. Very useful tips...
Will
Comment Link Wednesday, 03 September 2014 10:22 posted by Andrew
Hi there,

If you have a 64-bit machine then you will be able to optimize memory usage better.
You should run the Talend 64-bit application instead of the 32-bit one. i.e. TOS_DI-win-x86_64.exe

But first thing first, you need to install java 64 bit version (http://www.java.com). Please note that you can have both 32-bit and 64-bit Java Virtual Machine running on the same machine.
Please add the following parameter to your JVM Runtime/Configuration parameters:
-XX:+UseConcMarkSweepGC
This garbage collector parameter will free up/ release memory as soon as its no longer in use, so it can be used by other parts of your program.

As an example, I am running a server with a total memory capacity of 96GB.

In my configuration file or the Java run-time parameters I use the following settings:

-Xms32768M
-Xmx65536M
-XX:+UseConcMarkSweepGC

This works wonders for me. No more memory error issues.

I hope this helps.

Andrew
(agapare@hotmail.com)
Comment Link Friday, 22 August 2014 09:18 posted by sanvaibhav
Adding an option in Talend environment helped to solved issue.

Windows-->Preferences-->Talend-->Run/Debug - XX:-UseGCOverheadLimit

http://www.talendforge.org/forum/viewtopic.php?pid=143444#p143444

Further more, at safer side later I also increased Xmx to 10240m to avoid further complications
Comment Link Monday, 18 August 2014 15:42 posted by Will Munji
Hello Sanvaibhav - this error indicates that the Talend job runs, however it fails because of it's spending too much time in garbage collection.

I've seen jobs where the GC (garbage collection) does not happen fast enough, when running high-memory usage jobs - such as processing heavy XML files. In that case, increasing the heap size (Xmx) would be recommended. It may be that your JVM is struggling on memory and GC is not freeing up memory fast enough.

Other alternatives would be to include design considerations that leverage disk over RAM (for example storing data on disk with the tMap component), break up the job into smaller pieces (by looping through a large dataset instead of processing it all at once), kicking off an independent process (jvm thread) when using child jobs. Your current job may be trying to do too much in the single thread within the JVM...

Let me know what option you try and what works.

Thanks..
Comment Link Monday, 18 August 2014 11:25 posted by sanvaibhav
Hi Will,

Whether following error

"-Exception in thread "Thread-0" java.lang.OutOfMemoryError: GC overhead limit exceeded"

could be resolved using above workarounds? or we need special attention or method to solve this?