Print this page
Friday, 22 July 2016 14:43

Talend connection component retry logic & design for databases, web services etc...

Written by
Rate this item
(9 votes)

For the instances where for whatever reason, a Talend job does not always connect to a backend service layer - database, web service, ftp, salesforce, dropbox - on the first try, the job can be modified to retry the connection before failing the job. This solution shows a simple design to handle intermittent connection failures.

The following screenshots visually describe a retry solution for Talend component connections. You can update the code to use context variables or global variables for the values that are hard-coded - wait time (2 seconds), number of retries (5).

 retry001

 

Starting job - create a boolean context variable and set to true

retry002

// resetting loop flag
context.continueLooping = true;

log.info("Starting job...");

 

Set Condition on tLoop to variable on While loop

retry003

 

Now, connect to the resource - here MDM (could be FTP, web service, database etc...)

retry004

 

Using onComponentOK (for successful connection), set the boolean variable to false (to exit loop).

retry005

 // on success, exit loop

context.continueLooping = false;

log.info("Connection established on attempt # " + ((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")).toString());

Using onComponentError (for failed connection), retry for 5 times then proceed (which will fail the job); else retry the job.

retry006

if (((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) > 5)
{
log.error("Connection failed. Exiting retry loop.");
// exit loop
context.continueLooping = false;
}
else
{
log.info("Connection failed. Retrying...");
}

 

Read 20773 times Last modified on Sunday, 24 July 2016 16:41
Will Munji

Will Munji is a seasoned data integration, data warehousing and business intelligence (BI) architect & developer who has been working in the DW/BI space for a while. He got his start in BI working on Brio SQR (later Hyperion SQR) and the Crystal Decisions stack (Reports, Analysis & Enterprise) and SAP BusinessObjects / Microsoft BI stacks. He currently focuses on Talend Data Management Suite, Hadoop, SAP BusinessObjects BI stack as well as Jaspersoft and Tableau. He has consulted for many organizations across a variety of industries including healthcare, manufacturing, retail, insurance and banking. At Kindle Consulting, Will delivers DW/BI/Data Integration solutions that range from front-end BI development (dashboards, reports, cube development, T-SQL/ PL/SQL ...) to data services (ETL/ DI development), data warehouse architecture and development, data integration to BI Architecture design and deployment.

Latest from Will Munji

Related items