• Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » how to implement a FIFO for orchestrating external execution

#1 2010-06-23 17:45:34

nicolasdiogo
Member
Company: BrainPowered - BI Services UK
Registered: 2006-11-24
Posts: 380
Website

how to implement a FIFO for orchestrating external execution

hello,

i have call an external process some extracts.  and since it is a single thead process it maximises one of the CPUs and only runs one extract at a time.
as i have a multi-cpu machine available i am trying run multiple instances at the same time (not very clear - sorry!).

i have created a job that encapsulates the whole thing of calling the external process and only requires a couple of variables to run properly.

now to orchestrate multiple instances of this process, i have used a import java.util.Queue object as a context variable but it still only gets me ONE instance at a time.

is there any document or discussions on how to run a multiple instances of job?

thanks,

Nicolas

Offline

#2 2010-06-23 18:53:39

JohnGarrettMartin
Member
Registered: 2009-01-07
Posts: 762

Re: how to implement a FIFO for orchestrating external execution

have you tried calling your job as a sub-job of a master wrapper job?  I believe you can get the parallel execution you want using this method.

Offline

#3 2010-06-23 21:56:30

nicolasdiogo
Member
Company: BrainPowered - BI Services UK
Registered: 2006-11-24
Posts: 380
Website

Re: how to implement a FIFO for orchestrating external execution

thanks John,

at present i have a ParentJob calling a ChildJob

what you seem to suggest is to have a job in the middle?! like:

ParentJob => JobWrapper => ChildJob

is this correct?
if so could you explain how it works?

many thanks,

Nicolas

Offline

#4 2010-06-23 22:30:02

JohnGarrettMartin
Member
Registered: 2009-01-07
Posts: 762

Re: how to implement a FIFO for orchestrating external execution

I was thinking something along the lines of:

tParallelize -+-> Child job
                   --> Child job
                   --> Child job
                   --> Child job

Offline

#5 2010-06-24 10:38:15

nicolasdiogo
Member
Company: BrainPowered - BI Services UK
Registered: 2006-11-24
Posts: 380
Website

Re: how to implement a FIFO for orchestrating external execution

tParallelize ?

which version of TOS are you using?
i have 4.0.2 - can not find it.

Offline

  • Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » how to implement a FIFO for orchestrating external execution

Board footer

Powered by FluxBB