Talend community forum / tRunJob – parallel execution?

Vaiko · 2008-07-13 12:53:45

Tags: [java, multi thread, oracle, parallel execution, tos2.4.0, tos2.4.1, trunjob]

My expectation was, when I check the multi thread execution option and I put two tRunJob components into my job, they would run in parallel. But they don’t.

My problem is that I need to compare data in groups between several tables. The database and the client is big enough to do the different group and table combinations in parallel (and running them in sequence takes much longer than the time window I have for execution). I encapsulated the logic (access to the data) for the different tales into different Talend jobs, but now I’m not able to run the job framework in parallel because of the tRunJob limitation (they do not execute in parallel).

The screenshot example is a simplified case of my problem. I call a job with two different set of parameter and the sub job logs the parameter to show execution progress.

As you can see, job BB starts after job AA has finished. My expectation is that they run in parallel and would show up as alternated lines in the log.

Any help on this?

Uploaded Images

Last edited by Vaiko (2008-07-13 13:03:09)

shong · 2008-07-14 04:30:42

Hello Vaiko

Check the 'Multi Thread Execution' option on Job setting tab and you should set a large number to test it.

Best regards

shong

Uploaded Images

Vaiko · 2008-07-14 09:48:26

The option is set and I tried with 8 as well. They are not really executing at the same time.
The 3 streams at the lowest level do run parallel but not the 2 jobs on the top level. Did you try?

Uploaded Images

shong · 2008-07-14 11:59:14

Hello

Yes, Here is my test result:

Code:

Starting job fatherjob at 17:52 14/07/2008.
jobA1
jobA2
jobA3
jobA4
jobA5
jobA6
jobA7
jobA8
jobA9
jobA10
jobA11
jobA12
jobA13
jobA14
jobB101
jobA15
jobB102
jobA16
jobB103
jobA17
jobB104
jobA18
jobB105
jobA19
jobB106
jobA20
jobB107
jobA21
jobB108
jobA22
jobB109
jobA23
jobB110
jobA24
jobB111
jobA25
jobB112
jobA26
jobB113
jobA27
jobB114
jobA28
jobB115
jobA29
jobB116
jobB117
jobB118
jobB119
jobB120
jobA30
jobB121
jobA31
jobB122
jobA32
jobB123
jobA33
jobB124
jobA34
jobB125
jobA35
jobB126
jobA36
jobA37
jobA38
jobA39
jobA40
jobA41
jobA42
jobA43
jobA44
jobA45
jobA46
jobA47
jobA48
jobB127
jobB128
jobA49
jobA50
jobA51
jobA52
jobA53
jobA54
jobA55
jobA56
jobA57
jobA58
jobA59
jobA60
jobA61
jobB129
jobB130
jobB131
jobB132
jobB133
jobB134
jobB135
jobB136
jobB137
jobB138
jobB139
jobB140
jobB141
jobB142
jobB143
jobB144
jobB145
jobB146
jobB147
jobB148
jobB149
jobB150
jobB151
jobB152
jobB153
jobB154
jobB155
jobB156
jobB157
jobB158
jobB159
jobB160
jobB161
jobB162
jobB163
jobB164
jobB165
jobB166
jobB167
jobB168
jobB169
jobB170
jobB171
jobB172
jobB173
jobB174
jobB175
jobB176
jobB177
jobB178
jobB179
jobB180
jobB181
jobB182
jobB183
jobB184
jobB185
jobB186
jobB187
jobB188
jobB189
jobB190
jobB191
jobB192
jobB193
jobB194
jobB195
jobB196
jobB197
jobB198
jobB199
jobA62
jobA63
jobA64
jobA65
jobA66
jobA67
jobA68
jobA69
jobA70
jobA71
jobA72
jobA73
jobA74
jobA75
jobA76
jobA77
jobA78
jobA79
jobA80
jobA81
jobA82
jobA83
jobA84
jobA85
jobA86
jobA87
jobA88
jobA89
jobA90
jobA91
jobA92
jobA93
jobA94
jobA95
jobA96
jobA97
jobA98
jobA99
Job fatherjob ended at 17:52 14/07/2008. [exit code=0]

Best regards

shong

Uploaded Images

Vaiko · 2008-07-15 01:10:17

Interesting. The difference between your example and what I did is that your example has not multiple threads in the sub jobs. Mine has, with the 3 parallel running threads registering data into a database (I don’t count the log thread separate but I think that’s the 4th). Du you know if having sub threads is preventing the tRunJobs from parallel execution?

Thanks a lot for your time providing the example. I’m trying to understand why it is not working as I’m expecting. I will check what happens if I serialize the sub job. Your job has only 2 levels mine has 3 (the sub job calls the actual working thread in a loop created by the tForeach) this could be another reason, why what you did, runs different.

Vaiko · 2008-07-20 12:10:51

Hi shong

what do I need to do to get a code frame into a post like you did?

Vaiko · 2008-07-22 12:54:03

If someone is interested I found the problem.

1. If I call two times the same child job with different parameters the jobs will run in sequence (my original example).

2. If I call two different child jobs the jobs will run in parallel (shong’s example).

3. If I call two different jobs which then call the same job with the different parameters, the jobs will run in parallel. So the workaround is basically to build a proxy child job per child job call. Not nice but doable for my current case (I will add an example if someone is interested, when I have a little bit more time).

zguptn · 2009-10-06 23:07:21

Hi Vaiko/shong

I found the solution to the problem : Apart from the Master job, you need to ALSO set the "Enable Parallel Execution" in EACH Child job. Doing this will allow the Parent and Child run parallel.

Thanks
Z

Announcement

Unanswered posts

#1 2008-07-13 12:53:45

tRunJob – parallel execution?

#2 2008-07-14 04:30:42

Re: tRunJob – parallel execution?

#3 2008-07-14 09:48:26

Re: tRunJob – parallel execution?

#4 2008-07-14 11:59:14

Re: tRunJob – parallel execution?

Code:

#5 2008-07-15 01:10:17

Re: tRunJob – parallel execution?

#6 2008-07-20 12:10:51

Re: tRunJob – parallel execution?

#7 2008-07-22 12:54:03

Re: tRunJob – parallel execution?

#8 2009-10-06 23:07:21

Re: tRunJob – parallel execution?

Board footer