You are not logged in.
My expectation was, when I check the multi thread execution option and I put two tRunJob components into my job, they would run in parallel. But they don’t.
My problem is that I need to compare data in groups between several tables. The database and the client is big enough to do the different group and table combinations in parallel (and running them in sequence takes much longer than the time window I have for execution). I encapsulated the logic (access to the data) for the different tales into different Talend jobs, but now I’m not able to run the job framework in parallel because of the tRunJob limitation (they do not execute in parallel).
The screenshot example is a simplified case of my problem. I call a job with two different set of parameter and the sub job logs the parameter to show execution progress.
As you can see, job BB starts after job AA has finished. My expectation is that they run in parallel and would show up as alternated lines in the log.
Any help on this?
Last edited by Vaiko (2008-07-13 13:03:09)
Check the 'Multi Thread Execution' option on Job setting tab and you should set a large number to test it.
Yes, Here is my test result:
Starting job fatherjob at 17:52 14/07/2008. jobA1 jobA2 jobA3 jobA4 jobA5 jobA6 jobA7 jobA8 jobA9 jobA10 jobA11 jobA12 jobA13 jobA14 jobB101 jobA15 jobB102 jobA16 jobB103 jobA17 jobB104 jobA18 jobB105 jobA19 jobB106 jobA20 jobB107 jobA21 jobB108 jobA22 jobB109 jobA23 jobB110 jobA24 jobB111 jobA25 jobB112 jobA26 jobB113 jobA27 jobB114 jobA28 jobB115 jobA29 jobB116 jobB117 jobB118 jobB119 jobB120 jobA30 jobB121 jobA31 jobB122 jobA32 jobB123 jobA33 jobB124 jobA34 jobB125 jobA35 jobB126 jobA36 jobA37 jobA38 jobA39 jobA40 jobA41 jobA42 jobA43 jobA44 jobA45 jobA46 jobA47 jobA48 jobB127 jobB128 jobA49 jobA50 jobA51 jobA52 jobA53 jobA54 jobA55 jobA56 jobA57 jobA58 jobA59 jobA60 jobA61 jobB129 jobB130 jobB131 jobB132 jobB133 jobB134 jobB135 jobB136 jobB137 jobB138 jobB139 jobB140 jobB141 jobB142 jobB143 jobB144 jobB145 jobB146 jobB147 jobB148 jobB149 jobB150 jobB151 jobB152 jobB153 jobB154 jobB155 jobB156 jobB157 jobB158 jobB159 jobB160 jobB161 jobB162 jobB163 jobB164 jobB165 jobB166 jobB167 jobB168 jobB169 jobB170 jobB171 jobB172 jobB173 jobB174 jobB175 jobB176 jobB177 jobB178 jobB179 jobB180 jobB181 jobB182 jobB183 jobB184 jobB185 jobB186 jobB187 jobB188 jobB189 jobB190 jobB191 jobB192 jobB193 jobB194 jobB195 jobB196 jobB197 jobB198 jobB199 jobA62 jobA63 jobA64 jobA65 jobA66 jobA67 jobA68 jobA69 jobA70 jobA71 jobA72 jobA73 jobA74 jobA75 jobA76 jobA77 jobA78 jobA79 jobA80 jobA81 jobA82 jobA83 jobA84 jobA85 jobA86 jobA87 jobA88 jobA89 jobA90 jobA91 jobA92 jobA93 jobA94 jobA95 jobA96 jobA97 jobA98 jobA99 Job fatherjob ended at 17:52 14/07/2008. [exit code=0]
Interesting. The difference between your example and what I did is that your example has not multiple threads in the sub jobs. Mine has, with the 3 parallel running threads registering data into a database (I don’t count the log thread separate but I think that’s the 4th). Du you know if having sub threads is preventing the tRunJobs from parallel execution?
Thanks a lot for your time providing the example. I’m trying to understand why it is not working as I’m expecting. I will check what happens if I serialize the sub job. Your job has only 2 levels mine has 3 (the sub job calls the actual working thread in a loop created by the tForeach) this could be another reason, why what you did, runs different.
If someone is interested I found the problem.
1. If I call two times the same child job with different parameters the jobs will run in sequence (my original example).
2. If I call two different child jobs the jobs will run in parallel (shong’s example).
3. If I call two different jobs which then call the same job with the different parameters, the jobs will run in parallel. So the workaround is basically to build a proxy child job per child job call. Not nice but doable for my current case (I will add an example if someone is interested, when I have a little bit more time).
I found the solution to the problem : Apart from the Master job, you need to ALSO set the "Enable Parallel Execution" in EACH Child job. Doing this will allow the Parent and Child run parallel.