Talend community forum / Problem with context variables

raghu12345 · 2012-05-30 14:52:37

i tried to use 2 context variables for fetching max id value of 2 tables,
it is fetching correctly later i am using this max values in 2 sub jobs for sequence number along with data as follows

tmssqlinput---->tjavarow---->(component ok)--->tmssqlinput---->tmap----->tmssqloutput

tmssqlinput---->tjavarow---->(component ok)--->tmssqlinput---->tmap----->tmssqloutput

running these 2 subjobs parallely by using tparallelize component

it is giving sequence no as

1
2
3
4
5
7
9
11
and so on for first table

6
8
10
12
14
16
and so on for second table

Last edited by raghu12345 (2012-06-01 14:21:05)

janhess · 2012-05-30 16:35:45

What do you expect? If you want independant sequences use different sequence id.

raghu12345 · 2012-05-31 08:52:36

i want sequence number.
after loading first time and when i need to load data again into same tables, it should start the sequence number
from max of sequence number already inserted into table.

for example

for fisrt load

1
2
3
4
and so on upto 1034

for second load it should start from 1035 number

1035
1036
1037
and so on

for saving the previous load max number i am using context variable

It is working fine for one job but i need to run 2 subjobs parallely so i used 2 context variable for each job
then it is mulfunctioning

raghu12345 · 2012-06-01 14:18:45

when i run the jobs separately the context variable works fine but when i run the same jobs parallely the sequence number is not coming correctly as required. my job structure looks looks like this

tparallelize-------(job1)
|
|
------------(job2)

and each jobs is like

tmssqlinput---->tjavarow---->(component ok)--->tmssqlinput---->tmap----->tmssqloutput

1) In first tmssqlinput i am fetching maximum seq num
2) In tjavarow i am saving the maximum seq value in context variable
3) In tmap i use the same context variable in Numeic.sequnce("S1",context.v1,1)

I am using same logic in both jobs

Last edited by raghu12345 (2012-06-01 14:32:35)

djugal · 2012-06-01 14:55:52

it cant work in parallel as the sequence of execution of job 1 has to be completed to fetch the last record
try onsubjobok

raghu12345 · 2012-06-04 07:53:32

sorry you are not getting my problem.

I have two jobs named job1 and job2 with the following flow.

job1----------->tmssqlinput---->tjavarow---->(component ok)--->tmssqlinput---->tmap----->tmssqloutput

and i am using
1) In first tmssqlinput i am fetching maximum seq num
2) In tjavarow i am saving the maximum seq value in context variable
3) In tmap i use the same context variable in Numeic.sequnce("S1",context.v1,1)

The job2 is similar to job1 as follows

job2---------->tmssqlinput---->tjavarow---->(component ok)--->tmssqlinput---->tmap----->tmssqloutput

and i am using
1) In first tmssqlinput i am fetching maximum seq num
2) In tjavarow i am saving the maximum seq value in context variable
3) In tmap i use the same context variable in Numeic.sequnce("S1",context.v1,1)

when i run single job at once it will load correctly but I need to run these two jobs at the same time then the problem comes when i run the below job

tparallelize-------(job1)
|
|
------------(job2)

Last edited by raghu12345 (2012-06-04 08:29:04)

djugal · 2012-06-04 17:46:50

you see the user guide mentions that in parallel processing context/global variables does not work.

you have to do a onsubjobok, else you can use a bulk load to load load data faster if needed.

raghu12345 · 2012-06-05 06:11:35

thank you Djugal

Announcement

Unanswered posts

#1 2012-05-30 14:52:37

Problem with context variables

#2 2012-05-30 16:35:45

Re: Problem with context variables

#3 2012-05-31 08:52:36

Re: Problem with context variables

#4 2012-06-01 14:18:45

Re: Problem with context variables

#5 2012-06-01 14:55:52

Re: Problem with context variables

#6 2012-06-04 07:53:32

Re: Problem with context variables

#7 2012-06-04 17:46:50

Re: Problem with context variables

#8 2012-06-05 06:11:35

Re: Problem with context variables

Board footer