Talend community forum / [resolved] Generate tdelimitedfiles with counters

rubensilva · 2011-12-23 18:15:01

Tags: [counters, CSV file, delimited file, variables]

Hello all.
I am new in Talend, I didn't find any post regarding this matter.
Lets say I want to put in a csv file some data, for ex: id, name.

Result:

1,John
2,Joe
3,Will

The id will be the counter. The counter now is 3, and I want the next file(that will replace the last one) to put the new names only, so after the last counted number(3).

So the next result would be:

4,Richard
5,Anna
6,Paul
7, Etc

Counter now is 7...and so on....

Thank you all a Merry Xmas

pedro · 2011-12-26 04:07:44

Hi

You can declare a context variable 'context.new1' and use this method in tMap or other component to set the expression of column 'id'.

Code:

Numeric.sequence(context.new1,1,1)

Best regards!
Pedro

rubensilva · 2011-12-29 00:15:03

Hello.
Maybe is my mistake(i am new in talend)
What I did:
Tfirebirdinput to tmap and then into tlogrow
contextjob as you sugested and the expression(row1.ID.Numeric.sequence(context.new1,1,1) on tmap might have the error:
"Numeric cannot be resolved or is not a field".

Please take a look the screen shots.

Uploaded Images

shong · 2011-12-29 02:15:34

Hi
The expression should be:

Code:

Numeric.sequence(context.new1,1,1)

I would like to clarify that you want to append new records into the existing file or create a new file and insert records?

Best regards
Shong

rubensilva · 2011-12-29 15:55:33

Hello.
It's working now(thanks!) but it's not what I want.
Maybe I've explained in the wrong way.
What I want is to extract from a db only new records into a csv files.

file_x:
id,name
1,John
2,Joe
3,Will

New Filex,

4,Richard
5,Anna
6,Paul
7, Etc

And so on.

Thanks in advance.

pedro · 2011-12-30 03:13:00

Hi

You mean, when first time you run the job, this variable may be 3.
And second time it has to start with 4?

You may save this value into a txt file or something.
When you run the job every time, get the value from file and pass it to a new context variable.

Best regards!
Pedro

rubensilva · 2011-12-31 01:59:40

Thanks for the reply.
I am catching the idea, but I can I get that value from the file?

For ex:

Select id, name from table where id>"the last id saved on the file"?

pedro · 2011-12-31 02:42:49

Hi

You may create a subjob like this.
tFileinputDelimited--main-->tJava(get last id from file).
And in the end of your job, add another subjob.
tFixedInputRow(save context variable)--main-->tFileOutputDelimited.
Use OnSubjobOk to link all these subjobs.

Best regards!
Pedro

rubensilva · 2012-01-02 21:23:10

Hello and Happy New Year!

I am trying your solution but I am missing something.

I have tfirebirdinput_1("select id, name, current_timestamp from table order by id desc") -> tmap_1 -> tsamplerow_1(get 1st line) -> tfileoutputdelimited_1(id) - this way i am saving last id.

After this, I am not understanding how can I query last id saved on the file... Has i said before, all I want is select id, name from table where id > than last query id(saved on the file)

Thanks again for your help.

Uploaded Images

rubensilva · 2012-01-05 10:35:45

Hello.
Anyone?

rubensilva · 2012-01-20 23:57:46

Hello!
Anyone Can help me?

Thanks!

shong · 2012-01-21 02:11:47

Hi
Read the last id from that file and store it into a global variable on tJavaRow, for example:
....
tFileInputDelimited--main---tJavaRow
|
onsubjob
|
tfirebirdinput_3--main---tLogRow

on tFileInputDelimited: read the last id from the file where the last id is saved.

on tJavaRow:
globalMap.put("last_id",input_row.id);

on tfirebirdinput_3:
"select id, name from table_name where id>"+(Integer)globalMap.get("last_id")

Best regards
Shong

Announcement

Unanswered posts

#1 2011-12-23 18:15:01

[resolved] Generate tdelimitedfiles with counters

#2 2011-12-26 04:07:44

Re: [resolved] Generate tdelimitedfiles with counters

Code:

#3 2011-12-29 00:15:03

Re: [resolved] Generate tdelimitedfiles with counters

#4 2011-12-29 02:15:34

Re: [resolved] Generate tdelimitedfiles with counters

Code:

#5 2011-12-29 15:55:33

Re: [resolved] Generate tdelimitedfiles with counters

#6 2011-12-30 03:13:00

Re: [resolved] Generate tdelimitedfiles with counters

#7 2011-12-31 01:59:40

Re: [resolved] Generate tdelimitedfiles with counters

#8 2011-12-31 02:42:49

Re: [resolved] Generate tdelimitedfiles with counters

#9 2012-01-02 21:23:10

Re: [resolved] Generate tdelimitedfiles with counters

#10 2012-01-05 10:35:45

Re: [resolved] Generate tdelimitedfiles with counters

#11 2012-01-20 23:57:46

Re: [resolved] Generate tdelimitedfiles with counters

#12 2012-01-21 02:11:47

Re: [resolved] Generate tdelimitedfiles with counters

Board footer