Thanks for your reply.  I want to just store the line number of last read row instead of complete data from each file .

Example :
log file1
rows were added to this file.

log file2
rows were added to this file.

This ETL Job has to keep track of which files got changed from the previous run.
I'm using tfileList (Iterate) ----> tFileProperties. From tFileProperties, I get information like the file name, file size, modified time. I keep track of all these information in a database (table). so each time, when this ETL Job runs, it compares the new value from tFileProperties with the one in the table on modified time.

but now for each file, I have to keep track of  the last line number read from the file and also store this value in a table,
so next time when this ETL Job runs it has to read rows from  each of this log files starting from that line number onwards.

is this possible. Please let me know, if I'm not clear.

Thanks for all your help!!!

Last edited by doradora (2012-04-17 02:17:00)

Thanks Pedro. But doesn't help me much.

I have many files in a directory and any of those files might get new rows added to them.
I have to loop thru all the files in that directory and for each file, keep track of the last row read.

and hence in my previous example, I had 2 files and both the files got updated with new rows.
so in this case tFixedFlowInput  won't help, as it  will have only the last files information.

Please let me know, if I'm not clear. I can have a skype meeting with you and can show you the talend Job (what I have put together so far). 


Thanks.

Last edited by doradora (2012-04-17 18:37:42)

Thanks pedro so much for your prompt response and help.

I need one more favor from you. we have a directory structure like   /dw/logs/2012/04/01
The directory structure is in this order /dw/logs/year/month/days

I have to loop thru previous day as well as today's directory.

example.  I have to loop thru /dw/logs/2012/04/17 and /dw/logs/2012/04/18.

how to create context variables  with such values and use those context variables in tfilelist and also how to iterate  thru the tfilelist.


can you please give me some examples.

Thanks

Thanks pedro so much.  It worked perfectly fine. Thanks for helping me. I have been working on talend only for the past 4 weeks and hence slowly learning  things.

Hi Pedro,

I need your help again.

This is what I'm trying to do :
1) database (select  file name where the status="Available")
2) goes to tFlowToIterate
3) tFileInputdelimited (read each file from tFlowToIterate).
4) tMap (it does 2 things. It inserts into a staging table  and also update a fileStats table).

The issue, I'm having is with the tMap, I want to update the fileStats table only when the file changes.

Right now what it does is, suppose I have 860,000 millions in the  each of the tFileInputdelimited file.
then for each row it updates the fileStats table and that is having a huge performance degradation.

what I want to do is update this fileStats table only once for each file i.e whenever the file changes, store the previous file name value in some variable and then update this table.


Please let me know, if I'm not clear.  I can explain or show what I'm trying to do over skype, so you can get a better picture.

Thanks

Thanks Pedro. Sorry I  missed some information. In the tMap I wrote a java function very similar to numeric.sequence to compute the row line number. I want to update the fileStats table with the row line number whenever the file name changes.

suppose for example, I have 10 files in a directory. First I load all the 10 files information like file path, file name, modification time to a table.

Then I do as below:

This is what I'm trying to do :
1) database (select  file name where the status="Available")
2) goes to tFlowToIterate
3) tFileInputdelimited (read each file from tFlowToIterate).
4) tMap (it does 2 things. It inserts into a staging table  and also update a fileStats table).

The issue, I'm having is with the tMap. I want to update the fileStats table (with the row line number)only when the file changes.

Right now what it does is, suppose I have 860,000 millions in the  each of the tFileInputdelimited file.
then for each row it updates the fileStats table and that is having a huge performance degradation.

what I want to do is update this fileStats table only once for each file i.e whenever the file changes, store the previous file name value in some variable and then update this table.


Please let me know, if I'm not clear.  I can explain or show what I'm trying to do over skype, so you can get a better picture.

Thanks

To this new java routine, I pass the file name and the line number( from where I have to read the file) and then for each row read, it increments the context.variable(line number)by 1.

Last edited by doradora (2012-04-24 05:41:50)

I have this new java routine in the tMap. The input to the tMap is the tfileinputdelimited and there are 2 output, one to insert into a staging table the data from the tFileinputdelimited and the other one is to update to the fileStats table.
Right now it is updating the fileStats table for each row from the tFileInputdelimited, which is causing  huge performance issue. what I want is to update the fileStats only one for each file with the correct line number counter.

so in the tmap, I don't know how to do it.  I need to conditionally update the table with the line number counter when the file name changes.

suppose for eg:

I have file1, file2, file3...

I want to update the fileStats table with the line number counter for each file only once and not for each row read from the tfileinputdelimited.