Talend community forum / Iterating through files

lshannon · 2012-03-27 17:27:17

I have a job right now that loads three CSV files into a DB and then run a series of jobs against that to build out a data warehouse. In the future there may be as many as 5 or 10.

Right now I have a separate job running for each CSV file. What I would like is a single job that goes through a folder running on each CSV that it finds. Does someone have a sample on this?

Thanks,

Luke

BRM · 2012-03-27 20:37:50

The component you want is tFileList. In that component you can specify a file mask and hook the iterate output to the rest of your processing.

lshannon · 2012-04-08 21:33:29

Hi, does anyone know if there is a way I can pass the output from tFileList into a tRunJob component?

lshannon · 2012-04-08 21:53:02

Ah, looks like you can add a context variable in the job, then set it as a parameter to the tRunJob and then pass this through an iteration between the tFileList and tRunJob:

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

geuder · 2012-05-04 11:47:54

I've set up a cron job that pulls AppStore Sales reports to a folder,
which I then let a reoccurring Talend job:
looking in, pulling out files, extracting, loading into a staging table, moves the used files.

-- Update --
Hmm, can't see the files I uploaded...

Last edited by geuder (2012-05-04 11:50:43)

Announcement

Unanswered posts

#1 2012-03-27 17:27:17

Iterating through files

#2 2012-03-27 20:37:50

Re: Iterating through files

#3 2012-04-08 21:33:29

Re: Iterating through files

#4 2012-04-08 21:53:02

Re: Iterating through files

#5 2012-05-04 11:47:54

Re: Iterating through files

Board footer