• Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » [resolved] Dynamic setting of fileName for tFileOutputXX

#1 2009-09-02 11:36:08

clems787
New member
Registered: 2009-08-28
Posts: 7

[resolved] Dynamic setting of fileName for tFileOutputXX

Hi,

I'm current working on a job that has to write in different files. The number of files is defined at runtime, as well as the name of each file.

I have access to the list of all files in an entry of the globalMap.

My first thought was to iterate on this list in a tJavaFlex component, store the current filename in the globalMap, and use this globalMap entry in the tFileOutputXX filename field.

The problem is that this filename value is setup only once, at initialization of the subjob, and not for each row going out of the tJavaFlex.

As a result, all my rows goes to a "null.txt" file ...

I'm waiting for your ideas to solve this problem !


Uploaded Images

Last edited by clems787 (2009-09-02 11:36:43)

Offline

#2 2009-09-03 08:32:34

shong
Talend team
Registered: 2007-08-29
Posts: 10305
Website

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Hello
Can you upload a screenshot of your job and show us more info about your job? You put the 'for' in the start code part of tJavaFlex, and this part code will be only executed one time.

Best regards

          shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#3 2009-09-03 16:32:13

clems787
New member
Registered: 2009-08-28
Posts: 7

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Here is a screenshot of the part of my job causing the problem.

The choice of my components might not be the good one ...


Uploaded Images

Offline

#4 2009-09-03 17:38:23

rbaldwin
Member
Registered: 2008-11-24
Posts: 126

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

In order to do what you're trying to do, you'll need to use a tFlowToIterate.  I've attached screenshots that show how to do this.  One shows the whole job, then each of the major components settings are shown, and the results of the job's execution is shown.  The tFlowToIterate component is the important one, because it will change how the job attempts to create the file - without it, it's at flow initialization time, which is before your tJavaFlex's start block is executed.  The tFixedFlowInput is simply there to pull the flow back into a row that goes to the tFileOutputDelimited.  In addition to the tJavaFlex to tFlowToIterate, you could also, presumably use a tForeach to loop through your list or a tLoop component.

Hope that helps.


Uploaded Images

Offline

#5 2009-09-03 17:58:12

clems787
New member
Registered: 2009-08-28
Posts: 7

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Ok, i think we are getting close to what i want but there is still some problems.

I tried your solution, but i dont get "one", "two", "three", "four" created, but "null", "one", "two", "three" ....

Any idea ?

Offline

#6 2009-09-03 19:02:58

rbaldwin
Member
Registered: 2008-11-24
Posts: 126

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Check the tFileOutput component and insure that you're using the row from the tJavaFlex.  In the screenshot I posted above, notice how the tFileOutputDelimited has row1.fileName as the variable that it's using - the whole value for the file name is:
"C:/TDQ/3.1.3/TDQ_EE-All-r26090-V3.1.3/workspace/examples/" + row1.fileName + ".txt"

If you're using the row number from the tFixedFlowInput - in my screenshot this is row2, so row2.fileName - it will be one value behind what you're looking for, which would explain the "null","one","two","three" that you're seeing.

Offline

#7 2009-09-04 09:26:16

clems787
New member
Registered: 2009-08-28
Posts: 7

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

You're right, it's working very well now !

Thanks for your help !

Just one more little thing, your mentioned that i could iterate on a list with tForeach of tLoop.
I don't see how i could pass a dynamic list (stored in the globalMap) in these components.

Can you give me some directions for this ?

Last edited by clems787 (2009-09-04 09:28:51)

Offline

#8 2009-09-04 15:57:13

rbaldwin
Member
Registered: 2008-11-24
Posts: 126

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

The screenshots attached to this post show how to use a tLoop to iterate through list elements.  In the tJava_2 component, I initialize a list and then place it into the globalMap.  I then configure the tLoop component to iterate through the elements in the list.  Finally the tJava_3 component prints the values I pull from the list with a System.out.println().

Hope that helps.


Uploaded Images

Offline

#9 2009-11-11 16:16:48

rlmagidson
Member
Registered: 2009-11-10
Posts: 13

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Hello rbaldwin and others...

I have a similar task to clems787 in that I need to create multiple CSV files whose name and content depends on a column value from a SQL Server table. There are about 50 different values for this column (ShopId) that will result in about 50 output files. The basic requirement restated: read the input table and write some of the columns out to a file whose name is "Shop" + ShopId + ".csv." There may be multiple rows written out, so I'll be appending to the CSV if it exists.

I've tried to use globalMap and context variables based on other solutions and tutorials, but since I'm not a Java programmer by trade I don't yet "get" what they are and realize that I'm pretty much shooting in the dark.

If you (or anyone) can help teach me, I'd certainly appreciate it!

Thanks in advance for any assistance
- Russell

Offline

#10 2011-12-09 06:19:47

Ankit
New member
Registered: 2011-12-05
Posts: 6

Re: [resolved] Dynamic setting of fileName for tFileOutputXX

Hi guys...
I also have related issue..
In my job i am reading data from CSV file..and  on the basis of first column in every row i create update the GlobalMap, and use map value to set files name..(because i want to create multiple files from CSV)..for do so:
I write this path:
"D:/Talend/TOS-Win32-r67267-V4.2.3/NEM/"+((String)globalMap.get("NetName")+"/"+((String)globalMap.get("NetName"))+".csv"

Here "NEtName" is key..
But it create one file named as "null.csv" instead of multiple files with different names and in different directory under "NEM" directory..
My flow is:

Delimitedfile------>toSetGlobalVar------------->tJavaRow------->toFileOutputDelimited


i also System.out.println(); to track vaues using tJavaRow here correct values are coming..


Thanks

Last edited by Ankit (2011-12-09 07:33:47)

Offline

  • Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » [resolved] Dynamic setting of fileName for tFileOutputXX

Board footer

Powered by FluxBB