• Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » [resolved] XML -> FTP -> Oracle

#1 2010-03-12 15:37:30

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

[resolved] XML -> FTP -> Oracle

Tags: [database, ftp, oracle, xml]

Hi Everybody,

I want to do sth about FTP on TOS.  I have a common subdirectory under the root directory on my FTP Server. There are a lot of XML files (same formaatted). The naming convention has a rule like ID-DDMMYYYY.xml I want to get this file and read each record in it. Also need the properties of file name. Then want to save all these info. to my Oracle DB Table. Can you help me about the roadmap that I should follow. For example which components and which order. TOS is too flexible that I lost myself inside the forum posts.

Best Regards,
Mevlut

Offline

#2 2010-03-15 03:59:39

shong
Talend team
Registered: 2007-08-29
Posts: 10357
Website

Re: [resolved] XML -> FTP -> Oracle

Hello
First, use tFTPGET to get all the xml file into a folder, then use tFileList to iterate each file and extract the records from the file, merge all the records and insert them into DB. The job looks like:
tFtpGet
   |
onsubjobok
   |
tFileList---iterate---tFileProperties-main--->tJavaRow--oncomponentok-->tFileInputXML--main-tMap---main---tOracleOutput

tJavaRow: get the file properties and store them into global vars, eg: globalMap.put("basename",input_row.basename);
tMap: define some new columns for file properties, for exmaple, add a new column call basename and set the expression as: (String)globalMap.get("basename")
Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#3 2010-03-15 07:28:27

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Hi shong,
I will try and back to here with my results and feedbacks...
Thank you very much...
Mevlut

Offline

#4 2010-03-15 08:52:02

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Hi again,
Q.1) tFileList? tFTPFileList?
Q.2) There is no "main" opiton from both tFileList & tFTPFileList. There is only iterate. So ?
Thanks...

Offline

#5 2010-03-15 09:47:02

shong
Talend team
Registered: 2007-08-29
Posts: 10357
Website

Re: [resolved] XML -> FTP -> Oracle

Hello
Q.1) tFileList? tFTPFileList?
==>tFileList
Q.2) There is no "main" opiton from both tFileList & tFTPFileList. There is only iterate. So ?
==>Yes, it is not a input/output component, so thare is not a 'main' connector, please see the user documentaion and learn its usage.

Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#6 2010-03-15 09:50:08

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

You said "tFileProperties-main--->tJavaRow"  :-))))
Thank you very much for responses...
Mevlut.

Offline

#7 2010-03-15 16:37:42

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Hi againn,

I tried what you said in order (as you suggeted) but havent succeded yet :-))
I have lots of xml files. I declared 4 context variables because of basename of my files have a format. I split them in cod pane of tJavaRow component and assign them to context variables.
Then I use the "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))" expression where the file needed and it works. untill the tFileInputXML component.

My XML :

<?xml version="1.0" encoding="UTF-8"?>
<stats>
    <stat>
        <key>002</key>
        <value>12032019</value>
    </stat>
    <stat>
        <key>003</key>
        <value>12032010</value>
    </stat>
</stats>

And my output table (BKS_STAT)  has columns (ID,KEY,VALUE)

Q.1) How can I use tFileInputXML component for the xml files that have same format but that will be apparent on run-time ?
Q.2) Also use tMap usage is not clear without knowing the answer of the Q.1 !!!
Q.2) Can I use Sequence (in Oracle) for ID column ?
Q.3) is sing Context for split basename into 4 pieces (I will use them to insert BKS_STAT table with the ID, KEY, VALUE) sensible ?

Thank you very muck for your responses,
Best regards,
Mevlut.

Last edited by mbecerikli (2010-03-15 16:39:16)

Offline

#8 2010-03-17 09:22:49

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Can Anybody (cantoine, esabot, shong) Help Me ???

Last edited by mbecerikli (2010-03-17 10:56:20)

Offline

#9 2010-03-18 09:33:11

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

NO COMMENT ???

Offline

#10 2010-03-18 09:55:37

shong
Talend team
Registered: 2007-08-29
Posts: 10357
Website

Re: [resolved] XML -> FTP -> Oracle

Hello
Sorry for delay!

Q.1) How can I use tFileInputXML component for the xml files that have same format but that will be apparent on run-time ?

As a newbie, it is better to create the xml metadata follows the wizards. In this case, the basic setting of tFileInputXml looks like as my screenshot.

Q.2) Can I use Sequence (in Oracle) for ID column ?

Of course, see [Forum, topic 1488] Oracle sequence while insert.

Q.3) is sing Context for split basename into 4 pieces (I will use them to insert BKS_STAT table with the ID, KEY, VALUE) sensible ?

Can you take an example to explain your request?

Best regards
Shong


Uploaded Images


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#11 2010-03-18 10:32:03

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

I am sorry for persistant demand Because of being placed in difficult circumstances! But I persuaded my manager to use TOS after a long and hard demo. But I achieved that we took a full liecence to use TOS    :-)
My Main Goal is to take data from our customers via XML. The put put their XML files,whose basenames are formatted (XXX-Cust_ID-YEAR-TERM-SYSDATE.xml) special to them , in a common FTP folder.
I will take these files from FTP and copy to my locale and then insert (second phase has also UPDATE but not now) datas into Oracle with both splitting basename and xml nodes together.
Now I follow the order you suggested to me :  [ tFTPGet -->(iterate)--> tFileList -->(iterate)--> tFileProperties -->(main)--> tJavaRow ] -->(OnComponentOk)--> [ tFileInputXML -->(Main)--> tMap -->(Main)--> tOracleOuptput ]
I got the files to my locale folder. Get the property of each file and parse the info by splitting basename into Context variables in tJavaRow. But at this point ı stopped. My files not predefined, they will be live at run-time. So I have to set properties of tFileInputXML &  tMap manually.

For tFileInputXML;,
--------------------------
filename => "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))"
LoopXPathQuery => "/stats/stat"
Schema => ???
Mapping => ???
--------------------------

And My Xml structure:
--------------------------
<?xml version="1.0" encoding="UTF-8"?>
<stats>
    <stat>
        <key>002</key>
        <value>12032019</value>
    </stat>
    <stat>
        <key>003</key>
        <value>12032010</value>
    </stat>
</stats>
--------------------------
What should I do about the tFileInputXML (firstly) properties?

Thanks for responses again,
Mevlut.

Offline

#12 2010-03-18 10:39:16

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Shong,
Now I' ve seen your Screenshot. I m trying now...
Thanks...

Offline

#13 2010-03-18 10:44:42

shong
Talend team
Registered: 2007-08-29
Posts: 10357
Website

Re: [resolved] XML -> FTP -> Oracle

Hello

What should I do about the tFileInputXML (firstly) properties?

Did you see my screenshot? I have showed how to define the schema, mapping based on your xml file.
As I said, you have to learn how to create xml metadata and how to use the metadata on tFileInputXML component, just create a simple job to learn how to extract records from a xml file and output it to console. for example:
1)Go to Repository-->Metadata and right click on File xml, select 'create file xml' to create a xml metadata follows the wizards.
2)Create a simple job,
tFileInputXML--main--tLogRow

on tFileInputXML, choose 'repository' on property type list and choose the xml metadata just created by you.

Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#14 2010-03-18 12:59:10

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

Shong,

Probably you, first wrote than attached the screenshot so I'd written my message before I saw your screenshot.
I did what you'd said and it worked. I m really appreciated for the help. Thank you very much.
I' m stronger with my TOS from now on!  :-))))
Now It' s time to improve the functionality of my job!  New task is the control of the element values of xml file;

1-) Check Whether the value null
2-) Check  the existence of values in the dimension tables (For instance; my output table is "orders" and I will check the the existence of the "customer_id" column from "customer" table)

I will collect the all results and then send mail them to a mail address. And then die and rollback the transaction for that file(!)
If it is sensible, How can I achieve this (Maybe tLogCatcher, tAssert.....etc.) ?

Thanks again !!!
Best regards
Mevlut

Last edited by mbecerikli (2010-03-18 13:12:01)

Offline

#15 2010-03-19 06:58:35

shong
Talend team
Registered: 2007-08-29
Posts: 10357
Website

Re: [resolved] XML -> FTP -> Oracle

Hello

1-) Check Whether the value null

Yes, you can check the value if it is null and handle the null value on tMap, for example:
tFileInputXML--row1-->tMap--tMysqlOutput
row1.columnName==null?value1:value2
There are many such case on forum, pleasr search and learn it.

2-) Check  the existence of values in the dimension tables (For instance; my output table is "orders" and I will check the the existence of the "customer_id" column from "customer" table)

Select the 'insert or update' item on the action on data list of txxxOutput component. Maybe you need do some inner join between order table and customer table.

I will collect the all results and then send mail them to a mail address. And then die and rollback the transaction for that file(!)
If it is sensible, How can I achieve this (Maybe tLogCatcher, tAssert.....etc.) ?

Please see the user documentaion and learn the following component:
tMysqlConnection, tMysqlCommite, tDie, tSendeMail.

Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#16 2010-03-19 07:27:45

mbecerikli
Member
Registered: 2010-02-04
Posts: 39

Re: [resolved] XML -> FTP -> Oracle

OK! I got how to use expression editor (java) and use join operations. And with tDie, I stop the job which is running. And send mail with tSendMail.
But I could not get how to collect error messages which I got from tMap (row1.columnName==null?value1:value2). I can store them in a context variable.
But How can I fire to put them in the expression editor in tMap. Checking with this and Join Operations, I just filters the data (?).
Be sure I am reading the Reference and User Guides and searching your on the internet. But I am new to TOS and trying to improve as fast as I can
I'm sorry by taking your valuable time :-(
Thanks,
Mevlut.

Offline

  • Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » [resolved] XML -> FTP -> Oracle

Board footer

Powered by FluxBB