• Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » Check file and upload file structure in database

#1 2012-01-09 15:32:27

oscarmsanz
Member
Registered: 2011-12-06
Posts: 78

Check file and upload file structure in database

hi

I have to validate the data of a tab-delimited files. If all is well in the office database. What I have to do is

1 - Validate if the correct number of fields in each record

2 - Validate the length of the field

3 - Validate that if they meet a decimal formats.

4 - Validate that certain fields have values ​​that are in database tables

5 - If no date fields have to satisfy a pattern

6 - Report of the errors of each record

I can not use tschemaconpliancecheck and delimited file option to verify the structure of each row against the schema

Assuming that the field delimiter is the tab end of the file I have a number of lines that are not fixed at the end as

-->-->-->-->-->
-> ->
->
->->->->->->

I can come sometimes 3 lines, other 4 and other n.

I also want to know how to load these files when satisfied that everything is correct

Regards,
Oscar Manso

Offline

#2 2012-01-10 10:45:01

oscarmsanz
Member
Registered: 2011-12-06
Posts: 78

Re: Check file and upload file structure in database

Hi

Can anyone answer me


Regards,
Oscar Manso

Offline

#3 2012-01-10 12:46:47

janhess
Member
Company: Newcastle University
Registered: 2009-05-19
Posts: 1137

Re: Check file and upload file structure in database

Why can't you use tschemaconpliancecheck and delimited file option?

Offline

#4 2012-01-10 12:57:33

vital_parsley
Member
Registered: 2011-12-08
Posts: 171

Re: Check file and upload file structure in database

OK...
1) Validate if the correct number of fields in each record
This can be done using tFileInputDelimited itself : In tFileInputDelimited --> Advanced Settings check the Check each row against schema.This will check if the row structure matches the defined structure for the component.If there is an error you have use Reject link of tFileInputDeimited to get the error row it also give the reject reason !!!

2 - Validate the length of the field
I think tFileInputDelimited wont check the length of the field against your defined structure.So for that link your tFileDelimited to tSchemaComplianceCheck component.The tSchemaComplianceCheck will check the length if it matches...Else just like tFileInputDelimited there is a Reject link for tSchemaComplianceCheck which will give you ejected records which exceed length and also the reason is shown.

3 - Validate that if they meet a decimal formats.
What kind of format is expected >>>> ?


4 - Validate that certain fields have values ​​that are in database tables
For this you will have to do an inner join with a tMap between your source file and the Database(tOracleInput) .Here you can simply forward rows which match.(Optionally : you can also catch the inner join rejected rows,which in your case does not seem applicable).

5 - If no date fields have to satisfy a pattern
This again can be done using tFileInputDelimited where you mention your pattern in the Schema in the Date pattern column e.g dd-MM-yyyy


6 - Report of the errors of each record
Each rejected record can be taken as i said in a Rejected links component(e.g tLogRow)

Offline

#5 2012-01-10 13:23:21

oscarmsanz
Member
Registered: 2011-12-06
Posts: 78

Re: Check file and upload file structure in database

hi

Although not validate the fields and the file is good in principle, how do I load it into database? Remember that the file you end up with a variable number of lines

Regards,
Oscar Manso

Offline

#6 2012-01-10 13:28:59

vital_parsley
Member
Registered: 2011-12-08
Posts: 171

Re: Check file and upload file structure in database

tOracleOutput component to load it into a Oracle DB

Last edited by vital_parsley (2012-01-10 13:30:05)

Offline

#7 2012-01-10 16:07:06

vital_parsley
Member
Registered: 2011-12-08
Posts: 171

Re: Check file and upload file structure in database

If u want to check the value of decimal then you can match it to a particular format using java`s regex in Filterrow component and also get the rejected rows .

Offline

#8 2012-01-10 16:13:04

oscarmsanz
Member
Registered: 2011-12-06
Posts: 78

Re: Check file and upload file structure in database

hi

Forget validations and tell me how to read a file that ends with a variable number of lines like the following to load it into database. I must load the entire file or nothing. These control lines are used but they are never fixed and my file would be correct.

-->-->-->-->-->
-> ->
->
->->->->->->

Regards,
Oscar Manso Sanz

Offline

#9 2012-01-10 16:18:20

janhess
Member
Company: Newcastle University
Registered: 2009-05-19
Posts: 1137

Re: Check file and upload file structure in database

Replace them with ""

Offline

#10 2012-01-10 20:21:32

vital_parsley
Member
Registered: 2011-12-08
Posts: 171

Re: Check file and upload file structure in database

i m not able to understand what you are saying !
i think you can check your validations and then OnSubJobOk you can load them.

Offline

#11 2012-01-11 08:43:43

oscarmsanz
Member
Registered: 2011-12-06
Posts: 78

Re: Check file and upload file structure in database

i m not able to understand what you are saying with Replace them with ""

Offline

#12 2012-01-11 09:57:19

lijolawrance
Member
Registered: 2010-01-27
Posts: 364

Re: Check file and upload file structure in database

Hi

Can you show 2-3 sample rows in this thread for better understanding


Regards
Lijo Lawrance

Offline

  • Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » Check file and upload file structure in database

Board footer

Powered by FluxBB