You are not logged in.
Announcement
Unanswered posts
|
Pages: 1
- Index
- » Talend Open Studio for Data Integration » Usage, Operation
- » Validate file format and if it is correct upload format database
#1 2011-12-16 11:07:20
- oscarmsanz
- Member
- Registered: 1970-01-01
- Posts: 78
Validate file format and if it is correct upload format database
Hello
First of all I'm using version 5 of Talend opend studio
I have to be able to somehow validate the format of a file. If all records are correct file I then load into the registry database and if not do nothing. Most of the files I have to deal with more than 60,000 records.
I've seen that both the component and tschemaCompliancecheck fileinputdelimited to check the structure of the records will result in two outputs. Erroneous and correct row. From here no how.
Step one example I have an echo in the end only file where I have a record to give you an idea of what I've seen so far.
I am interested to do something general indiqueis when I have one or more records in the file.
Regards,
Oscar Manso
Last edited by oscarmsanz (2011-12-16 11:22:23)
Offline
#2 2011-12-16 11:43:52
- pedro
- Member
- Registered: 2011-11-17
- Posts: 3682
Re: Validate file format and if it is correct upload format database
Hi Oscar Manso
You may create a job as the following images.
You said if all records are correct, you will load the file into database.
So I declare a context variable here and use a component tJava to set the value of this variable.
Then I use a run-if trigger to judge whether you load this file into database.
Hope this will help you.
Best regards!
Pedro
Last edited by pedro (2011-12-16 13:03:31)
Only Paranoid Survive.
Offline
#3 2011-12-16 11:58:51
- oscarmsanz
- Member
- Registered: 1970-01-01
- Posts: 78
Re: Validate file format and if it is correct upload format database
Hello Peter
The problem I have that are very large files that I have to load and use thashinput thashoutput and I think it is the best solution,
The final problem is what is saved in memory and from experience with other tools ELTL can get to not having enough memory on the machine when it comes to keeping records,
Therefore need a better alternative if available.
Your solution and thought.
Regards,
Oscar Manso Sanz
Offline
#4 2011-12-16 13:06:41
- pedro
- Member
- Registered: 2011-11-17
- Posts: 3682
Re: Validate file format and if it is correct upload format database
Hi Oscar
You are right. I hava corrected it and make a better workaround.
This time i just load the input file twice.
Hope someone can offer perfect workaround.
Best regards!
Pedro
Only Paranoid Survive.
Offline
#5 2011-12-16 13:19:20
- oscarmsanz
- Member
- Registered: 1970-01-01
- Posts: 78
Re: Validate file format and if it is correct upload format database
Anyone who knows a better solution than say so, because by using file with a large volume of data does not seem correpto unsar thashinput, because it can spilling the precious Blood thashouput computer memory
I know at other ETL tools like Oracle Wharehouse Builder I know for having cast a course, when you load a file only indicates the following.
File loading and file where the records are not loaded.
When you load the file is loaded as it is and if there is a mistake and want to download these data are usually delete these records database and a new file reloads.
I work for the client has asked me not to do this. Otherwise if the file has an erroneous structura not load it and while it loads well. Then if you get an error when doing insert into database will abort the entire file but that's another story.
Regards,
Oscar Manso
Last edited by oscarmsanz (2011-12-16 13:23:08)
Offline
Pages: 1
- Index
- » Talend Open Studio for Data Integration » Usage, Operation
- » Validate file format and if it is correct upload format database