i'm trying to transform a file of 5 millions records on a batch developped with talend on an aix server. it takes 2h30 to parse and tranform a fil postionnal to an other positional file , using tmap component. looking at the perl code generated ,
i saw that the component reads , processes,writes record by record. can we speed up this processing by having a cache for input and output file component .
Last summer, I made some tests with a write buffer, ie writting to the output only once the buffer of 1,000 (for example) lines was full. It made no variation on the total execution time.
What kind of transformations do you make in the tMap?
How do you configure your tFileOutputPositionnal?
Note: please, avoid topic post made only of capital letters, it means you're shouting.
I've tried also to cache writing out every 1K records, I've gained, few minutes on 2h30 hours. Maybe, I' should try to cache also file input component at the same time.
Thanks for the clues with sprintf, I'l try it also.
ps : sorry for the whole post in capital letters.