You are not logged in.

#1 2007-04-21 22:34:25

anhhai
Member
7 posts

performance issue

Tags: [performance, tos]

hie,

i'm trying to transform a file of 5 millions records on a batch developped with talend on an aix server. it takes 2h30 to parse and tranform a fil postionnal to an other positional  file , using tmap component. looking at the perl code generated ,

i saw that the component reads , processes,writes  record by record. can we speed up this processing by having a cache for input and output  file component .

regards.

Offline

#2 2007-04-22 12:07:19

plegall
Member
1586 posts

Re: performance issue

Last summer, I made some tests with a write buffer, ie writting to the output only once the buffer of 1,000 (for example) lines was full. It made no variation on the total execution time.

What kind of transformations do you make in the tMap?
How do you configure your tFileOutputPositionnal?
(screenshot welcomed)

Note: please, avoid topic post made only of capital letters, it means you're shouting.

Offline

#3 2007-04-23 11:22:54

plegall
Member
1586 posts

Re: performance issue

I've made some tests with printf instead of the Perl formatString dedicated function currently used. printf is  more than 7 times faster but is less powerful.

Offline

#4 2007-04-24 00:01:47

anhhai
Member
7 posts

Re: performance issue

I've tried also to cache writing out every 1K records, I've gained, few minutes on 2h30 hours. Maybe, I' should try to cache also file input component at the same time.
Thanks for the clues with sprintf, I'l try it also.


ps : sorry for the whole post in capital letters.

Offline

Board footer