Talend community forum / tUniq Out of memory error

dfarren · 2009-11-09 22:45:20

Hello.

I created a simple job that gets the data from a CSV file (230 million rows), sends the data thru component, then a tmap component and then inserts it into a mysql database.

Since the Job stops because it gets out of memory I decided to use disk to store temp data of tuniq. The problem is that the job is still having memory problems:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

My configuration is:
-Xms512
-Xmx1024

Working on a Xeon CPU@2.66Ghz with 1.7 Gb Ram.

Can my machine work with such a big database?

Job image attached.

Thanks,
Derek

Uploaded Images

shong · 2009-11-10 04:05:24

Hello
1)Allocates more memory to TOS if it is possible.
2)Aslo check the option' Only once each duplicated key' on tUniqRow.

Best regards

shong

dfarren · 2009-11-10 21:58:14

shong wrote:
Hello
1)Allocates more memory to TOS if it is possible.
2)Aslo check the option' Only once each duplicated key' on tUniqRow.

Best regards

shong

Did that, and still not working. My new memory configuration is -Xms1024, -Xmx1024.

Just to understand, why does talend use that much memory if I am storing the temp files on disk?

Thanks,
Derek

Last edited by dfarren (2009-11-10 21:58:54)

Announcement

Unanswered posts

#1 2009-11-09 22:45:20

tUniq Out of memory error

#2 2009-11-10 04:05:24

Re: tUniq Out of memory error

#3 2009-11-10 21:58:14

Re: tUniq Out of memory error

shong wrote:

Board footer