You are not logged in.
I am having trouble aggregating more than 800K rows. I am running on Windows XP 32-bit with 3 GB RAM, TIS 4.0.2
I have a table with 1.5M records which tAggregateRow is going through. After extracting about 900K rows I am getting Java heap error and execution aborts.
Runnign the same job on 64-bit Windows 7 with 4 GB resolves the problem but I have to make it work on 32-bit system.
As a workaround I splitted the table in two halves with around 750K rows each. tAggregateRow works fine in this scenario.
Are there any performance settings I can tune to resolve the problem?
Last edited by Peter (2010-10-07 23:19:40)
You have some workaround ways to make it happen with your 750K rows.
If the data is already into a Database, you can use the component tELTAgregate; it would generate the SQL and the aggregate is performed by the Database itself without this memory limitation. Otherwise you have to first push the Data into one database and then do you Aggregation. For the load you can use BULK_EXEC mechanism to accelerate the Load.
One of the other solution in into our Talend Integration Suite MPX edition which involved a component called : tFSAggregate to perform aggregation on large files.