You are not logged in.
Go to Windows-->Preferences-->Talend-->Run/Debug and change the JVM arguments.
I think the JDBC component load all results into memory in place of load packet by packet.
Since TOS 2.4 RC we can set a cursor to limit data load with tOracleInput and tPostresqlInput. I hope this will help you.
Try to post more precision on the error if you have, such as stack trace with line number and location of error.
postgresql (version 1.8.2) and I use the driver postgresql-8.2-508.jdbc3.jar
Just to be sure : If I understand all, I can't load my data packet by packet in TOS 2.3.3 and it's possible in TOS 2.4 RC with tOracleInput and tPostresqlInput. Is it correct ?
Can I expect you will make the same thing with tJDBCInput in future version ?
Because, I think it's very annoying for an ETL if it can't read several millions of lines in a database.
For me your request is not logical.
tJDBC is a generic component so we can't implement a driver specific feature like the cursor.
The correct request is : can you use the last official driver in the postgres component
And this is what I have reported in [Bugtracker, bug 4051, fixed] Update PostgresSQL driver
In TOS 2.3.3, I tried tJDBCInput and tPostgresqlInput to read 5 000 000 of lines and write them in a tDelimitedFile. In both cases, the job fails (outOfMemory : Java heap space). My JVM arguments were -Xms256m -Xmx2048m.
I can't try all txxxInput so I would like to know if TOS can read 5 000 000 lines in a database ?
thanks for your answer.
I tried the same thing, and I have the same problem. So I asked me if I am alone in this case or not ? And if I did a wrong thing in my job ? :-)
I conclude TOS in not capable to read around more than 2 500 000 lines. I think TIS can do that but I don't use TIS so... Somebody has the answer ?
normally the cursor is here for this kind of thing.
i think it's a BUG not a limitation,
talend team, can you confirm ?
Last edited by chris doudet (2008-06-17 12:42:41)
In 2.4.0, you have no more reason to use a tJDBC since the postgres jdbc driver have been updated in [Bugtracker, bug 4051, fixed] Update PostgresSQL driver
Thats why you should be able to use tPostGresInput with cursor.
I have to make some new tests on the question asked by Chris Doudet : is the cursor really working in 2.4 with Postgres.
I come back to you.
I'm working with TOS 3.0.0 and i'm experiencing the same problem : the cursor option in component tPostgresIntput seems not to work (i got a lack of memory error and changing the cursor size seems not chnaging anything)
Do you have some news about that problem ? Did the latest release (3.0.2) solve the problem ?
Hi, I get that error when I run my job:
Could not create the Java virtual machine
Error ocurred during initialization of VM
Could not reserve enough space for object heap
However, if I export that job to a .bat file, edit its contents and I remove the -Xms256M and -Xmx1024M options it runs right!
What happens if I remove those options? I'm fed up of trying several values for -Xms and -Xmx and none make it work. It just works if I remove these options.
PS: My job is very simple! Just a tFixedFlow and a tFileOutputDelimited
Last edited by Janick (2010-03-11 17:24:30)
In your case, it is not even able to initialize the JVM using the Xms parameter. Make sure you are entering the correct syntax for those parameters, and that you have enough space to be reserved for the smallest threshold. (Keep in mind that the syntax is case-sensitive).
Here is an interesting link showing six common mistakes in setting the java heap space:
http://javahowto.blogspot.com/2006/06/6 … -heap.html