Talend community forum / Java heap space, simple job!

Caro · 2008-06-02 16:27:27

My job is : tJDBCInput --> tFileDelimited
Ma database contains 5 000 000 records and I put following parameters : -Xms256m -Xmx2048m.

I have the error : outOfMemory : Java heap space. Why ?

Caro

shong · 2008-06-03 07:52:43

Hello

Go to Windows-->Preferences-->Talend-->Run/Debug and change the JVM arguments.

Best regards

shong

Caro · 2008-06-03 10:01:54

I already did that ! I put -Xmx2048M and it's the same thing. TOS is very greedy with memory!

Is it normal ?

Caro

amaumont · 2008-06-03 10:32:31

I think the JDBC component load all results into memory in place of load packet by packet.

Since TOS 2.4 RC we can set a cursor to limit data load with tOracleInput and tPostresqlInput. I hope this will help you.

Try to post more precision on the error if you have, such as stack trace with line number and location of error.

amaumont

Caro · 2008-06-03 11:18:49

Ok thanks a lot. I understand now why!
I hope future versions will have JDBC components with loading packet by packet. I think it will be better.

Regards,
Caro

mhirt · 2008-06-03 20:09:49

Caro,

This functionality is not available for all JDBC drivers.
What database are you connecting to ?

Regards,

Caro · 2008-06-04 09:54:55

postgresql (version 1.8.2) and I use the driver postgresql-8.2-508.jdbc3.jar

Just to be sure : If I understand all, I can't load my data packet by packet in TOS 2.3.3 and it's possible in TOS 2.4 RC with tOracleInput and tPostresqlInput. Is it correct ?
Can I expect you will make the same thing with tJDBCInput in future version ?
Because, I think it's very annoying for an ETL if it can't read several millions of lines in a database.

Thanks
Caro

mhirt · 2008-06-04 10:48:15

For me your request is not logical.
tJDBC is a generic component so we can't implement a driver specific feature like the cursor.

The correct request is : can you use the last official driver in the postgres component
And this is what I have reported in [Bugtracker, bug 4051, fixed] Update PostgresSQL driver

Regards,

Caro · 2008-06-04 12:40:42

Thanks for your answer.

In TOS 2.4 RC, I use a tPostresqlInput and I put the cursor at 100. Is it load my data by packets of 100 lines ?

Thanks,
Caro

Caro · 2008-06-04 14:04:38

Currently, TOS 2.4 RC doesn't use the last official driver in the postgres component ? So I can't try my simple job (with 5 000 000 lines) for the moment ?

Thanks for all your answer.

Caro

Caro · 2008-06-16 18:09:44

In TOS 2.3.3, I tried tJDBCInput and tPostgresqlInput to read 5 000 000 of lines and write them in a tDelimitedFile. In both cases, the job fails (outOfMemory : Java heap space). My JVM arguments were -Xms256m -Xmx2048m.

I can't try all txxxInput so I would like to know if TOS can read 5 000 000 lines in a database ?

Thanks

Caro

chris doudet · 2008-06-17 10:38:49

hi,
i use 2.4.0 and i try cursor (1000) on a 3 000 000 row : failed.
just a tpostgres to a tfileoutputdelimited.

Caro · 2008-06-17 10:45:44

thanks for your answer.
I tried the same thing, and I have the same problem. So I asked me if I am alone in this case or not ? And if I did a wrong thing in my job ? :-)
I conclude TOS in not capable to read around more than 2 500 000 lines. I think TIS can do that but I don't use TIS so... Somebody has the answer ?

thanks
Caro

chris doudet · 2008-06-17 12:42:24

hum,
normally the cursor is here for this kind of thing.
i think it's a BUG not a limitation,

talend team, can you confirm ?

Last edited by chris doudet (2008-06-17 12:42:41)

Caro · 2008-06-17 14:25:45

I think like you but if I must use a tJDBCIput, I can't read more than 2 500 000 lines, isn't it ?

mhirt · 2008-06-17 15:43:35

Caro,

In 2.4.0, you have no more reason to use a tJDBC since the postgres jdbc driver have been updated in [Bugtracker, bug 4051, fixed] Update PostgresSQL driver
Thats why you should be able to use tPostGresInput with cursor.

I have to make some new tests on the question asked by Chris Doudet : is the cursor really working in 2.4 with Postgres.
I come back to you.

Regards,

chris doudet · 2008-07-02 15:45:28

So.......
Is this cursor really work or not ?

chris doudet · 2008-07-11 10:19:47

up !
Still not working in 2.4.1

Last edited by chris doudet (2008-07-11 11:26:11)

chris doudet · 2008-07-25 11:29:33

up!
Still not working in 3.0.0M1

eguerin · 2008-10-01 17:06:29

Hi,

I've the same problem on Talend 2.3.3 and 2.4.1.
The "cursor limit" on a tPostgreSQLInput doesn't resolve the problem for me.
I try to select 12 millions rows but the jobs die by java heap space.

chris doudet · 2008-10-20 11:07:45

up !
still not working in 3.0.0

etletisme · 2008-12-04 15:07:33

Hi everybody,

I'm working with TOS 3.0.0 and i'm experiencing the same problem : the cursor option in component tPostgresIntput seems not to work (i got a lack of memory error and changing the cursor size seems not chnaging anything)

Do you have some news about that problem ? Did the latest release (3.0.2) solve the problem ?

Best regards

chris doudet · 2008-12-08 11:39:49

Hi all,

It works at last.

It was resolve in 2.4.4.

i don't know for the v3. To be tested

Janick · 2010-03-11 17:23:38

Hi, I get that error when I run my job:

Could not create the Java virtual machine
Error ocurred during initialization of VM
Could not reserve enough space for object heap

However, if I export that job to a .bat file, edit its contents and I remove the -Xms256M and -Xmx1024M options it runs right!

What happens if I remove those options? I'm fed up of trying several values for -Xms and -Xmx and none make it work. It just works if I remove these options.

Thank you

PS: My job is very simple! Just a tFixedFlow and a tFileOutputDelimited

Last edited by Janick (2010-03-11 17:24:30)

rdagher · 2010-03-11 17:52:48

Janick,

In your case, it is not even able to initialize the JVM using the Xms parameter. Make sure you are entering the correct syntax for those parameters, and that you have enough space to be reserved for the smallest threshold. (Keep in mind that the syntax is case-sensitive).

Here is an interesting link showing six common mistakes in setting the java heap space:
http://javahowto.blogspot.com/2006/06/6 … -heap.html

Regards,
Rabih

Announcement

Unanswered posts

#1 2008-06-02 16:27:27

Java heap space, simple job!

#2 2008-06-03 07:52:43

Re: Java heap space, simple job!

#3 2008-06-03 10:01:54

Re: Java heap space, simple job!

#4 2008-06-03 10:32:31

Re: Java heap space, simple job!

#5 2008-06-03 11:18:49

Re: Java heap space, simple job!

#6 2008-06-03 20:09:49

Re: Java heap space, simple job!

#7 2008-06-04 09:54:55

Re: Java heap space, simple job!

#8 2008-06-04 10:48:15

Re: Java heap space, simple job!

#9 2008-06-04 12:40:42

Re: Java heap space, simple job!

#10 2008-06-04 14:04:38

Re: Java heap space, simple job!

#11 2008-06-16 18:09:44

Re: Java heap space, simple job!

#12 2008-06-17 10:38:49

Re: Java heap space, simple job!

#13 2008-06-17 10:45:44

Re: Java heap space, simple job!

#14 2008-06-17 12:42:24

Re: Java heap space, simple job!

#15 2008-06-17 14:25:45

Re: Java heap space, simple job!

#16 2008-06-17 15:43:35

Re: Java heap space, simple job!

#17 2008-07-02 15:45:28

Re: Java heap space, simple job!

#18 2008-07-11 10:19:47

Re: Java heap space, simple job!

#19 2008-07-25 11:29:33

Re: Java heap space, simple job!

#20 2008-10-01 17:06:29

Re: Java heap space, simple job!

#21 2008-10-20 11:07:45

Re: Java heap space, simple job!

#22 2008-12-04 15:07:33

Re: Java heap space, simple job!

#23 2008-12-08 11:39:49

Re: Java heap space, simple job!

#24 2010-03-11 17:23:38

Re: Java heap space, simple job!

#25 2010-03-11 17:52:48

Re: Java heap space, simple job!

Board footer