Talend community forum / TOP on MAC OS X seems to be very instable - almost unusable

kejessen · 2010-09-13 12:24:52

I am considering using TOP, and have been trying out for a couple of days, but frankly I am about to give up: TOP seems to me to be very unstable and unpredictable:

Situation:

* My MAC is a MAC Pro with 8 cores and 10 Gb RAM (app. 2 years old)
* The data i would want to analyse is in a local mySQL database in 3 tables with app. 100.000 to 150.000 rows and app. 10 rows

Very often when I do something I get java heap space error

And the really annoying thing about this is that when you restart TOP after such a java heap space error the TOP workspace is corrupted

I have tried to change -Xmx from defalut 256m to a higher value. For a couple of days I could set -Xmx2048m but then after a day or so this generated a "jvm terminated" on starting up TOP

Now I am not able to go higher than -Xmx512m which is ridiculous on a machine with 10 Gb RAM

Right now I can make and run a Column set analysis on one table with 150.000 rows/9 columns without a java heap error but when I make a minor change in the analysis (rearranging the column sequence) and try to save the analysis then I get a java heap error

There is a lot of discussion about java heap in this forum, but alas no conclusive advices and absolutely nothing about java heap and TOP

I am not an expert on java run time tweaking, and don't want to be. And if being a java run time expert i a prerequisite for running TOP i an useful way I think I have to give up TOP and find some other ways to get my analysis done

Please give me some advise or I will have to give up TOP as uselessly unstable

/Karl Erik Jessen
Branchehuset
Copenhagen

scorreia · 2010-09-13 12:40:12

Hi Karl,

first, if you are using the 4.1.0-M4 release, it's not a stable release but a milestone release and now the 4.1.0 RC1 is out. So you may find issues.
In this case, you can file your issues in our bugtracker (currently with the 4.1.0 RC1, there is 200$ reward to the guys that report the higher number of bugs, see [Forum, topic 12274] Bug squashing session before 4.1.0, win $200 on Amazon! ).

The current stable release is 4.0.3.

What kind of analysis are you doing when you get these memory issues?
If you are doing column analysis, it is advisable to use the "SQL engine" parameter at the bottom of the analysis editor.
Otherwise, if you use "java engine", then avoid the use of these indicators which are memory consumers: quartiles, median, frequency tables.

Give us more details about your analyses so that we can help.

kejessen · 2010-09-13 13:50:08

scorreia wrote:
The current stable release is 4.0.3.

I am using 4.0.3. I started out with this version too, and changed for a short while to 4.1.0-M4 but that didn't seem to change much

scorreia wrote:
What kind of analysis are you doing when you get these memory issues?

The analysis that has been causing problems is a "Column set Analysis" and I think the problem is that this kind of analysis is writing a table with all the rows that it has been analysing at the bottom of the results page

When I run a "Column Analysis" on the same table there doesn't seem to be problems (and no table of rows at the bottom)

What I really need is some solid and authoritative advice on how to set jvm memory limits (f.x -Xmx, -Xms) There is a lot of discussion on this forum and the internet in general, but I have not been able to se any "trends" in these advices.

Especially I find it strange that I am not able to set -Xmx1024m or -Xmx2048m but only -Xmx512 - I mean with 10 Gb RAM !

Has this anything to do with how my eclipse is set up? There are 11 different eclipse packages at http://eclipse.org/downloads/

Do you recommend any of these packages? (I don't use ecliplse for anything else than being a platform for TOP and eventually TOS)

/Karl Erik Jessen
Branchehuset
Copenhagen

scorreia · 2010-09-13 14:59:52

I don't see any reason why you could not set -Xmx=2048m.

In which file did you set the -Xmx=2048m when you run TOP?
I see two files called TalendOpenProfiler-macosx-carbon.ini, one at the root of TOP and one in TalendOpenProfiler-macosx-carbon.app/Contents/MacOS/
I don't know which file is taken into account when you run TOP on Mac, so I would suggest to modify both of them.

Then, I'm not sure to understand what you are saying about Eclipse. Are you running TOP from Eclipse? In this case, you should be able to set the memory parameters in the launch configuration.

Apart from these considerations, you may declare a bug about the column set analysis. We should provide better ways to handle memory issues on this kind of analysis.

kejessen · 2010-09-13 15:33:30

scorreia wrote:
I don't see any reason why you could not set -Xmx=2048m.

Neither do I

scorreia wrote:
In which file did you set the -Xmx=2048m when you run TOP?
I see two files called TalendOpenProfiler-macosx-carbon.ini, one at the root of TOP and one in TalendOpenProfiler-macosx-carbon.app/Contents/MacOS/

I changed the one in TalendOpenProfiler-macosx-carbon.app/Contents/MacOS/ I also set the -data /new/workspace setting here, and it worked, so I am pretty sure that it is the ini file inside the .app package that is used

I have attached a screenshot of the jvm-terminated error I get when setting -Xmx2048m, and a copy of the TalendOpenProfiler-macosx-carbon.ini that is causing the problem - maybe you can get inspired

scorreia wrote:
Then, I'm not sure to understand what you are saying about Eclipse. Are you running TOP from Eclipse? In this case, you should be able to set the memory parameters in the launch configuration.

My mistake. I thought that eclipse was needed as some kind of platform to run TOP/TOS. I have deleted eclipse now, and it has changed nothing in the way TOP runs/not runs.

/Karl Erik Jessen
Branchehuset
Copenhagen

Uploaded Images

kejessen · 2010-09-13 15:34:39

Wasn't allowed to attach an .ini file - here is the content:

-data
/Users/kej/Talend/TOP/workspace
-startup
../../../plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar
--launcher.library
../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.200.v20090520-1835
--launcher.XXMaxPermSize
256m
-vmargs
-Xms40m
-Xmx2048m
-Xdock:icon=../Resources/Eclipse.icns
-XstartOnFirstThread
-Dorg.eclipse.swt.internal.carbon.smallFonts

scorreia · 2010-09-13 15:51:01

From the jvm error log, it seems that the memory parameters are well taken into account by the application.
So now, maybe your data do not fit in memory when analyzed in a column set analysis.

In column set analysis, you should avoid the use of columns which have a lot of distinct values. This kind of analysis aggregates data, but if you have an ID column, there will be no aggregation and you will load all the data in memory.
What do you want to know about your 150.000 rows?

kejessen · 2010-09-13 16:05:45

scorreia wrote:
From the jvm error log, it seems that the memory parameters are well taken into account by the application.
So now, maybe your data do not fit in memory when analyzed in a column set analysis.

The error message ("jvm terminated" - the one I made a screendump of i my prior post) is when I start TOP:

When I start TOP with -Xmx2048m or -Xmx1024m, I get the jvm terminated message immediately. When clicking "OK" on this message TOP quits.

So this problem is not related to any specific analysis - it is solely related to the size of the -Xmx setting

/Karl Erik Jessen
Branchehuset
Copenhagen

scorreia · 2010-09-13 16:18:54

Sorry, I did not understand that the crash was during the start of TOP.

I tried with TOP 4.1.0 RC1 on my linux computer and it can run without error.
Does it happen even when you start with a clean workspace?

kejessen · 2010-09-13 16:28:04

Yes

/Karl Erik Jessen

scorreia · 2010-09-13 16:32:03

unfortunately, I cannot help more here as I cannot reproduce it.
Could you open a bug at http://www.talendforge.org/bugs/bug_rep … d_page.php please. Someone at Talend will try to reproduce your issue on a Mac.

scorreia · 2010-09-13 19:09:14

Thanks for your bug report.
I put the link here [Bugtracker, bug 15609, open] jvm terminated error when -Xmx2048m or -Xmx1024m

Announcement

Unanswered posts

#1 2010-09-13 12:24:52

TOP on MAC OS X seems to be very instable - almost unusable

#2 2010-09-13 12:40:12

Re: TOP on MAC OS X seems to be very instable - almost unusable

#3 2010-09-13 13:50:08

Re: TOP on MAC OS X seems to be very instable - almost unusable

scorreia wrote:

scorreia wrote:

#4 2010-09-13 14:59:52

Re: TOP on MAC OS X seems to be very instable - almost unusable

#5 2010-09-13 15:33:30

Re: TOP on MAC OS X seems to be very instable - almost unusable

scorreia wrote:

scorreia wrote:

scorreia wrote:

#6 2010-09-13 15:34:39

Re: TOP on MAC OS X seems to be very instable - almost unusable

#7 2010-09-13 15:51:01

Re: TOP on MAC OS X seems to be very instable - almost unusable

#8 2010-09-13 16:05:45

Re: TOP on MAC OS X seems to be very instable - almost unusable

scorreia wrote:

#9 2010-09-13 16:18:54

Re: TOP on MAC OS X seems to be very instable - almost unusable

#10 2010-09-13 16:28:04

Re: TOP on MAC OS X seems to be very instable - almost unusable

#11 2010-09-13 16:32:03

Re: TOP on MAC OS X seems to be very instable - almost unusable

#12 2010-09-13 19:09:14

Re: TOP on MAC OS X seems to be very instable - almost unusable

Board footer