Talend community forum / Out Of Memory Error Salesfore Metadata Wizard

vcrocca · 2009-03-18 19:26:04

When running the wizard against a large Salesforce table the preview errors out and generates the error below. The system memory spikes during the preview process. If I stop the preview and go to the Guess Schema step it errors out also with the same error. How can I correct this?

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.OutputRecord.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.sendAlert(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.warning(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.closeInternal(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.close(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.close(Unknown Source)
at java.io.BufferedInputStream.close(Unknown Source)
at org.apache.axis.transport.http.SocketInputStream.close(SocketInputStream.java:52)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.close(Unknown Source)
at org.apache.xerces.impl.io.UTF8Reader.close(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager.closeReaders(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.cleanup(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.sforce.soap.partner.SoapBindingStub.queryMore(SoapBindingStub.java:2912)
at org.talend.salesforce.SforceManagementImpl.query(SforceManagementImpl.java:634)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.tSalesforceInputProcess(ShadowFileInputToDelimitedOutput.java:4428)

Preview error. Some settings must be changed.
org.talend.designer.runprocess.shadow.ShadowFilePreview.preview(ShadowFilePreview.java:88)
org.talend.repository.ui.utils.ShadowProcessHelper.getCsvArray(ShadowProcessHelper.java:326)
org.talend.repository.ui.wizards.metadata.connection.files.salesforce.SalesforceStep2Form$PreviewProcessor.nonUIProcessInThread(SalesforceStep2Form.java:408)
org.talend.commons.ui.swt.thread.SWTUIThreadProcessor$1.run(SWTUIThreadProcessor.java:74)

pfisher · 2009-03-20 19:27:16

Hello, could you please paste the contents of your TISEE-win32-x86.ini file located in the root directory of your TOS folder.

Note:

Depending on the .exe you are running TOS with ini file could be: TISEE-win32.wpf OR TISEE-win32-x84_64. Please paste the correct one!

Thanks.

vcrocca · 2009-03-20 19:49:30

Here you go

-vmargs -Xms40m -Xmx500m -XX:MaxPermSize=512m

pfisher · 2009-03-20 23:17:06

Could you please try and increase the -XX:MaxPermSize=512m to: -XX:MaxPermSize=768m and see if this helps?

Thanks.

Volker Brehm · 2009-03-22 23:03:27

Hi,

I think you should increase -Xmx (for example to 768m). I feel -XX:MaxPermSize is too high and could be reduced.

Bye
Volker

vcrocca · 2009-03-23 15:05:01

Tried several options including:

-vmargs -Xms256m -Xmx768m -XX:MaxPermSize=768m

Still getting the same out of memory error.

Volker Brehm · 2009-03-23 16:29:06

Hi,

you could try tor raise -Xmx768m some more and do not use MaxPermSize. Example:
-vmargs -Xms256m -Xmx1536m

Additional it looks like your job needs much memory at all. We could give some more advice if you can give us more information concerning your job design and your data.

Bye
Volker

vcrocca · 2009-03-23 16:38:20

Thanks all I am doing is running the Salesforce Metadata setup wizard to pull Case data from our Salesforce production environment. I have not even started to build a job.

The Wizard crashes on Step 3 when it tries to retrieve the preview data. My guess is that it is trying to pull our entire Case database in the preview instead of limiting it to a few hundred rows.

It works fine when I try it against a test database that has only a couple of dozen records in it.

Volker Brehm · 2009-03-23 23:13:51

Hi,

could you please open a bug in the BugTracker. In meantime modifying -Xmx would be the only solution if there is no option to limit data. I never used Salesforce so I'm not sure if it is possible to limit.

Bye
Volker

pfisher · 2009-03-23 23:26:37

When you create a Bug Tracker Report please link to this forum topic Just so we keep everything connected.

vcrocca · 2009-03-24 15:17:46

Bugtracker thread is here

http://www.talendforge.org/bugs/view.php?id=6747

vcrocca · 2009-03-26 18:19:03

I increased the Java memory values as indicated below.

-Xmx3800m -Xms3800m

Now I do not receive an out of memory error I get the error below at Step 3 in the Metadata wizard.

Exception in component tSalesforceInput
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.net.SocketException: Connection reset
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.skipString(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.sforce.soap.partner.SoapBindingStub.queryMore(SoapBindingStub.java:2912)
at org.talend.salesforce.SforceManagementImpl.query(SforceManagementImpl.java:634)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.tSalesforceInputProcess(ShadowFileInputToDelimitedOutput.java:4469)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.runJobInTOS(ShadowFileInputToDelimitedOutput.java:6624)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.main(ShadowFileInputToDelimitedOutput.java:6540)

{http://xml.apache.org/axis/}hostname:TALEND-DEV

java.net.SocketException: Connection reset
at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:701)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.sforce.soap.partner.SoapBindingStub.queryMore(SoapBindingStub.java:2912)
at org.talend.salesforce.SforceManagementImpl.query(SforceManagementImpl.java:634)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.tSalesforceInputProcess(ShadowFileInputToDelimitedOutput.java:4469)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.runJobInTOS(ShadowFileInputToDelimitedOutput.java:6624)
at sftestprod.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.main(ShadowFileInputToDelimitedOutput.java:6540)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.skipString(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
... 13 more

Preview error. Some settings must be changed.
org.talend.designer.runprocess.shadow.ShadowFilePreview.preview(ShadowFilePreview.java:88)
org.talend.repository.ui.utils.ShadowProcessHelper.getCsvArray(ShadowProcessHelper.java:326)
org.talend.repository.ui.wizards.metadata.connection.files.salesforce.SalesforceStep2Form$PreviewProcessor.nonUIProcessInThread(SalesforceStep2Form.java:408)
org.talend.commons.ui.swt.thread.SWTUIThreadProcessor$1.run(SWTUIThreadProcessor.java:74)

Volker Brehm · 2009-03-26 22:24:56

I think this should be handled together with your memory problem. I just want to say add a note to the Bug as I see that it is closed...

I do not have any deeper knowledge concerning Salesforce. So I have to pass :-(
Maybe a timeout happens?

Bye
Volker

By the way: I could understand the reason why the last problem had been closed but from my point of view some more analyzing should had been done.

vcrocca · 2009-03-27 14:05:31

Thanks Volker! I am still pursuing this issue with Talend. There will probably be another bug submitted.

Announcement

Unanswered posts

#1 2009-03-18 19:26:04

Out Of Memory Error Salesfore Metadata Wizard

#2 2009-03-20 19:27:16

Re: Out Of Memory Error Salesfore Metadata Wizard

#3 2009-03-20 19:49:30

Re: Out Of Memory Error Salesfore Metadata Wizard

#4 2009-03-20 23:17:06

Re: Out Of Memory Error Salesfore Metadata Wizard

#5 2009-03-22 23:03:27

Re: Out Of Memory Error Salesfore Metadata Wizard

#6 2009-03-23 15:05:01

Re: Out Of Memory Error Salesfore Metadata Wizard

#7 2009-03-23 16:29:06

Re: Out Of Memory Error Salesfore Metadata Wizard

#8 2009-03-23 16:38:20

Re: Out Of Memory Error Salesfore Metadata Wizard

#9 2009-03-23 23:13:51

Re: Out Of Memory Error Salesfore Metadata Wizard

#10 2009-03-23 23:26:37

Re: Out Of Memory Error Salesfore Metadata Wizard

#11 2009-03-24 15:17:46

Re: Out Of Memory Error Salesfore Metadata Wizard

#12 2009-03-26 18:19:03

Re: Out Of Memory Error Salesfore Metadata Wizard

#13 2009-03-26 22:24:56

Re: Out Of Memory Error Salesfore Metadata Wizard

#14 2009-03-27 14:05:31

Re: Out Of Memory Error Salesfore Metadata Wizard

Board footer