Talend community forum / XML and WebService job with Iterate.

aoliveira · 2009-05-11 16:14:42

Hello,

I have a job that does the following :

Reads from a database, creates an xml file. Uses the in memory xml variable to send a *string* to the webservice with the contents of the xml file(the whole xml structure as string).

Because of the large volume of data I had to read each record of the xml and send it to webservice one by one.

My xml looks like this example :

<xml>
<dataImport>
<information>
<row>Value</row>
<row2>Value2</row2>
</information>
<information>
<row>Value</row>
<row2>Value2</row2>
</information>
<information>
<row>Value</row>
<row2>Value2</row2>
</information>
</dataImport>
</xml>

What I had to do is to send the "information" tag, one by one to the webservice.

To do that, I created the attached job. The problem is that, the dataset if big, and as I get close to the row #498 I always get this error message :

Code:

Exception in component tWebServiceInput_1
java.net.BindException: Address already in use: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:369)
    at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:420)
    at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:403)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.lookForImports(SymbolTable.java:859)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.lookForImports(SymbolTable.java:865)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.lookForImports(SymbolTable.java:865)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.lookForImports(SymbolTable.java:865)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.populate(SymbolTable.java:710)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.populate(SymbolTable.java:734)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.add(SymbolTable.java:543)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.populate(SymbolTable.java:518)
    at org.apache.axis.wsdl.symbolTable.SymbolTable.populate(SymbolTable.java:495)
    at org.apache.axis.wsdl.gen.Parser$WSDLRunnable.run(Parser.java:361)
    at java.lang.Thread.run(Unknown Source)

If I run the same job with something like a small 100 rows sample, it works.

This is the code for my tJava job :

Code:

String tmp = new String();
tmp = loop_tAFOX_1.asXML();
tmp = "<efiscRequest><importRequest action=\"INSERT\">" + tmp + "</importRequest></efiscRequest>";
globalMap.put("xmlAsString",tmp);

I have to do that response parsing because the response I get from the WebService is also a String, but it's a xml content string. So I convert that to rows using this code :

Code:

public static String getXmlField(String xml,String fieldName) throws DocumentException 
    {
        String returnString = null;
        if(xml != null)
        {
            Document dResponse = org.dom4j.DocumentHelper.parseText(xml);
            if(dResponse != null)
            {
                Node no = dResponse.selectSingleNode("/efiscResponse/importResponse");
                Element ele = (Element)no;
                for(Iterator i = ele.elementIterator(); i.hasNext();)
                {
                    Element chd = (Element)i.next();
                    if(chd.getName().equals(fieldName) && chd.getText() != null)
                    {
                        returnString = chd.getText();
                        break;
                    }
                }
            }
        }
        return returnString;
    }

Sorry for the long post, I'm still learning Talend and it seems like I'm doing something really wrong here. It seems like it keeps the connections to the webservice open or something like it....

I would really appreciate any comments on how to do this job better.

Thanks!

Alvaro Oliveira.

Uploaded Images

shong · 2009-05-13 14:12:13

Hello

It seems like it keeps the connections to the webservice open or something like it....

As tWebserviceInput access server frequently, it is normal for the address to remain in use for a few minutes after
closing the Socket, so it throws this exception, to aviod it, you can add a tSleep component after tWebServiceInput so as to let the job sleep some time, eg:
--iterate-->tWebServiceInput--iterate-->other transaction
|
onComponentOk
|
tSleep

Best regards

shong

Announcement

Unanswered posts

#1 2009-05-11 16:14:42

XML and WebService job with Iterate.

Code:

Code:

Code:

#2 2009-05-13 14:12:13

Re: XML and WebService job with Iterate.

Board footer