Talend community forum / FTPGet gets unstable on large volumes of data

umair · 2008-04-30 10:58:47

I have an FTP Get component which fetches files from an FTP server.
At any given time, there can be as many as 2,000 files.
The flow goes like:

tFTPGEt ->(onsubJobOK)-> tFileList ->(iterate) -> tMapper (insertion to mysql)

(Attached Image)
It takes a long time to fetch files form FTP server, approx 2 minutes.
And then when it starts inserting the filenames into MYSQL, it terminates in the middle, by throwing the following error:

(Edit: Actually it's mysql which fails. I tried with FTPGET disabled and it fails again after approx 1965 insertions. Can this be due to the 'iterate' on FileList??)

Exception in component tMysqlInput_1
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The driver was unable to create a connection due to an inability to establish the client portion of a socket.

This is usually caused by a limit on the number of sockets imposed by the operating system. This limit is usually configurable.

For Unix-based platforms, see the manual page for the 'ulimit' command. Kernel or system reconfiguration may also be required.

For Windows-based platforms, see Microsoft Knowledge Base Article 196271 (Q196271).
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2001)
at com.mysql.jdbc.Connection.<init>(Connection.java:716)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.Connection.getInstance(Connection.java:295)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.tFileList_1Process(SDW_CDR_MainJob.java:526)
at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.tFTPGet_1Process(SDW_CDR_MainJob.java:222)
at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.runJobInTOS(SDW_CDR_MainJob.java:1091)
at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.main(SDW_CDR_MainJob.java:1010)
Caused by: java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:168)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:262)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1926)
... 14 more

Uploaded Images

Last edited by umair (2008-04-30 11:04:52)

shong · 2008-05-05 07:50:23

Hello

This exception isn't caused by Talend and your job is ok. It seems it was a bug from MySQL itself. You can go hereand find more related information.

Best regards

shong

umair · 2008-05-05 18:45:24

Hmm...i tried searching for this Bug...but i cant get any indication weather it has been resolved or not.

shong · 2008-05-06 04:53:01

Hello

1)Try to install another version of MySQL and test it.
2)First, output the file name in a temporary file, then read this file and load the data into database, if you still meet the problem, you can limite the lines(see the screenshot) or use a tSampleRows component which allow you to choose a list of line numbers and/or a list of ranges.

Best regards

shong

Uploaded Images

umair · 2008-05-19 18:31:02

quote:
2)First, output the file name in a temporary file, then read this file and load the data into database, if you still meet the problem, you can limite the lines(see the screenshot) or use a tSampleRows component which allow you to choose a list of line numbers and/or a list of ranges.

I implemented this example and the result was pretty interesting.

I have divided the job into 2 jobs.
One job having a tFileList iterating over a directory, reads files and inserts filenames into a temporary file.
The second job, reads filenames from that temporary file and inserts them into MYSQL.
This works perfectly and no mysql exception is thrown.

Now, i am facing the similar problem in a different scenario:

My second scenario required reading the content of the files in the same directory.
I am again using tFileList to read the files in that directory.
The number of files is around 5,000 and this job makes use of quiet a many DB components.

I now get the same error message on this particular job.

Any comments?

[Edit: i think, tFileList is somehow creating problems due to the iterate action.....??? ]

Last edited by umair (2008-05-19 19:12:09)

Announcement

Unanswered posts

#1 2008-04-30 10:58:47

FTPGet gets unstable on large volumes of data

#2 2008-05-05 07:50:23

Re: FTPGet gets unstable on large volumes of data

#3 2008-05-05 18:45:24

Re: FTPGet gets unstable on large volumes of data

#4 2008-05-06 04:53:01

Re: FTPGet gets unstable on large volumes of data

#5 2008-05-19 18:31:02

Re: FTPGet gets unstable on large volumes of data

Board footer