quote:
2)First, output the file name in a temporary file, then read this file and load the data into database, if you still meet the problem, you can limite the lines(see the screenshot) or use a tSampleRows component which allow you to choose a list of line numbers and/or a list of ranges.

I implemented this example and the result was pretty interesting.


I have divided the job into 2 jobs.
One job having a tFileList iterating over a directory, reads files and inserts filenames into a temporary file.
The second job, reads filenames from that temporary file and inserts them into MYSQL.
This works perfectly and no mysql exception is thrown.

Now, i am facing the similar problem in a different scenario:

My second scenario required reading the content of the files in the same directory.
I am again using tFileList to read the files in that directory.
The number of files is around 5,000 and this job makes use of quiet a many DB components.

I now get the same error message on this particular job.

Any comments?

[Edit: i think, tFileList is somehow creating problems due to the iterate action.....??? ]

Hmm...i tried searching for this Bug...but i cant get any indication weather it has been resolved or not.

I have an FTP Get component which fetches files from an FTP server.
At any given time, there can be as many as 2,000 files.
The flow goes like:

tFTPGEt ->(onsubJobOK)-> tFileList ->(iterate) -> tMapper (insertion to mysql)

(Attached Image)
It takes a long time to fetch files form FTP server, approx 2 minutes.
And then when it starts inserting the filenames into MYSQL, it terminates in the middle, by throwing the following error:

(Edit: Actually it's mysql which fails. I tried with FTPGET disabled and it fails again after approx 1965 insertions. Can this be due to the 'iterate' on FileList??)

Exception in component tMysqlInput_1
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The driver was unable to create a connection due to an inability to establish the client portion of a socket.

This is usually caused by a limit on the number of sockets imposed by the operating system. This limit is usually configurable.

For Unix-based platforms, see the manual page for the 'ulimit' command. Kernel or system reconfiguration may also be required.

For Windows-based platforms, see Microsoft Knowledge Base Article 196271 (Q196271).
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:2001)
    at com.mysql.jdbc.Connection.<init>(Connection.java:716)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
    at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
    at com.mysql.jdbc.Connection.getInstance(Connection.java:295)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.tFileList_1Process(SDW_CDR_MainJob.java:526)
    at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.tFTPGet_1Process(SDW_CDR_MainJob.java:222)
    at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.runJobInTOS(SDW_CDR_MainJob.java:1091)
    at sample_etl.sdw_cdr_mainjob.SDW_CDR_MainJob.main(SDW_CDR_MainJob.java:1010)
Caused by: java.net.BindException: Address already in use: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:168)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:262)
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:1926)
    ... 14 more