#1 2011-06-25 21:02:15

David Costelloe
Guest

Error when using File Delimetered CSV

The following error is being thrown whe I try to load a large csv file:

Exception in thread "main" java.lang.Error: Unresolved compilation problem:
    The code of method tFileInputDelimitedProcess(Map<String,Object>) is exceeding the 65535 bytes limit

    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.tFileInputDelimitedProcess(ShadowFileInputToDelimitedOutput.java:12219)
    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.runJobInTOS(ShadowFileInputToDelimitedOutput.java:24747)
    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.main(ShadowFileInputToDelimitedOutput.java:24645)

Preview error. Some settings must be changed.
Note: Preview errors are generally due to a wrong encoding setting.
org.talend.designer.runprocess.shadow.ShadowFilePreview.preview(Unknown Source)
org.talend.repository.ui.utils.ShadowProcessHelper.getCsvArray(Unknown Source)
org.talend.repository.ui.wizards.metadata.connection.files.delimited.DelimitedFileStep2Form$PreviewProcessor.nonUIProcessInThread(Unknown Source)
org.talend.commons.ui.swt.thread.SWTUIThreadProcessor$1.run(Unknown Source)

Thanks
David......

#2 2011-06-30 09:12:16

dforrestca
Member
Registered: 2011-04-01
Posts: 19

Re: Error when using File Delimetered CSV

Getting the same error message about exceeding the 65k using the tFileInputJson component. 65k? Talend folks, please, you need to fix this.
Thanks

Offline

#3 2011-06-30 09:54:44

shong
Talend team
Registered: 2007-08-29
Posts: 10359
Website

Re: Error when using File Delimetered CSV

Hi dforrestca

This is a well-known problem, you can find some explanation in topics:
http://www.talendforge.org/bugs/view.php?id=6766
http://www.talendforge.org/bugs/view.php?id=14512

Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#4 2011-06-30 18:07:00

dforrestca
Member
Registered: 2011-04-01
Posts: 19

Re: Error when using File Delimetered CSV

A well known problem is definitely a bug, yes? Thank you for the response. Anyway, the other postings pointed to mostly refer to splitting up a job to solve the issue.  We are having a problem with the 65k limit reading in a json data stream with the the Tfileinputjson component. Please see the attached pic of a job setup to split the input stream. The main glitch with this is I think tJoin was intended for database input as the lookup feature seems to be  not working for us.
Anyone have an example they wish to share of combining to file inputs into one? Tried using the json doc "_id" as a matching element, but, still seems not to join them up.
Any help appreciated,
Thanks


Uploaded Images

Offline

#5 2011-07-01 04:25:42

shong
Talend team
Registered: 2007-08-29
Posts: 10359
Website

Re: Error when using File Delimetered CSV

Hi
The problem is there are so many columns in the tFileInputJason component, the generated code exceed the 65k byte limit.

Please see the attached pic of a job setup to split the input stream. The main glitch with this is I think tJoin was intended for database input as the lookup feature seems to be  not working for us.

Do you want to merge the two input stream or join them?

Best regards
Shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the leader of open source data management and application integration solutions!

Offline

#6 2011-07-01 08:09:38

dforrestca
Member
Registered: 2011-04-01
Posts: 19

Re: Error when using File Delimetered CSV

Hi Shong,

Thanks again for reply. I think we want to join? What we would love to do is read the json stream in all at once, but, that 65k barrier is killing us. SOoo... I thought to split the json schema into two logical parts and read the entire URL by both of them in separately and then join then back together as one. I hope we can find a better way to do this long term, but, need to have it working now. Its fantastic that we can pull a row from a couchdb database by URL using the tInputjsonFile and we need this to work with a whole stream. Part of the issue too is that talend throws errors when there is json data without a mapping from the json pathing associated  to the schema. In other words, let's say the stream contains Elements A,B,C,D, E, F. With one tFileJson the schema and json pathing are to A,B, and C, the second tFileJson has D,E,F. The first one will throw the "json data structure....may not be right" error because elements D,E,and F are not in the schema/path and the second fFIleJson throws the error because elements A,B,C are not  in the schema/path. Please, there must be a workaround we can do until you guys resolve the 65k issue.  We could also restructure the generated code with a little refractoring.

Please advise...

Thanks and regard,
dforrestca

Offline

#7 2011-07-01 18:44:46

dcostelloe
New member
Registered: 2011-04-16
Posts: 5

Re: Error when using File Delimetered CSV

Hi,
Being new I can't load the prevew to setup the tfile component to be able to split error I am getting in the preview is:
Exception in thread "main" java.lang.Error: Unresolved compilation problem:
    The code of method tFileInputDelimitedProcess(Map<String,Object>) is exceeding the 65535 bytes limit

    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.tFileInputDelimitedProcess(ShadowFileInputToDelimitedOutput.java:12219)
    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.runJobInTOS(ShadowFileInputToDelimitedOutput.java:24747)
    at importmasterdata.shadowfileinputtodelimitedoutput_0_1.ShadowFileInputToDelimitedOutput.main(ShadowFileInputToDelimitedOutput.java:24645)

Preview error. Some settings must be changed.
Note: Preview errors are generally due to a wrong encoding setting.
org.talend.designer.runprocess.shadow.ShadowFilePreview.preview(Unknown Source)
org.talend.repository.ui.utils.ShadowProcessHelper.getCsvArray(Unknown Source)
org.talend.repository.ui.wizards.metadata.connection.files.delimited.DelimitedFileStep2Form$PreviewProcessor.nonUIProcessInThread(Unknown Source)
org.talend.commons.ui.swt.thread.SWTUIThreadProcessor$1.run(Unknown Source)

Offline

#8 2011-07-01 20:09:14

dcostelloe
New member
Registered: 2011-04-16
Posts: 5

Re: Error when using File Delimetered CSV

What would be nice is a tutorial of a work around for anyone that gets this error.

Thanks

Offline

#9 2011-07-01 23:00:56

dforrestca
Member
Registered: 2011-04-01
Posts: 19

Re: Error when using File Delimetered CSV

Hi Dcostelloe,

Shong has another posting out there on another topic that brings three spreadsheets together in one stream.. http://www.talendforge.org/forum/viewtopic.php?id=11978
Using that as an example, we went forward and have been getting the desired output, though it still needs a little tweaking.... see below


Uploaded Images

Offline

#10 2011-07-02 16:49:43

dcostelloe
New member
Registered: 2011-04-16
Posts: 5

Re: Error when using File Delimetered CSV

Ah yeah looks like there are multiple files, I only have one large file which I can not Split out around 1000 columns with 29924 rows which is causing the issue with limitation. I can't split out the file as this is how the data is provided.
Will try the suggestion in the forums again.

Offline

#11 2011-07-03 16:11:45

dcostelloe
New member
Registered: 2011-04-16
Posts: 5

Re: Error when using File Delimetered CSV

This is what I tried:

Took 50% on Part1
50% on Part2


Uploaded Images

Last edited by dcostelloe (2011-07-03 16:17:07)

Offline

Board footer

Powered by FluxBB