Talend community forum / Delimited File Import Process Jumps to 100%

Li_SpeedyG · 2007-02-05 17:00:53

Hi,

I have just stumbled on your product and I have to say that this is a GREAT idea. And the GUI looks fantastic.

I have started to look into this as the front end to an ERP and Data Warehouse for Company Metrics using an Oracle Back end DB.
I have many data sources/types from which I need to draw information. To that end I have looked at a very simple data import from a delimited file. The first of these files uses a "|" as the field delimiter other files use TABS, etc...

As I try to import the file I am able to see the data in it's raw format. However, once I press the <Next> button the process task shoots up to 100% and the computer becomes unusable for hours or until I do a hard reboot. I must say that I have also been able to get a bit further "on occasion" and when I use the delimiter of "|" each character of each record is split into it's own field. And, while there are only 3-4 fields (in reality) the program splits it into 34-40 fields. Obviously a problem..

I would be very pleased to help you develop your application in any way that I can. Unfortunately, since I am not fluent in Perl or Java I cn oly be of limited use on the programming side. However, sine my applications are rather large, I should be able to assist in the testing and prototyping of the features and documentation. So, hopefully I can work with you to further your application.

Thank You,
GG

My Test machine configuration:
------------------------------------
Windows XP Home
Java 1.5
Perl (Your test script ran ok)

Import File Example:
------------------------
NAME|OCT-2006|NOV-2006|DEC-2006
AI424138|0001|GLUCOSE, FASTING, SERUM|93|185|286
AI424138|0002|UREA (BUN)|93|185|286
AI424138|0003|CHOLESTEROL, TOTAL|93|185|286
AI424138|0004|TRIGLYCERIDES|93|185|286
AI424138|0005|ALKALINE PHOSPHATASE|93|186|287
AI424138|0006|AST|93|186|287
AI424138|0007|ALT|93|186|287
etc..

ccarbone · 2007-02-06 01:11:12

Li_SpeedyG wrote:
As I try to import the file I am able to see the data in it's raw format. However, once I press the <Next> button the process task shoots up to 100% and the computer becomes unusable for hours or until I do a hard reboot.

My Test machine configuration:
------------------------------------
Windows XP Home
Java 1.5
Perl (Your test script ran ok)

Hi Li_SpeedyG

What's your version of Perl? (to know it, use "perl -v" command on a command line)
Can you try to import Demo's project (thanks the "import Demo Project" on the login form) and to play the job that generates DelimitedFile and read the generated DelimitedFile? Can you try to import the generated delimited file on the repository?

Li_SpeedyG · 2007-02-06 02:42:28

Hi CCarbone,

THanks for the reply..
I was able to do as you requested and generated a CSV file "demo_delimited.csv" that contained approx 1000 lines. I then tried to load the data to parse it and it behaved the same as my data.. The "javaw" process shot to 99% and I had to re-start the system to regain control..

Any help would be appreciated..

Thank You,
GG

My Perl version is:
-----------------
C:\>perl -v

This is perl, v5.8.8 built for MSWin32-x86-multi-thread
(with 50 registered patches, see perl -V for more detail)
-----------------

Java is 1.5.0 Rel 4

rbillerey · 2007-02-07 09:32:25

Hi Li_SpeedyG,

I had the same problem with JVM 1.5.0_05 (under XP Home ) which was solved by an upgrade to JVM 1.5.0_11 ( last release ).

Regards.

plegall · 2007-02-07 09:33:19

I've reproduced the "bug". In the "Field Separator" of a tFileInputDelimited, you don't have a simple string, you have a regular expression. The pipe "|" is a special character in a regular expression. To use the pipe as the separator, set the property to '\|'.

Li_SpeedyG · 2007-02-07 14:54:57

rbillerey wrote:
Hi Li_SpeedyG,

I had the same problem with JVM 1.5.0_05 (under XP Home ) which was solved by an upgrade to JVM 1.5.0_11 ( last release ).

Regards.

Thanks for the heads up. However, on a clorporate level we are bound by release 4 du to incomatabliliy issues with other systems. Is there something else I can try? If not I may not be able to use this application.

Li_SpeedyG · 2007-02-07 14:57:43

plegall wrote:
I've reproduced the "bug". In the "Field Separator" of a tFileInputDelimited, you don't have a simple string, you have a regular expression. The pipe "|" is a special character in a regular expression. To use the pipe as the separator, set the property to '\|'.

Thank you.
I will try the fix if the application will allow me to return to the point where I can imput these again.
Hopefully there is a work-around for the Java release issues.

THanks Again

Li_SpeedyG · 2007-03-01 20:39:15

plegall wrote:
I've reproduced the "bug". In the "Field Separator" of a tFileInputDelimited, you don't have a simple string, you have a regular expression. The pipe "|" is a special character in a regular expression. To use the pipe as the separator, set the property to '\|'.

Hi Again plegall,

I have updated to Java 11 and I now seem to be able to view the results. However, I still do not seem to be able to separate the data as required. I used your technique both in the "file delimited" and "File Regex" with the following results:
FILE DELIMITED -- Trying various types of encoding Field Length = Custom RegExp, Custom RegExp = \| -- The output takes each character and developes a seperate field, even the | has a seperate field.

FILE REGEX -- Regular Expression Settings = \| -- The output is a sigle field labled 1 with every entry also 1

I would appreciate your help on resolving this issue.

Thank You,
GG

rbillerey · 2007-03-04 21:02:51

Hi,

Here's a workaround to solve your problem : choose "custom ansi" and use \\| as a delimiter. Note that it's a workaround and next release of TOS will work with \| .

Hope this helps.

Best Regards

Li_SpeedyG · 2007-03-06 03:12:40

rbillerey wrote:
Hi,

Here's a workaround to solve your problem : choose "custom ansi" and use \\| as a delimiter. Note that it's a workaround and next release of TOS will work with \| .

Hope this helps.

Best Regards

Thanks,

I will give it a try and report back..

yazuna · 2008-07-24 17:38:32

Hello,
While trying to get custom regexp as a field separator for various space chars I got no response (single column or errors for a proper a regexp..
i.e. "\\s+" nor "[\t ]+" nor anything seem to work. Is it really regexp evaluator there?
cheers

yazuna · 2008-07-24 17:48:28

hmm, checked the TOS code - you should never put GUI elements for not implemented feature without any comments... there is no regexp separator logic there at all. it's a waste of time for many people...

or I am missing something?

cheers,
Chris

olektrolek · 2008-07-24 17:59:45

test

plegall · 2008-07-24 22:05:36

yazuna wrote:
hmm, checked the TOS code - you should never put GUI elements for not implemented feature without any comments...

You are perfectly right. That's why we avoid to do this.

yazuna wrote:
there is no regexp separator logic there at all. it's a waste of time for many people...

or I am missing something?

TOS 2.4.1, Perl project. Input file:

Code:

1;plegall,2007-02-07 09:33:19
2|Li_SpeedyG    2007-02-07 14:54:57
3#yazuna~Today 17:38:32

It works "like a charm" :-)

Uploaded Images

plegall · 2008-07-24 22:15:09

Hum... I've also tried with a Java project and I understand the reason of your post, it doesn't seem to work at all. I'll check with Java guys if it's supposed to work and if yes, why it doesn't. To make this understandable, keep in mind that this wizard was designed for the first version of Talend Open Studio when Perl was the only generated language

mhirt · 2008-08-02 01:15:16

You're right guys. I have created [Bugtracker, bug 4681, fixed] FileInputDelimited wizard unimplemented options

Thx for your support

yazuna · 2008-10-08 12:14:49

Thank you for the follow up.
Version 3.0.0 just shows an error now, so still no possible to use it there. 2.4.2 works, thanks again.
Cheers

yazuna · 2008-10-08 12:29:18

Sorry, I was to fast - even in 2.4.2 still cannot make it to accept/work with anything like "\s+" or "\\s+" or even "[\t\x0B]+".
Could you please comment if the bug is really fixed?
Cheers

yazun · 2008-10-08 12:41:05

ah right,
the option was just removed from the list..

Announcement

Unanswered posts

#1 2007-02-05 17:00:53

Delimited File Import Process Jumps to 100%

#2 2007-02-06 01:11:12

Re: Delimited File Import Process Jumps to 100%

Li_SpeedyG wrote:

#3 2007-02-06 02:42:28

Re: Delimited File Import Process Jumps to 100%

#4 2007-02-07 09:32:25

Re: Delimited File Import Process Jumps to 100%

#5 2007-02-07 09:33:19

Re: Delimited File Import Process Jumps to 100%

#6 2007-02-07 14:54:57

Re: Delimited File Import Process Jumps to 100%

rbillerey wrote:

#7 2007-02-07 14:57:43

Re: Delimited File Import Process Jumps to 100%

plegall wrote:

#8 2007-03-01 20:39:15

Re: Delimited File Import Process Jumps to 100%

plegall wrote:

#9 2007-03-04 21:02:51

Re: Delimited File Import Process Jumps to 100%

#10 2007-03-06 03:12:40

Re: Delimited File Import Process Jumps to 100%

rbillerey wrote:

#11 2008-07-24 17:38:32

Re: Delimited File Import Process Jumps to 100%

#12 2008-07-24 17:48:28

Re: Delimited File Import Process Jumps to 100%

#13 2008-07-24 17:59:45

Re: Delimited File Import Process Jumps to 100%

#14 2008-07-24 22:05:36

Re: Delimited File Import Process Jumps to 100%

yazuna wrote:

yazuna wrote:

Code:

#15 2008-07-24 22:15:09

Re: Delimited File Import Process Jumps to 100%

#16 2008-08-02 01:15:16

Re: Delimited File Import Process Jumps to 100%

#17 2008-10-08 12:14:49

Re: Delimited File Import Process Jumps to 100%

#18 2008-10-08 12:29:18

Re: Delimited File Import Process Jumps to 100%

#19 2008-10-08 12:41:05

Re: Delimited File Import Process Jumps to 100%

Board footer