Talend community forum / How to extract data from a website?

Chris8080 · 2008-04-01 11:30:19

Hi,

i´ve got two websites. One Website wich supports SOAP, imports and so on.
Another Website wich keeps about 7000 html documents with an identical format with information in tables on it.

Now, with the relaunch, I have to transport content from the 7000 files to a database / CMS / SOAP.

I saw, that talend is able to connect to http.
Can I also extract data from html tables?

Thank you.
Bye, Chris

cahsohtoa · 2008-04-01 12:48:46

Ithink that There isn't any way to extract data from a html table but if you have only table you may use a regular expression

Volker Brehm · 2008-04-01 22:58:34

Hello Chris,

as Olivier wrote, there is no special component. I had the same problem and it ends up in a tJavaRow with many regex. But that depends on your html structure. I've experimented a little bit with html2xml converter. If you search in google you should find different tools (including open source). At the end I could'nt use them because my input was very "unwell formed".

If you found a solution please give a us a feedback.

Bye
Volker

Alain COUTHURES · 2008-04-03 10:00:47

I have written an OpenSource function for converting bad HTML to well-formed XML (http://sourceforge.net/projects/light-html2xml) and I would appreciate to test it with your input.

It is a single-pass automat and it does not need specific objects. It is not yet written in Java but in C# and in PHP5 (I will soon rewrite it in Java, especially if you're interested in...).

cahsohtoa · 2008-04-03 10:27:21

Yes I think that it would be a really good idea to write it in java then I will create a specific talend component to perform this action

rbillerey · 2008-04-03 14:25:19

Hi,

We use for internal stats some Talend jobs using http://cpan.uwinnipeg.ca/module/HTML::TokeParser in tPerl/tPerlRow. We may push on the stack a new component if you need it.

Hope this helps

Alain COUTHURES · 2008-04-04 12:20:28

The Java version of the html2xml function I have written is now downloadable at http://sourceforge.net/projects/light-html2xml

Please send me your comments and remarks about it so I will fix bugs.

Last edited by Alain COUTHURES (2008-04-04 12:22:55)

Ali · 2009-05-14 11:02:48

Yes u can extract all data from 7000 pages. i m also working on this.

hammerwd · 2009-05-14 11:51:51

I found another helpful thing for this:
http://www.iopus.com/imacros/firefox/?ref=fxmoz
Amazing tool to automate the web, even data extraction works fine.
One could combine the output which is e.g. Excel with Talend to get it into another database.

nhudinhthuan · 2009-07-24 06:19:43

User vder software, extract data from Amazon.com output to xml format. view screenshot: http://binhgiang.sourceforge.net/xmlalbum/slides/vietspider%20xml%20list%20detail%201.html

and download from: http://binhgiang.sourceforge.net/site/download.jsp

Techsmart · 2009-12-15 05:44:36

I would suggest Automation Anywhere. Great tool for web data extraction and automating any task. Free Trial available for download at:
http://www.automationanywhere.com/download/freeTrial.htm

Just try it out!

mlange · 2010-01-04 16:11:15

You can also try tHTTPTableInput. This component has been designed for extracting data directly from HTML Pages.

http://www.talendforge.org/exchange/tos … php?eid=72

Regards
Martin

Janib · 2010-08-01 08:42:37

Have you ever wonder if you can have full contents from your desired website into a single Excel Document?

If so, I have the solution for you at fairly cheaper price.

I can extract most of the website data and compile it in a single ms-excel 2003 format within just few days.

It can be any website, from a simple site to complex sites like b2b portals or whatever you can come up with.

Contact me with your website and requirements.

Regards,

Janib Soomro
janib4all@hotmail.com

s.downloader · 2010-12-07 22:12:06

I can make it for you. site.downloader@gmail.com

Honed · 2011-09-02 20:53:50

Talend, I am having trouble in getting HTML table data to excel using talend v4.2.2. I saw there is a component thttptable for previous version.
Can you help in this regard?

W4shington · 2011-11-30 13:28:29

Hello Honed,

I'm having the same problem, when i try to catch data from the html page that cames with the component everything works fine, but this page is very simple does not have any divs, or blockquotes, is structured only using tables, when i try to use a page that uses more html tags, like blockquotes, is like tHTTPTableInput does not recognize the Tables, so it launch a

"Exception in component tHTTPTableInput_1 java.lang.ArrayIndexOutOfBoundsException:"

Does anyone here has the same problem or know how to solve this?

Thanks

Announcement

Unanswered posts

#1 2008-04-01 11:30:19

How to extract data from a website?

#2 2008-04-01 12:48:46

Re: How to extract data from a website?

#3 2008-04-01 22:58:34

Re: How to extract data from a website?

#4 2008-04-03 10:00:47

Re: How to extract data from a website?

#5 2008-04-03 10:27:21

Re: How to extract data from a website?

#6 2008-04-03 14:25:19

Re: How to extract data from a website?

#7 2008-04-04 12:20:28

Re: How to extract data from a website?

#8 2009-05-14 11:02:48

Re: How to extract data from a website?

#9 2009-05-14 11:51:51

Re: How to extract data from a website?

#10 2009-07-24 06:19:43

Re: How to extract data from a website?

#11 2009-12-15 05:44:36

Re: How to extract data from a website?

#12 2010-01-04 16:11:15

Re: How to extract data from a website?

#13 2010-08-01 08:42:37

Re: How to extract data from a website?

#14 2010-12-07 22:12:06

Re: How to extract data from a website?

#15 2011-09-02 20:53:50

Re: How to extract data from a website?

#16 2011-11-30 13:28:29

Re: How to extract data from a website?

Board footer