I am a Talend Open Source newbie (1 week) and I need a component to extract a list of hyperlinks from an html page I download with tFileFetch.

The specific hyperlinks I need to extract download data files.  If I get a complete list of hyperlinks (one per row in a file)
in a second step I can filter the list for the one's I am interested in and then in a third step I can iterate over the list and use string functions (from Talend Code\Routines) to build the URLs I want to pass to another tFileFetch to download the 50+ data files on a daily basis.

I have successfully downloaded the HTML page by feeding the original HTML link to tFileFetch.

By HTML hyperlinks I mean everything between "<A" and "</A>".

In general, extracting hyperlinks can be done with Regular Expressions or an XML/XQUERY, but Talend's components
assume something close to a regular row and column structure (a schema) and blow up with malformed or loosely structured HTML.
Slightly off topic -- one exception (for my application) might be Exchange component tHTTPTableInput (how to install in TOS?).

I researched the topic and found convoluted Regular Expressions (RegEx):
<a.*href=('|")?(http\://.*?(?=\1)).*>\s*([^<]+|.*?)?\s*</a>
http://vidmar.net/weblog/archive/2009/09/10/matching-links-with-regular-expression-in-html.aspx

and this interesting February 2008 blog post "Showdown – Java HTML Parsing Comparison"
on extracting hyperlinks using an XML/XQUERY from Java.
http://www.benmccann.com/dev-blog/java-html-parsing-library-comparison/

"So, to test the [Java callable] parsing libraries, I decided to do exactly that and see if I could parse the HTML well enough to extract links from it using an XQuery. The contenders were NekoHTML, HtmlCleaner, TagSoup, and jTidy. "
* * *
"I gave each library an InputStream created from a URL (referred to as urlIS in the code samples below) and expected an org.w3c.dom.Node in return once the parse operation was completed. [I need a flat file with one link per row.]"
* * *
"Finally, to judge the ability to parse the HTML, I ran the XQuery “//a” to grab all the <a> tags from the document [Exactly what I need!!!]."

NOTE: Compare the XML/XQUERY ""//a" to the Regular Expression "<a.*href=('|")?(http\://.*?(?=\1)).*>\s*([^<]+|.*?)?\s*</a>".

"The only one of these parsing libraries I had used before was jTidy. It was able to extract the links from 5 of the 10 documents. However, the clear winner was HtmlCleaner. It was the only library to successfully clean 10/10 documents. "
* * *
"One drawback to HtmlCleaner is that it’s not available in a Maven repository.  Sometimes NekoHTML may be easier to use for this reason."
http://www.benmccann.com/dev-blog/java-html-parsing-library-comparison/

The blog post does not give the complete Java code:
"I implemented each library in its own class extending from an AbstractScraper [Java class, code not shown] implementing a Scraper [Java] interface I created. [not shown]"
* * *
"The implementation specific [Java] code for each library is below"

So, if we can get the complete Java code from the blog post author can this be implemented in a custom code tJava component?

As I mentioned at the beginning, I have downloaded a page using tFileFetch
and if I can get a complete list of hyperlinks (one per row in a file)
in a second step I can filter the list (using ? Talend component) for the URL's I am interested in
and then in a third step I can iterate over the list and use string functions (from Talend Code\Routines)
to build the URLs I want to pass to another tFileFetch to download the 50+ data files on a daily basis.

But first, I have to get over this hump (extracting the links) -- can you help?

Thanks

Jim