You are not logged in.

Unanswered posts



Important! This site has been replaced. All content here is read-only. Please visit our brand-new community at https://community.talend.com/. We look forward to hearing from you there!



#1 2006-10-18 11:36:01

dloreal
Member
3 posts

dloreal said:

"GROUP BY" like in tMap ?

Is it possible to do following query feature with tMap component : GROUP BY  with some calculation related on this grouping ?

Offline

#2 2006-10-19 15:52:09

dloreal
Member
3 posts

dloreal said:

Re: "GROUP BY" like in tMap ?

Stupid question or ... ?

Offline

#3 2006-10-19 16:13:04

plegall
Talend Team


plegall said:

Re: "GROUP BY" like in tMap ?

Your question is not stupid at all, it's just the opposite. The answer is "no". And some of us (me included) have been working on it this morning.

The processing you need will be available in a "tAggregate" component. This kind of component is special regarded to other existing components and not handled in the current "Perl code generation model" (see [wiki] component_creation for details). This kind of component is special because the first output row must wait for the last input row. Our discussions have lead us to the conclusion that we needed some improvement in the code generator engine. This requires more than a couple of days to be stable and releasable.

It's possible to simulate a tAggregate with the existing components (with a tHash and a tPerlFlex) but the solution is not very "visually understandable". Would you like me to show this solution or would you prefer to wait for the availability of the tAggregate?

Offline

#4 2006-10-19 16:21:06

fbonan
Member
53 posts

fbonan said:

Re: "GROUP BY" like in tMap ?

Not stupid at all!
What you describe is what we call tAggregate.

In fact you cannot simulate the tAggregate with a tMap.
tMap component work on a row by row basis. To perform calculation (for a tAggregate or even for a tSort), you need a component that will work on a complete data set (eg. you have to bufferize complete input before you can output any value) (I make it simple a little bit...).
Thus a tMap cannot siumlate this buffering behaviour.

We had a big discussion yesterday and this morning about tAggregate (and tSort) implementation with the R&D team and established a roadmap : 2 weeks seems to be enough to get those component working.

Regards,
Fabrice

PS : our tAggregate will support common calculations like : Sum, Max, Min, Avg, First, Last...and custom op and will support some optimization (memory, disk, group, sort...)

Offline

#5 2006-10-19 16:22:57

dloreal
Member
3 posts

dloreal said:

Re: "GROUP BY" like in tMap ?

Thanks for the answer (and the product).

I will wait for the stable version.

Offline

#6 2006-11-07 14:08:21

Hugo
Guest

Hugo said:

Re: "GROUP BY" like in tMap ?

I take it this component will essentially be a perl associative array (hash)?
Will you be able to configure setting on the component so that it can run in memory (for performance and smaller data loads) or be tied to a file in case of large data volumes?

#7 2006-11-08 09:49:06

favre
Member
12 posts

favre said:

Re: "GROUP BY" like in tMap ?

a tAggregate and a tSort, It's a good news !!

One request : a tFilter Component

Offline

#8 2006-11-08 10:50:11

plegall
Talend Team


plegall said:

Re: "GROUP BY" like in tMap ?

Hugo wrote:

I take it this component will essentially be a perl associative array (hash)?
Will you be able to configure setting on the component so that it can run in memory (for performance and smaller data loads) or be tied to a file in case of large data volumes?

You're perfectly right, tAggregate will first fill a hash and once no input left it will output its content.

The first release of the component will be "memory" only, but in a near future we'll have to provide an option to store hashes on disk. I've already tried  DBM::Deep some months ago and was quite disappointed by low performances.

Offline

#9 2006-11-08 10:52:35

plegall
Talend Team


plegall said:

Re: "GROUP BY" like in tMap ?

favre wrote:

One request : a tFilter Component

Can you please create a new topic about "filtering"? Indeed, at Talend, we've started a discuss about a component dedicated to filtering and I would find useful to make this discussion public with users.

Offline

Board footer

Talend Contributor Agreement - Talend Website Privacy Policy