You are not logged in.
I extract data from a big DataWarehouse Oracle.
Currently, I work on a table which contains about 10 000 000 rows. So I need to filter this data. In Sql I use the where clause but, with Talend I want use tFilterRow. I see a big problem :
when I transmit my filters in SQL with the where clause it's the server which stocks Oracle which manages the filters. But when I use tFilterRow, in fact all the rows are extract and they are filtered in "local". The exection is very different (30 seconds versus 3 hours...)
It's necessary to give my filter in SQL at Oracle or I need to param something ?
Said ad-hoc (I´m a newbie to TOS):
1.- Customizing application env. (like increasing JVM´s Heap for such a volume of data)
2.- For TOS developer: “Buffering data within Java routines, which handle high volume data, whereby code sets the size of buffer in dependency of configured heap size, set by user ”.
Isn´t a good practice to move the TOS toward easy processing of Higher Requirements?
Sorry but I don't understood your answer.
You want say that improve SQL builder (particulary where clause) for the next version of Talend is in prevision or you want say that I need to develop the component ?
Thx for your answer.
If you ask tOracleInput to read 10 000 000 rows, in all the cases, it will be very long.
Simply add your filter to your sql request in tOracleInput.
Why are you speaking about "Improving SQLBuilder" ?
Last edited by mhirt (2008-05-14 19:00:03)
an ETL like Talend is very interisting because the user need to param your soft by a GUI. But in SQL Builder, for the where clause you need to add manually the where condition. For me, I think it will be very interisting if SQL Builder purpose the tFilterRow component in order to create the SQL request with where clause.
Do you see what I want say ?