Talend community forum / Large amount of data update in databse

jijli · 2011-09-26 19:14:28

Tags: [database, date, db lock, deployment, development, mssql, mysql, oracle, PostgreSQL]

I have a job to upload a large amount of flat file to update/insert to a database.

flatfile-->tMap-->tMSSqloutput
|
tMSSqlInput

1. When the job runs, it will lock the database.
Is there any good idea to make it commit fast and won't lock the table at run time?

2. tMSSqlInput is just a lookup to see if the ids in flatfile are in the database.
I use tMap to do this. So it takes very long to load the whole table into hard disk. Is there a good way to do the lookup?

Thanks.

shong · 2011-09-27 07:55:28

Hi
1. Why it lock the database? It is locked by the database server? If so, you need to change the related parameter, like timeout.

2. The lookup table and the output table are the same?

Best regards
Shong

jijli · 2011-09-27 08:06:28

1. When Talend does table update, talend job will keep going, but other transaction will be lock. Until talend job is done.
I just found there is an option in advance(tMsSqlOutput) to set commit number. I am going to set it from 10000 to 100. I hope this bulk load could be done during day time and this database is a transactional database. Do you see if there is any problem?

2. The lookup table and output table are different tables.

Thanks for your help.

Announcement

Unanswered posts

#1 2011-09-26 19:14:28

Large amount of data update in databse

#2 2011-09-27 07:55:28

Re: Large amount of data update in databse

#3 2011-09-27 08:06:28

Re: Large amount of data update in databse

Board footer