• Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » Losing rows when pushing data into SQL

#1 2012-03-29 16:50:01

Steve.ward
New member
Registered: 2012-03-19
Posts: 1

Losing rows when pushing data into SQL

Tags: [database, mssql]

Hi

I have created a job to pull data from Salesforce and push into SQL in talend version 4.1.0M1.

I have created a Salesforce metadata object which imports all fileds with no filter ciriteria i.e. all rows shou;d be imported.  I have a Tmap which connects the Salesforce object to a SQL output node.  The schema of the SQL node is copied from the salesforce repository object.

I Import c116,000 rows from the Salesforce object which I can see pass thorugh my Tmap to my SQL output node with no loss.  i.e. row count going into tmap is the same as row count coming out of Tmap, and I have no other filters at all.

When I select all from my sql database i only see c110,000 records.

I allow the sql node to create the table in my SQL databse so all field lengths are the same as surce data.

I get the message of data truncatation which I am currently looking into, but thisis only seen a few times, no where near enough to explain the 1000's of records missing.

My question is what is happening to the mising rows?

Any help would be appreciated, I am going stir crazy!

Please see attached screenshot.  In screenshot the csv output is simply to view output to another destiniation rather than SQL so I can try to see if this is a database entry issue or something else.

Regards

Steve

Offline

#2 2012-03-29 17:13:44

BRM
Member
Registered: 2011-05-23
Posts: 11

Re: Losing rows when pushing data into SQL

I had this same problem too.  I could not figure out why rows were being lost.  My workaround was to extract the records from SF to a *.csv file and then load the *.csv to the database.  Its inelegant but saved me pulling my hair out like you are.

Here is my original post http://www.talendforge.org/forum/viewtopic.php?id=18280

Good luck.

Offline

  • Index
  •  » Talend Open Studio for Data Integration » Usage, Operation
  •  » Losing rows when pushing data into SQL

Board footer

Powered by FluxBB