Talend community forum / Problem with tWaitForfile

MartinAPG · 2012-07-20 12:37:37

Hi there,

I am using tFTPGet to sychronize a local Folder with subfolders with an FTP Folder and its subfolders. (Aim is to get all the newly created files in all subfolders)
the folder structure is like this \2012\20120101\..
I want to process only the new files and therefore I copy (with tFilecopy (String)globalMap.get("tWaitForFile_1_CREATED_FILE")) the new ones into a folder new and thean write them into my sql database and am using tWaitForFile tor that purpose to check the local folder for the new files transferred by tftpget.
As long as I scan only 1 folder with tWaitforfile everything works fine.
But when I give tWaitforFile as parameter path+((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")).substring(53) which is the actual folder iterated at that moment in tFTPget, tWaitforfile triggers every file in the folders as "newlycreated" (even already existing ones). (ref. pic2.jpg)

When I give him the folder path\2012\ and "include subdirectories" he triggers not a single new file neither if it is directly created in \2012\ nor in its subfolders.
Is there an issue with tWaitforFile and too many subfolders?

Thx in advance for help.

Uploaded Images

Last edited by MartinAPG (2012-07-20 12:39:41)

janhess · 2012-07-20 17:50:23

Is the tWaitForFile looking at the folder you are ftping the files to? If so they will always all be newly created.

MartinAPG · 2012-07-23 08:20:24

yes it is looking at that folder, but the local folder already has all files already except the new files which arrive 20-30 every day.. so we are talking about 1 folder on the ftp per calender day in each folder 1-5 files for each hour...adding hour files daily as new data comes in....so I have the exact same Filestructure locally as remotely, in order that tftpget copies only the new ones (which don't exist yet locally) the idea was that twaitforfile triggers on those new files and copies them into a seperate folder in order to be processed by tfilelist on that folder (that process deletes then the files so that they won't be imported uselessly again. The reasen I chose this solution is there no way of knowing how the new files will be named on the ftp server because they include the actual timestamp when they where created in the name.

so again why does twaitforfile triggers on all files even locally existing (when the directory parameter is a variable "current_filepath" from tftpget) or reacts to none if I give him a folder with subfolders (flag include subdirectories) to scan e.g. ../2012/.
waitforfile only works correctly when he is given one specific folder with no subdirs.

any help would be very much appreciated.

Last edited by MartinAPG (2012-07-23 08:22:08)

shong · 2012-08-02 08:25:13

Hi
You don't add any condition on tFTPGet component, so it always get all the files into local folder and override them if some of files already exist. To get the only news file from FTP server, we usually process it like this:
tFTPFileList--iterate--tFTPFileProperties---main--tJavaRow--runIf--tFtpGet
|
other processing after donwload all the new files from FTP server.

tFTPFileProperties: get the file properties, including basename, abs_path, mtime, etc.
on tJavaRow, compare the mtime with a special time, for example, compare the mtime with current timestamp, if current_timestamp-etime<=24 hours, we can say it is a new file, and then get the file!

This is an example code on tJavaRow:

if(TalendDate.getCurrentDate().getTime()-input_row.mtime<=24*60*60*1000){
globalMap.put("isNewFile",true);
}else{
globalMap.put("isNewFile",false);
}

Set the condition expression of runIf:
(Boolean)globalMap.get("isNewFile")

Best regards
Shong

MartinAPG · 2012-08-09 13:02:43

Hi Shong,

Thx for your reply...my user for the support account is not in operation yet that is why I tried my hopes here.
Well I give the parameter "do not overwrite" in tFTPget so it does not make sense that he overwrites indeed he does not overwrite so your point is for naught.

My problem is not with tftpGet (he gets only the new files) but with tftpwaitforfile which is watching the superior folder where the new files are placed in subfolders.

best regards,
Martin

Announcement

Unanswered posts

#1 2012-07-20 12:37:37

Problem with tWaitForfile

#2 2012-07-20 17:50:23

Re: Problem with tWaitForfile

#3 2012-07-23 08:20:24

Re: Problem with tWaitForfile

#4 2012-08-02 08:25:13

Re: Problem with tWaitForfile

#5 2012-08-09 13:02:43

Re: Problem with tWaitForfile

Board footer