You are not logged in.

#1 2008-05-21 17:36:30

strpdsnk
Member
328 posts

delete files more than 45 days (example) old?

So, I figure once I get a list of these files, I can just use tFileDelete to delete them.  (I assume I would feed into tFileList and then make a connection to tFileDelete, like in the pic, but I'm not positive.)


What I don't know is how to read a directory and get a list of the (zip) files that are older that today minus x # of days.
mini_11768_1.JPG

Offline

#2 2008-05-22 08:23:13

shong
Talend Team


Re: delete files more than 45 days (example) old?

Hello

The tFileProperties component(under File-->Management family) can fit your need. With tFileProperties component, you can get the create time of the file.

Best regards

         shong


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the global leader of open source data management and application integration solutions!

Offline

#3 2008-05-22 18:46:10

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

what version is that in?  Cause, its not showing up for me in TOS 2.3.3r13892


Ah, ok, I see it for the Perl version, but I'm in Java...
[img]http://www.talendforge.org/forum/img/members/1327/mini_11814_file management.JPG[/img]

Last edited by strpdsnk (2008-05-22 18:50:39)

Offline

#4 2008-05-22 19:18:12

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

this bug topic ([Bug] 3545) seems to indicate that the java module is completed in 2.4.0RC1.  are the fixes that are in 2.3.3 also in the latest 2.4.0 release?  (RC2)


Guess I'm about to find out.

Offline

#5 2008-05-22 19:41:42

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

ok, so... is there somewhere I can find documentation on tFileProperties?  cause its not in the most recent documentation available here http://www.talend.com/resources/documentation.php
and its not available in the help linked from TOS : http://127.0.0.1:50949/help/nav/2_2

or can someone tell me how to use this thing? please!

Last edited by strpdsnk (2008-05-22 20:02:40)

Offline

#6 2008-05-23 03:53:00

shong
Talend Team


Re: delete files more than 45 days (example) old?

Hello

The tFileProperties component describes the properties of one file, include:
abs_path: the absolute path
dirname: the direcotry name
basename: the file name
mode_string: read or write
size: the size of file
mtime: the create time of file, return millisecond (long)
mtime_string: the create time of file, return a date (String)

Here is a simple scenario(see the screenshot)

Best regards

         shong
mini_11834_1.pngmini_11834_2.pngmini_11834_3.png


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the global leader of open source data management and application integration solutions!

Offline

#7 2008-05-29 20:34:05

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

ok... now I can't figure out how I can filter on mtime_string and return either abs_path or basename to tFileList or tFileDelete

I mean, I think I can do some of the filtering in something like tMap, but I don't know if thats the right way to do it

Offline

#8 2008-05-30 03:37:14

shong
Talend Team


Re: delete files more than 45 days (example) old?

Hello

ok... now I can't figure out how I can filter on mtime_string and return either abs_path or basename to tFileList or tFileDelete

Here is a Java example, Please see the screenshot.

 /**
     * Returns the number of milliseconds since January 1, 1970, 00:00:00 GMT
     * represented by this <tt>Date</tt> object.
     *
     * @return  the number of milliseconds since January 1, 1970, 00:00:00 GMT
     *          represented by this date.
     */
    public long getTime() {
        return getTimeImpl();
    }

Best regards

         shong
mini_12164_1.pngmini_12164_2.pngmini_12164_3.pngmini_12164_4.pngmini_12164_5.png


Email:shong@talend.com
Choose Talend, Enjoy Talend!
New & Event: Talend Help Center
Talend-->the global leader of open source data management and application integration solutions!

Offline

#9 2008-05-30 20:15:16

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

So I tried this process like you said, and it works perfectly... that is, until I try to change the # of days to anything greater than 24

I tried a bunch of different days, and all the ones <=24 worked, and all the ones >24 caused all the files to be deleted

Picture Information
1. contents of the folder before I run the clean up process
2. process with tFilterRow set to delete files created before 25 days
3. results of running the process defined in #2, where all files are deleted (whether they meet the filter criteria or not : before 5/6/08)
2. process with tFilterRow set to delete files created before 24 days
3. results of running the process defined in #4, where only the appropriate files are deleted (any files before 5/7/08)

Note:
tFileOutputDelimited_2 creates out_orig.txt                  (a list of all the files found in tFileList_3)
tFileOutputDelimited_1 creates out.txt                         (a list of all the files deleted by tFileDelete_1)
tFileOutputDelimited_3 creates out_notdeleted.txt        (a list of all the files rejected by tFilterRow_1)


I find it kind of odd that the advanced filter in tFilterRow_1 works where ### (in the quoted formula below) is <=24, but not greater...

((TalendDate.getCurrentDate().getTime()-(table1.mtime+86400000*###))>0

[img]http://www.talendforge.org/forum/img/members/1327/mini_12201_0 - Folder Before Job.JPG[/img][img]http://www.talendforge.org/forum/img/members/1327/mini_12201_1 - Job Setup.JPG[/img][img]http://www.talendforge.org/forum/img/members/1327/mini_12201_2 - Job Output.JPG[/img][img]http://www.talendforge.org/forum/img/members/1327/mini_12201_3 - Job Setup.JPG[/img][img]http://www.talendforge.org/forum/img/members/1327/mini_12201_4 - Job Output.JPG[/img]

Last edited by strpdsnk (2008-05-30 20:16:23)

Offline

#10 2008-05-30 21:28:24

strpdsnk
Member
328 posts

Re: delete files more than 45 days (example) old?

Ok, I figured out how to get around the problem I posted above:
1. set ### (in the post above) to a context variable (ex: context.NumDays_Past)
2. set the Type of that context variable to Double
      a. You could use Float or Long, in place of Double, here - they both worked for me
      b. Don't use Integer or Short, they cause all the files in the folder to delete.
      c. Don't use Big Integer, it causes an error (Exception in thread "main" java.lang.Error: Unresolved compilation problem: The operator * is undefined for the argument type(s) int, BigDecimal)


(TalendDate.getCurrentDate().getTime()-(86400000*context.NumDays_Past))>table1.mtime

Offline

#11 2012-02-28 04:42:07

ainiesta
Guest

Re: delete files more than 45 days (example) old?

Hi, I am having error to this context.NumDays_Past

nullpointerexception.

what values you put in context.NumDays_Past?

Regards.

#12 2014-01-09 11:53:57

jzhao_talend
Member
1 post

Re: delete files more than 45 days (example) old?

Hi,I have reproduce the problem.
This is cause by 60*60*24*1000*25 out of scope of Integer type(result is: -2134967296)
So the result alway false
Solution:change 60*60*24*1000*25 ---to---> 60*60*24*1000*25L(result is: 2160000000)
Change the result is long type.

Offline

#13 2014-07-14 11:02:10

RSH
Member
102 posts

Re: delete files more than 45 days (example) old?

Dear All,

I am trying to delete file which are 100 days old , I have used same approach which is mentioned in below email but I AM UNABLE TO UNDERSTAND FOLLOWING THINGS.

STEPS:
1) please check my log files which are in format of: (Name of the JOB_currentdate_time)
Example:==> Log_InsertDIMItems_2014-07-11-13-27-18
This file i have created on Friday 11th July
I want to delete files which are older than 100 days.

2) Screenshot of my job:

3) Code in tFilter Component: Advance Code:
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*24))>0

By using this query(24 days limit) then 0 files are deleted , but by using 25 all files are getting deleted which is confusing for me. Please advise where I am wrong ?

I am not getting how this is happening.

Can anyone please guide me on same.

My requirement is of deleting files which are 1oo days old .

Thanks and Regards
RSH
mini_140035_1.jpgmini_140035_2.jpgmini_140035_3.pngmini_140035_5.jpgmini_140035_6.jpg

Last edited by RSH (2014-07-14 11:08:59)

Offline

#14 2014-07-14 11:08:16

sanvaibhav
Member
1703 posts

Re: delete files more than 45 days (example) old?

Hi RSH,

tFileList-->tFileProperties-->tJavaRow-->tFileDelete

Use this flow, assign file names to context variables in tJavaRow based on your date condition and use that context variable in tFileDelete component for file to delete..

try this

vaibhav


DI DW Consultant and Trainer

Offline

#15 2014-07-14 11:13:59

RSH
Member
102 posts

Re: delete files more than 45 days (example) old?

Hi vaibhav,

Thank you for reply.
Its not possible to declare context for each file because we have near about 30 different names of files. Is there any possibility to manipulate on current snapshot which i provided?
I am not getting where I made mistake ?

Regards
RSH

Offline

#16 2014-07-14 11:24:36

RSH
Member
102 posts

Re: delete files more than 45 days (example) old?

If I use :
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*25))>0

All files are getting DELETED (--NOT SURE HOW)
Please find below is an snapshot of result:

mtime    mtime_string    GetDate    Calculated_Data_OF_Advance_Filter
1,40508E+12    Fri Jul 11 13:27:20 CEST 2014    1,40533E+12    2390162346
1,40508E+12    Fri Jul 11 13:27:20 CEST 2014    1,40533E+12    2390162867
1,40508E+12    Fri Jul 11 13:27:19 CEST 2014    1,40533E+12    2390163079
1,40508E+12    Fri Jul 11 13:27:20 CEST 2014    1,40533E+12    2390162220
1,40508E+12    Fri Jul 11 13:27:20 CEST 2014    1,40533E+12    2390162663


If i use
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*24))>0

NO file is deleted..

Last edited by RSH (2014-07-14 11:29:55)

Offline

#17 2014-07-14 11:40:29

sanvaibhav
Member
1703 posts

Re: delete files more than 45 days (example) old?

Hi,

mtime is unix timestamp in millisecond.. 86400000 represents one day

TalendDate.getCurrentDate().getTime()-86400000*2) - (input_row.mtime)>0

Above code deletes file which are older than two days.. you replace 2 by 100, then it should do the job... use wherever required in your design...

Try this.

vaibhav


DI DW Consultant and Trainer

Offline

#18 2014-07-14 12:06:47

RSH
Member
102 posts

Re: delete files more than 45 days (example) old?

Hi Vaibhav,

I used the query which you suggested, but still strange result.
All files from server got deleted sad
Not sure how this is working out:

Query:
(TalendDate.getCurrentDate().getTime()-86400000*100) - (File_prop.mtime)>0

Offline

#19 2014-07-15 05:04:28

RSH
Member
102 posts

Re: delete files more than 45 days (example) old?

Hello ,

Can anyone please guide me on my issue please ?

Thanks in advance.

Regards
RSH

Offline

#20 2014-07-15 06:43:08

umeshrakhe
Member
400 posts

Re: delete files more than 45 days (example) old?

can you try below code. it will help you on this issue.

use this code in If condition.

new Date(File_prop.mtime).before(TalendDate.addDate(TalendDate.getCurrentDate(), -44, "dd"))==true

you can convert File_prop.mtime date into Date format using new Date(File_prop.mtime) constructor then you can add or minus days from date and compare with current date .

Last edited by umeshrakhe (2014-07-15 07:16:55)


Umesh

Offline

Board footer

Talend Contributor Agreement - Talend Website Privacy Policy