Talend community forum / [resolved] tFlowMeterCatcher gets duration of the job?

klmc · 2010-02-24 03:10:46

Hi,

I am wondering which component I used in order to obtain the duration of the job? I used tflowMeterCatcher, I am able to obtain the job name, the start date & time, the total number of rows processed; but cannot obtain the duration of the job.

Below is my workflow:
tMySQLInput --> Main --> tMap --> tFlowMeter --> tOracleOutput --> Reject --> tFileOutputDelimited --> if --> tSendMail

tflowMeterCatcher --> tMap --> tOracleOuptut

Any help is really appreciated! Thanks!!

shong · 2010-02-24 03:21:22

Hello

I am wondering which component I used in order to obtain the duration of the job?

tChronometerStart and tChronometerStop fit your need.

Best regards

shong

klmc · 2010-02-26 01:55:07

Hi Shong,

Thanks again!! I used tChronometerStart & tChronometerStop; it gives the duration of a job. However, how do I link tStatsCatcher & tFlowMeterCatcher to obtain the following information on my job and output the data into a table in the database:

Job Name
Job Start Time
Job End Time
Row Processed

I tried to use tFlowMeterCatcher with tStatCatcher as a lookup with the tMap and join them with the job name. However, when executed only job name and the number of row processed are saved in the database. I am not able to obtain the Job Start Time & Job End Time? Is the tStatCatcher moment refer to Job Start Time or Job End Time?

Thanks again!

rdagher · 2010-02-26 17:58:32

Hi klmc,

One possibility of doing this is that in your tMap, set the startTime and endTime values in the output row which you can get from tChronometerStart and tChronometerStop.

However, I have previously raised a bug on those two components since I wasn't able to extract their values in a Java. But Shong suggested a workaround to get their values. For more information on this, check the forum topic http://talendforge.org/forum/viewtopic.php?id=9645

Hope this helps!

Rabih Dagher

klmc · 2010-02-27 03:15:58

Hi rdagher,

Thanks for your help! I created the flow as
tStatCatcher_1 --> Main --> tMap2 --> Main --> tOracleOutput_2

In tMap2, I created 2 variable expression:
globalMap.get("tChronometer_Start1")
globalMap.get("tChronometer_Stop1")

After running the process, the following data is saved in the database:
Job Name Start Time Stop Time Message Type Message
j_rating 1267235883244 1267235883603 end success

Can you or anyone please help. How can I generate the Start Time & Stop Time in Date format? In addition, I really don't know to join the tFlowMeterCatcher with tStatCatcher to get the total number of rows processed to the database.

Thanks again!!!

shong · 2010-02-27 03:20:16

Hello

How can I generate the Start Time & Stop Time in Date format?

Convert long type to date type on tMap, for example:
new java.util.Date(row1.startTime)

Best regards

shong

klmc · 2010-03-04 23:20:42

Hi Shong,

I tried the solution you mentioned in the thread; however, I encountered the following java exception error:

Starting job j_rating at 13:28 04/03/2010.

[statistics] connecting to socket on port 4119
[statistics] connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6671)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7099)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6943)
the end is near
Job j_rating ended at 13:28 04/03/2010. [exit code=4]

Can you please help me again!

Thanks!

Uploaded Images

Last edited by klmc (2010-03-04 23:24:48)

shong · 2010-03-05 06:53:25

Hello

In tMap2, I created 2 variable expression:
globalMap.get("tChronometer_Start1")
globalMap.get("tChronometer_Stop1")

1)The correct expression should be:
globalMap.get("tChronometerStart_1")
globalMap.get("tChronometerStop_1")
Please check it!
2)Why don't you define the data type as long/Long replace Ojbect when you define the two vars? and set the expression as:
(Long)globalMap.get("tChronometerStart_1")
(Long)globalMap.get("tChronometerStop_1")
then conver long to Date on output table:
new java.util.Date(Var.start_Time)

Best regards

shong

klmc · 2010-03-08 20:58:39

Hi Shong,

I did what you suggested; however, I still cannot obtain the tChronometer Start & Stop time in DATE format. When I run the job, I encountered the following error:

Starting job j_rating at 11:52 08/03/2010.

[statistics] connecting to socket on port 3992
[statistics] connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6681)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7124)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6968)
the end is near
Job j_rating ended at 11:52 08/03/2010. [exit code=4]

Did I miss anything? Can you please help!
Thanks a bunch!

Uploaded Images

shong · 2010-03-09 03:27:55

Hello
Does the tChronometerStart_1 and tChronometerStop_1 components exist in your job? For example:
tChronometerStart_1
|
onSubjobok
|
tJava( or any subJobs)
|
onsubJobok
|
tChronometerStop_1
tStatCatcher_1 --> Main --> tMap2 --> Main --> tOracleOutput_2

Best regards
Shong

klmc · 2010-03-09 03:50:25

Hi Shong,

The tChronometerStart_1, tChronometerStop_1 and tStatCatcher_1 component exist in my job. Did I miss anything?

Thanks again!

Uploaded Images

shong · 2010-03-09 04:04:39

Hello
Don't define the vars on tMap, use the global var directly on expression, for example:
new java.util.Date((Long)globalMap.get("tChronometerStart_1"))

Best regards
Shong

klmc · 2010-03-09 18:49:46

Hi Shong,

Thanks for the quick response! I tried as what you suggested and add the coding in the expression; however when I ran the job, it still gave the following error:

Starting job j_rating at 09:44 09/03/2010.

[statistics] connecting to socket on port 4209
[statistics] connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6683)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7114)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6958)
the end is near
Job j_rating ended at 09:44 09/03/2010. [exit code=4]

Uploaded Images

rdagher · 2010-03-09 19:06:55

klmc,

I think I know why you're getting the null exception. It might be that when a log event is triggered, the durations in your tChronometerStart and tChronometerStop are not yet set. Keep in mind that those are set after the component finished executing.

Try accessing them after a tPostJob component:

tPostJob ------onComponentOk-----> tFixedFlowInput ------main---------> tOracleOutput

In your tFixedFlowInput, you access the tChronometerStart and tChronometerStop. This will make sure that they finished executing.

Hope this helps!

Rabih

klmc · 2010-03-10 03:07:00

Hi Rabih,

Thanks for your help! I will definitely try the method you suggested; however, I have a question regarding the tPostJob component! Does tPostJob component trigger only when a job complete without any errors?

Really appreciate everyone who help me in this thread!!

rdagher · 2010-03-10 14:48:59

klmc, the tPostJob component is triggered after the main job completes, no matter if there was or not an error.

dna1718 · 2012-02-01 12:35:10

I need to extract basic information about a job every time its run and output this into a DB.

I need information such as the following:

Username - of the person running the Job
Project Name
Job Name
Start Time
End Time
Row Count
Result – Success or Failure.

What would I create this Job very simply?

Announcement

Unanswered posts

#1 2010-02-24 03:10:46

[resolved] tFlowMeterCatcher gets duration of the job?

#2 2010-02-24 03:21:22

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#3 2010-02-26 01:55:07

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#4 2010-02-26 17:58:32

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#5 2010-02-27 03:15:58

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#6 2010-02-27 03:20:16

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#7 2010-03-04 23:20:42

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#8 2010-03-05 06:53:25

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#9 2010-03-08 20:58:39

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#10 2010-03-09 03:27:55

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#11 2010-03-09 03:50:25

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#12 2010-03-09 04:04:39

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#13 2010-03-09 18:49:46

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#14 2010-03-09 19:06:55

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#15 2010-03-10 03:07:00

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#16 2010-03-10 14:48:59

Re: [resolved] tFlowMeterCatcher gets duration of the job?

#17 2012-02-01 12:35:10

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Board footer