Talend community forum / TAC server extremely chatty with subversion

tomasso · 2012-04-13 21:45:55

Our sysadmins came to me this week and asked why the admin server is making so many http requests to subversion.

I explained to them that Talend is always checking for updates from svn, but then I asked more about frequency.

In about 5.5 days, our dev server has made almost 2 million requests to svn (prod is a little more than 1/3 of that number). Apparently, this has been going on since we stood up our instances

The majority of the requests are OPTIONS or PROPFIND.

Now I would expect to see requests from time to time (for example, I expect multiple calls to svn when a job that is set to run latest version and/or regenerate on change is triggered), but this appears to be much more than that.

I'm wondering if anyone else has seen such a level of chattiness with svn or knows how to address this.

pedro · 2012-04-16 05:39:45

Hi

Please check the three parameters in TAC-->Configuration.
TAC shouldn't communicate with SVN server so frequently.

Or check the error log of TAC whether there are repeated errors.

Regards,
Pedro

tomasso · 2012-04-16 17:05:51

The only three parameters for Svn in TAC==>Configuration are:
* Server location url
* Username
* Password

All are verified.

I've looked at the TAC log files, and there have been less than a dozen errors during the same timeframe.

Are there settings in configuration.properties that might cause this?

amaumont · 2012-04-20 17:15:04

By default in the 'configuation.properties', the settings about the automatic SVN refresh has a period of 5 minutes.
At each period, there are the following operations for each project:
- get all the names of the branches for the project
- for each branch, send svn command "SVN INFO" to know if the repository has changed
- if the reprository has changed (checking revision), do an "SVN UP" to update the files in the local reprository

The period can be changed in the 'configuration.properties' file with the property:

Code:

# Number of seconds between each ProcessItem refresh, unit in second.
scheduler.conf.DataProjectRefreshFrequency=300

Therefore, as you noticed other svn requests are required when tasks are launched according their configuration.
For example when a task has the option "Regenerate on change" enabled or the version equals to "Latest", we need to update the local reprository to know if changes have been done on the job attached to the tasks which requires that the job should be regenerated and deployed again before its launch.

I think these are the main operations which do requests on SVN when no one user is using the TAC GUI.

I hope this will answer to your question.

Best regards,

Axel

Last edited by amaumont (2012-04-20 17:29:11)

amaumont · 2012-05-24 18:24:18

For your information, we detected an issue which could explain the issue described by tomasso.

The symptoms are the following:
- more than 10 time slower than normal response time when getting tasks list
- too many requests on svn when getting task list or each time a tasks is modified (status, and any other fields)

Please look at the issue http://jira.talendforge.org/browse/TDI-21147

This issue exists since 5.0.0 and will be fixed for future 5.0.3 and 5.1.1.

It is possible to create a patch for this issue.

Announcement

Unanswered posts

#1 2012-04-13 21:45:55

TAC server extremely chatty with subversion

#2 2012-04-16 05:39:45

Re: TAC server extremely chatty with subversion

#3 2012-04-16 17:05:51

Re: TAC server extremely chatty with subversion

#4 2012-04-20 17:15:04

Re: TAC server extremely chatty with subversion

Code:

#5 2012-05-24 18:24:18

Re: TAC server extremely chatty with subversion

Board footer