You are not logged in.

Unanswered posts



Important! This site has been replaced. All content here is read-only. Please visit our brand-new community at https://community.talend.com/. We look forward to hearing from you there!



#1 2017-04-06 01:16:44

stamrake
Member
14 posts

stamrake said:

Help on Log4j setting for different LogStash servers

Hi All,
I need help on setting Log4j setting for Different Logstash servers.

Scenario:
I have 3 different Logstash servers, and 3 different Job servers.
Logstash server is different than Job Server.
I have added Logstash appender in Log4j setting of my Studio. (See below screen shot)



Issue: 
I have set log4j in my studio only for 1 logstash server not for all 3.
I am running the job in 3 different job servers. 
I don't want to change the setting for each of the servers (DEV, QA and PROD)

If i set up 3 logstash apenders for each logstash servers, logs will be going to (DEV, QA and PROD) together, which i dont want.


Question:
Is there any way to setup LOG4J so that i get the logs when i run the job in any of the job servers.
I want the DEV logs, when i run the job in DEV
QA logs when i run the job in QA, like wise for PROD. 


Note: 
Logging works fine if i setup 1 Logstash appender in my LOG4J setting.
I can't use "Localhost" as Logstash and Job servers are different.

Please let me know if you have any idea on this.

Offline

#2 2017-04-06 02:58:59

vapukov
Member
470 posts

vapukov said:

Re: Help on Log4j setting for different LogStash servers

first of all - screenshot not attached

but in any case - You can define context variable for any settings, and create 3 context group - 1 per Job server
depending from server You choose proper context group for Job

Offline

#3 2017-04-06 04:39:11

stamrake
Member
14 posts

stamrake said:

Re: Help on Log4j setting for different LogStash servers

Hi,
Thanks for the reply,
Please find below screen shot




I tried using context variable, but if i provide context variable as below, It is not working in LOG4J setting.

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="context.logstash_hostname"/>
        <param name="Port" value="8052"/>
    </appender>
mini_Logstash.png_20170405-2039.png

Last edited by stamrake (2017-04-06 04:40:06)

Offline

#4 2017-04-06 04:58:59

vapukov
Member
470 posts

vapukov said:

Re: Help on Log4j setting for different LogStash servers

try change:

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="context.logstash_hostname"/>
        <param name="Port" value="8052"/>
</appender>

to 

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>

I have only 1 log stash at the moment, so it hard to test, but at least it not return errors

Last edited by vapukov (2017-04-06 05:00:11)

Offline

#5 2017-04-06 06:09:06

stamrake
Member
14 posts

stamrake said:

Re: Help on Log4j setting for different LogStash servers

I tried appender using context variable as below:
<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>


It is not accepting context variable in Studio log4j setting.
It is not giving error but logs are not getting logged.

Offline

#6 2017-04-06 17:27:39

stamrake
Member
14 posts

stamrake said:

Re: Help on Log4j setting for different LogStash servers

Few things i wanted to confirm:

1. Are you able to get component level logs using context variable in log4j setting.
2. I am giving logstash appender in Studio log4j setting (See attached), SO is this the only way to get component level logs or is there any other way i can add logstash appender to get component level logs.

mini_Logstash.png_20170406-0927.png

Offline

#7 2017-04-07 21:35:52

stamrake
Member
14 posts

stamrake said:

Re: Help on Log4j setting for different LogStash servers

I got the solution for this issue:
For each of the 3 job servers, we need to include Logserver IP address in the "hosts file" located at "/ect/hosts"
What i did is as below

  • Added logstash IP address in Job Server 1 "hosts file"
    XX.XX.XXX.XXX    talend_logstash.aexp.com

  • Added logstash IP address in Job Server 2 "hosts file"
    YY.YY.YYY.YYY    talend_logstash.aexp.com

  • Added logstash IP address in Job Server 3 "hosts file"
    ZZ.ZZ.ZZZ.ZZZ    talend_logstash.aexp.com

Now i am using below details in logstash appender:

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="talend_logstash.aexp.com"/>
        <param name="Port" value="8052"/>

    </appender>

Offline

Board footer

Talend Contributor Agreement - Talend Website Privacy Policy