Skip to main content
In this guide, you’ll learn how to ship Jenkins audit trail logs from your controller to an external Elasticsearch cluster hosted on Elastic Cloud Observability. We’ll cover:
  1. Setting up Elastic Cloud Observability
  2. Installing the Elastic Agent on your Jenkins controller
  3. Configuring the agent to read Jenkins audit logs
  4. Verifying audit entries in Kibana

​
1. Set Up Elastic Cloud Observability

  1. Sign up for an Elastic Cloud trial on elastic.co.
  2. In the Elastic Cloud console, go to Observability → Logs and click Collect and analyze logs.
The image shows a webpage from Elastic's observability onboarding, offering options to collect and analyze logs, monitor application performance, and monitor infrastructure. There are multiple browser tabs open at the top.
  1. Choose Host system logs as your resource type.
The image shows a webpage from Elastic, asking "What type of resource are you monitoring?" with options for monitoring system logs, log files, OpenTelemetry, Azure, AWS, and Google Cloud Platform.
  1. Follow the prompts to install the Elastic Agent and generate an API key for onboarding.
The image shows a webpage from Elastic's Kibana interface, guiding users on how to add observability data by collecting system logs. It includes instructions for installing an Elastic Agent and mentions an API key creation.
Your Elastic Cloud trial includes full access to Observability features. Save the API key securely, as you’ll need it for agent enrollment.

​
2. Install the Elastic Agent on the Jenkins Controller

On your Jenkins controller shell, download and run the provided installation script. Replace the placeholders with your API key and Cloud endpoint URL: 
curl \
  https://<your-cloud-endpoint>/plugins/observabilityOnboarding/assets/standalone_agent_setup.sh \
  -o standalone_agent_setup.sh && \
sudo bash standalone_agent_setup.sh \
  YOUR_API_KEY \
  https://<your-cloud-endpoint>/internal/observability_onboarding \
  8.15.3 \
  <ENROLLMENT_ID>
This script will:
  • Download and unpack the Elastic Agent
  • Enroll the agent using your API key and endpoint
  • Place the main config at /opt/Elastic/Agent/elastic-agent.yml

​
3. Configure the Agent to Stream Audit Logs

Edit the agent configuration as root: 
sudo vi /opt/Elastic/Agent/elastic-agent.yml

​
3.1 Define the Elasticsearch Output

Locate (or add) the outputs section and update it with your Cloud URL and API key: 
outputs:
  default:
    type: elasticsearch
    hosts: ['https://your-cluster-id.us-central1.gcp.cloud.es.io:443']
    api_key: 'YOUR_API_KEY'

​
3.2 Add Log File Inputs

Below outputs:, include inputs to capture Jenkins audit files from /var/log/jenkins/custom*: 
inputs:
  - id: jenkins-audit-logs
    type: logfile
    data_stream:
      dataset: system.auth
      type: logs
    streams:
      - id: jenkins-audit-logs-stream
        type: logs
        data_stream:
          namespace: default
        paths:
          - /var/log/jenkins/custom*
        exclude_files:
          - '*.gz'
        multiline:
          pattern: '^'
          match: after
        tags: ['jenkins','audit']

  - id: system-messages
    type: logfile
    data_stream:
      dataset: system.syslog
      type: logs
    streams:
      - id: system-messages-stream
        type: logs
        data_stream:
          namespace: default
        paths:
          - /var/log/messages*
          - /var/log/syslog*
          - /var/log/system*
        exclude_files:
          - '*.gz'
        multiline:
          pattern: '^'
          match: after
Ensure indentation and quotation marks are correct in elastic-agent.yml. A YAML syntax error will prevent the agent from starting.

​
3.3 Restart the Agent

Apply your changes: 
sudo systemctl restart elastic-agent

​
4. Verify Logs in Kibana

  1. In Kibana, open Observability → Logs.
  2. Refresh the interface; you should see Jenkins audit events streaming in.
The image shows a webpage from Elastic's Kibana interface indicating that logs are being shipped, with options for downloading a config file, troubleshooting, and exploring logs.
  1. Click Explore logs to filter, search, and analyze your audit data.
The image shows a screenshot of the Elastic Observability Logs Explorer interface, displaying log details and content breakdown for a specific time range. It includes fields like error messages, service infrastructure, and cloud provider information.

​
Sample Audit Entries

pipeline logs-system.auth-1.62.1 failed with message: Provided Grok expressions do not match field value: [Nov 10, 2024 3:33:43,556 PM] job/monitor-jenkins/ #35 Started by user siddharth, Parameters:[]
Or on your controller: 
Nov 10, 2024  3:30:09,212 PM job/monitor-jenkins/ #31 Started by user siddharth, Parameters:[]
Nov 10, 2024  3:33:44,394 PM monitor-jenkins #36 Started by user siddharth, Parameters:[] on node #unknown# started at 2024-11-10T15:33:43Z completed in 929ms completed: SUCCESS
Continue generating Jenkins audit events; they’ll flow automatically into your Elastic Cloud instance.

Watch Video