> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Troubleshooting

> Practical guidance for monitoring and troubleshooting NGINX including access and error logs, stub_status metrics, observability integrations, centralized logging, config validation and safe reloads.

In this lesson we'll cover practical techniques to monitor and troubleshoot NGINX. Topics include access and error logs, the lightweight `stub_status` endpoint, integrating with observability platforms, collecting and querying logs, and safe configuration reloads. These are the core practices for maintaining uptime, diagnosing issues, and feeding monitoring dashboards (Prometheus, Grafana, Datadog, etc.).

## Logging (Access logs)

NGINX writes request details to access logs via its HTTP logging module. Access logs are essential for troubleshooting traffic patterns, identifying client errors (4xx), server errors (5xx), and measuring performance.

Define a custom log format with `log_format` inside the `http` block. For example:

```nginx theme={null}
http {
    log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
}
```

Enable access logging with the `access_log` directive (default locations are typically `/var/log/nginx/access.log`).

Common built-in variables used in log formats:

| Variable | Meaning |
| - | - |
| `\$remote_addr` | Client IP address |
| `\$remote_user` | Authenticated user (empty if not used) |
| `\$time_local` | Local timestamp of the request |
| `\$request` | Request line (method, URI, protocol) |
| `\$status` | HTTP response status code |
| `\$body_bytes_sent` | Bytes sent to the client |
| `\$http_referer` | Referring URL or `-` |
| `\$http_user_agent` | User-Agent header |
| `\$http_x_forwarded_for` | X-Forwarded-For header (useful behind proxies) |

![A slide titled "Log Format Options" listing common web server log fields (e.g., remote\_addr, remote\_user, time\_local, request, status, http\_referer, \$http\_user\_agent) with brief descriptions. A small © KodeKloud credit appears at the bottom.](https://mintcdn.com/kodekloud-c4ac6d9a/ij3FNIPjp1w0Myd0/images/Nginx-for-Beginners/Performance/Monitoring-Troubleshooting/log-format-options-web-server-fields.jpg?fit=max\&auto=format\&n=ij3FNIPjp1w0Myd0\&q=85\&s=e092bd983323f3ff2e5644408ee3bc37)

Example: enable access logs in a server block and tail them in real time.

Configuration file:

```bash theme={null}
$ cat /etc/nginx/sites-available/example.conf
```

```nginx theme={null}
server {
    listen 80;
    server_name example.com www.example.com;

    root /var/www/example.com/html;
    index index.html;

    access_log /var/log/nginx/access.log;

    location / {
        try_files $uri $uri/ =404;
    }
}
```

Tailing the access log:

```bash theme={null}
$ cd /var/log/nginx/
$ tail -f access.log
```

Sample access log entry produced by the `custom` format above:

```text theme={null}
192.168.1.1 - - [26/Jan/2025:14:23:35 +0000] "GET /images/logo.png HTTP/1.1" 200 512 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
```

How to use access logs when troubleshooting:

* Reproduce the issue while tailing logs (`tail -f`) to ensure requests reach this host.
* If no entry appears, verify DNS, load balancer, or upstream/proxy routing.
* Add fields (for example `\$host`, `\$request_time`, `\$upstream_addr`) to your log format to capture additional context for later analysis.

Reading a log entry (quick guide)

* First token: remote IP (e.g., `192.168.1.1`).
* Next token: remote user (`-` when unauthenticated).
* Timestamp: in square brackets.
* Request line: double-quoted (method, URI, HTTP version).
* Status code: e.g., `200`, `404`.
* Bytes sent: numeric value.
* Referer and User-Agent follow, possibly `-` if not present.

## Error logs

NGINX error logs record problems NGINX encounters (configuration errors, file access errors, upstream timeouts). High volumes or repeated patterns indicate configuration or runtime issues. Look in `/var/log/nginx/error.log` (path may vary by distribution).

## NGINX stub\_status (basic runtime metrics)

The `stub_status` module exposes lightweight runtime metrics: active connections, accepts, handled, total requests, and the number of connections in reading/writing/waiting states. Keep this endpoint restricted and avoid logging access requests to it.

Do not expose `stub_status` publicly. Restrict access (e.g., `allow 127.0.0.1; deny all;`) and disable access logging for the endpoint to avoid polluting logs.

Example `stub_status` configuration:

```nginx theme={null}
server {
    listen 81;
    server_name example.com;

    location /nginx_status {
        stub_status;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}
```

Notes:

* Using a nonstandard port like `81` reduces accidental exposure.
* `access_log off;` prevents internal monitoring calls from generating log noise.
* `allow`/`deny` restricts who can query the endpoint.

Query locally:

```bash theme={null}
$ curl localhost:81/nginx_status
```

Sample output:

```text theme={null}
Active connections: 309
server accepts handled requests
16630948 16630948 31070465
Reading: 11 Writing: 218 Waiting: 38
```

This shows active connections plus cumulative accepts/handled/requests and current connection states.

## Monitoring platforms

Observability platforms can collect metrics and logs for richer dashboards and alerting. Common solutions:

* Prometheus + Grafana (popular open-source stack)
* Datadog (SaaS monitoring + logs)
* New Relic, Dynatrace (APM + infrastructure)
* ELK/EFK stacks for centralized logs

![A slide titled "Monitoring Tools" showing a boxed area with logos for common observability platforms. Visible logos include Prometheus, Grafana, Datadog, Dynatrace, and New Relic.](https://mintcdn.com/kodekloud-c4ac6d9a/ij3FNIPjp1w0Myd0/images/Nginx-for-Beginners/Performance/Monitoring-Troubleshooting/monitoring-tools-observability-platform-logos.jpg?fit=max\&auto=format\&n=ij3FNIPjp1w0Myd0\&q=85\&s=56d4add6ce5b47abe290ae2d7c419198)

Examples:

* Prometheus uses an exporter (e.g., nginx-vts-exporter or the official Prometheus nginx exporter) to scrape metrics, and Grafana visualizes them.
* Datadog installs an agent that collects system metrics (CPU, memory, disk, network) and NGINX-specific metrics (connections, requests, upstream stats).

![A Datadog system metrics dashboard for an nginx host showing multiple widgets and time-series charts. It displays CPU usage, a processes memory treemap, load averages, disk latency, network traffic, and memory/disk breakdowns.](https://mintcdn.com/kodekloud-c4ac6d9a/ij3FNIPjp1w0Myd0/images/Nginx-for-Beginners/Performance/Monitoring-Troubleshooting/datadog-nginx-system-metrics-dashboard.jpg?fit=max\&auto=format\&n=ij3FNIPjp1w0Myd0\&q=85\&s=3d5b4dcc88aca5c50e467c2e89b9a045)

NGINX Plus (commercial) exposes additional metrics such as per-upstream and per-status code metrics, cache stats, and health check details which many agents can ingest out of the box.

## Collecting and searching logs

Centralized logging (ELK/EFK, Splunk, Datadog Logs) makes it easy to query and correlate large volumes of logs with metrics and traces. Example application-level errors you might aggregate:

```text theme={null}
2018/05/16 22:02:02 [error] 1244#1244: *203 open() "/var/www/html/order/2480" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /order/2480 HTTP/1.1"
2018/05/16 22:02:02 [error] 1244#1244: *202 open() "/var/www/html/send/notice" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /send/notice HTTP/1.1"
2018/05/16 22:02:02 [error] 1244#1244: *186 open() "/var/www/html/bad" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /bad HTTP/1.1"
```

Correlate these with access logs and metrics to identify root causes (missing files, application routes failing, 502/504 upstream errors).

## Configuration syntax validation and safe reloads

Validate configuration changes before applying them:

```bash theme={null}
$ nginx -t
```

Successful test:

```text theme={null}
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

If the test fails, `nginx -t` shows the file and line number with an error message:

```text theme={null}
nginx: [emerg] unknown directive "worker_connection" in /etc/nginx/nginx.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed
2025/01/29 23:46:03 [emerg] 71439#71439: "server" directive is not allowed here in /etc/nginx/sites-enabled/default:39
```

After a successful test, reload without restarting the master process to avoid dropping connections:

```bash theme={null}
$ nginx -s reload
```

Always run `nginx -t` before reloading. Use `nginx -s reload` to apply changes without dropping active connections — restarting can cause a brief interruption.

## Quick troubleshooting commands

| Purpose | Command |
| -: | :- |
| Validate config | `nginx -t` |
| Reload config gracefully | `nginx -s reload` |
| Check systemd service status | `systemctl status nginx` |
| Start service | `systemctl start nginx` |
| Tail access log | `tail -f /var/log/nginx/access.log` |
| Query stub\_status locally | `curl localhost:81/nginx_status` |
| Check that NGINX responds locally | `curl --head localhost` |

## Checking service status

Check the systemd service to verify NGINX is running:

```bash theme={null}
$ systemctl status nginx
```

Sample active output:

```text theme={null}
● nginx.service - A high performance web server and a reverse proxy server
  Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Active: active (running) since Tue 2025-01-28 00:18:10 UTC; 1 day 23h ago
   Docs: man:nginx(8)
 Main PID: 2164 (nginx)
   Tasks: 2 (limit: 1130)
  Memory: 24.7M
   CPU: 16.367s
  CGroup: /system.slice/nginx.service
          ├─2164 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
          └─73741 nginx: worker process
```

If the service is `inactive (dead)`, try starting it:

```bash theme={null}
$ systemctl start nginx
```

If it fails to start, examine the journal and NGINX error logs. Common causes include configuration mistakes and port conflicts (e.g., Apache also listening on port 80).

## Verifying NGINX is serving requests

Quick local check:

```bash theme={null}
$ curl --head localhost
```

Example response:

```http theme={null}
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 30 Jan 2025 00:10:05 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 28 Jan 2025 00:18:09 GMT
Connection: keep-alive
ETag: "67982241-264"
Accept-Ranges: bytes
```

A `200 OK` from localhost indicates NGINX serves requests; if remote clients cannot connect, check firewall rules, load balancer, DNS, or network routing.

## Firewall and network considerations

Ensure ports 80 (HTTP) and 443 (HTTPS) are open at both the cloud/provider level and the host firewall.

Ubuntu (ufw):

```bash theme={null}
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload
```

RHEL / CentOS / Fedora (firewalld):

```bash theme={null}
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --reload
```

If either cloud or host firewall blocks these ports, external traffic will not reach your site.

## Closing

This article covered NGINX access and error logging, the `stub_status` endpoint for lightweight runtime metrics, integrating with observability platforms (Prometheus/Grafana, Datadog, etc.), centralized log collection, and safe procedures for validating and reloading configuration. For production systems, centralize logs, export NGINX metrics to a monitoring system, and alert on key signals (status code spikes, latency increases, resource exhaustion) to respond proactively.

Further reading and references:

* [NGINX Official Docs](https://nginx.org/en/docs/)
* [Prometheus Monitoring NGINX (exporter)](https://github.com/nginxinc/nginx-prometheus-exporter)
* [Datadog NGINX Integration](https://docs.datadoghq.com/integrations/nginx/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/nginx-for-beginners/module/4a5db5c4-df5f-4291-84f0-013d1c4ce235/lesson/b7d13e4f-c250-4bf4-a964-c072516a654a)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.