Monitoring Troubleshooting

In this lesson we’ll cover practical techniques to monitor and troubleshoot NGINX. Topics include access and error logs, the lightweight stub_status endpoint, integrating with observability platforms, collecting and querying logs, and safe configuration reloads. These are the core practices for maintaining uptime, diagnosing issues, and feeding monitoring dashboards (Prometheus, Grafana, Datadog, etc.).

Logging (Access logs)

NGINX writes request details to access logs via its HTTP logging module. Access logs are essential for troubleshooting traffic patterns, identifying client errors (4xx), server errors (5xx), and measuring performance. Define a custom log format with log_format inside the http block. For example:

http {
    log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
}

Enable access logging with the access_log directive (default locations are typically /var/log/nginx/access.log). Common built-in variables used in log formats:

Variable	Meaning
`\$remote_addr`	Client IP address
`\$remote_user`	Authenticated user (empty if not used)
`\$time_local`	Local timestamp of the request
`\$request`	Request line (method, URI, protocol)
`\$status`	HTTP response status code
`\$body_bytes_sent`	Bytes sent to the client
`\$http_referer`	Referring URL or `-`
`\$http_user_agent`	User-Agent header
`\$http_x_forwarded_for`	X-Forwarded-For header (useful behind proxies)

A slide titled "Log Format Options" listing common web server log fields (e.g., remote_addr, remote_user, time_local, request, status, http_referer, $http_user_agent) with brief descriptions. A small © KodeKloud credit appears at the bottom.

Example: enable access logs in a server block and tail them in real time. Configuration file:

$ cat /etc/nginx/sites-available/example.conf

server {
    listen 80;
    server_name example.com www.example.com;

    root /var/www/example.com/html;
    index index.html;

    access_log /var/log/nginx/access.log;

    location / {
        try_files $uri $uri/ =404;
    }
}

Tailing the access log:

$ cd /var/log/nginx/
$ tail -f access.log

Sample access log entry produced by the custom format above:

192.168.1.1 - - [26/Jan/2025:14:23:35 +0000] "GET /images/logo.png HTTP/1.1" 200 512 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

How to use access logs when troubleshooting:

Reproduce the issue while tailing logs (tail -f) to ensure requests reach this host.
If no entry appears, verify DNS, load balancer, or upstream/proxy routing.
Add fields (for example \$host, \$request_time, \$upstream_addr) to your log format to capture additional context for later analysis.

Reading a log entry (quick guide)

First token: remote IP (e.g., 192.168.1.1).
Next token: remote user (- when unauthenticated).
Timestamp: in square brackets.
Request line: double-quoted (method, URI, HTTP version).
Status code: e.g., 200, 404.
Bytes sent: numeric value.
Referer and User-Agent follow, possibly - if not present.

Error logs

NGINX error logs record problems NGINX encounters (configuration errors, file access errors, upstream timeouts). High volumes or repeated patterns indicate configuration or runtime issues. Look in /var/log/nginx/error.log (path may vary by distribution).

NGINX stub_status (basic runtime metrics)

The stub_status module exposes lightweight runtime metrics: active connections, accepts, handled, total requests, and the number of connections in reading/writing/waiting states. Keep this endpoint restricted and avoid logging access requests to it.

Do not expose stub_status publicly. Restrict access (e.g., allow 127.0.0.1; deny all;) and disable access logging for the endpoint to avoid polluting logs.

Example stub_status configuration:

server {
    listen 81;
    server_name example.com;

    location /nginx_status {
        stub_status;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

Notes:

Using a nonstandard port like 81 reduces accidental exposure.
access_log off; prevents internal monitoring calls from generating log noise.
allow/deny restricts who can query the endpoint.

Query locally:

$ curl localhost:81/nginx_status

Sample output:

Active connections: 309
server accepts handled requests
16630948 16630948 31070465
Reading: 11 Writing: 218 Waiting: 38

This shows active connections plus cumulative accepts/handled/requests and current connection states.

Monitoring platforms

Observability platforms can collect metrics and logs for richer dashboards and alerting. Common solutions:

Prometheus + Grafana (popular open-source stack)
Datadog (SaaS monitoring + logs)
New Relic, Dynatrace (APM + infrastructure)
ELK/EFK stacks for centralized logs

A slide titled "Monitoring Tools" showing a boxed area with logos for common observability platforms. Visible logos include Prometheus, Grafana, Datadog, Dynatrace, and New Relic.

Examples:

Prometheus uses an exporter (e.g., nginx-vts-exporter or the official Prometheus nginx exporter) to scrape metrics, and Grafana visualizes them.
Datadog installs an agent that collects system metrics (CPU, memory, disk, network) and NGINX-specific metrics (connections, requests, upstream stats).

A Datadog system metrics dashboard for an nginx host showing multiple widgets and time-series charts. It displays CPU usage, a processes memory treemap, load averages, disk latency, network traffic, and memory/disk breakdowns.

NGINX Plus (commercial) exposes additional metrics such as per-upstream and per-status code metrics, cache stats, and health check details which many agents can ingest out of the box.

Collecting and searching logs

Centralized logging (ELK/EFK, Splunk, Datadog Logs) makes it easy to query and correlate large volumes of logs with metrics and traces. Example application-level errors you might aggregate:

2018/05/16 22:02:02 [error] 1244#1244: *203 open() "/var/www/html/order/2480" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /order/2480 HTTP/1.1"
2018/05/16 22:02:02 [error] 1244#1244: *202 open() "/var/www/html/send/notice" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /send/notice HTTP/1.1"
2018/05/16 22:02:02 [error] 1244#1244: *186 open() "/var/www/html/bad" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /bad HTTP/1.1"

Correlate these with access logs and metrics to identify root causes (missing files, application routes failing, 502/504 upstream errors).

Configuration syntax validation and safe reloads

Validate configuration changes before applying them:

$ nginx -t

Successful test:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If the test fails, nginx -t shows the file and line number with an error message:

nginx: [emerg] unknown directive "worker_connection" in /etc/nginx/nginx.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed
2025/01/29 23:46:03 [emerg] 71439#71439: "server" directive is not allowed here in /etc/nginx/sites-enabled/default:39

After a successful test, reload without restarting the master process to avoid dropping connections:

$ nginx -s reload

Always run nginx -t before reloading. Use nginx -s reload to apply changes without dropping active connections — restarting can cause a brief interruption.

Quick troubleshooting commands

Purpose	Command
Validate config	`nginx -t`
Reload config gracefully	`nginx -s reload`
Check systemd service status	`systemctl status nginx`
Start service	`systemctl start nginx`
Tail access log	`tail -f /var/log/nginx/access.log`
Query stub_status locally	`curl localhost:81/nginx_status`
Check that NGINX responds locally	`curl --head localhost`

Checking service status

Check the systemd service to verify NGINX is running:

$ systemctl status nginx

Sample active output:

● nginx.service - A high performance web server and a reverse proxy server
  Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Active: active (running) since Tue 2025-01-28 00:18:10 UTC; 1 day 23h ago
   Docs: man:nginx(8)
 Main PID: 2164 (nginx)
   Tasks: 2 (limit: 1130)
  Memory: 24.7M
   CPU: 16.367s
  CGroup: /system.slice/nginx.service
          ├─2164 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
          └─73741 nginx: worker process

If the service is inactive (dead), try starting it:

$ systemctl start nginx

If it fails to start, examine the journal and NGINX error logs. Common causes include configuration mistakes and port conflicts (e.g., Apache also listening on port 80).

Verifying NGINX is serving requests

Quick local check:

$ curl --head localhost

Example response:

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 30 Jan 2025 00:10:05 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 28 Jan 2025 00:18:09 GMT
Connection: keep-alive
ETag: "67982241-264"
Accept-Ranges: bytes

A 200 OK from localhost indicates NGINX serves requests; if remote clients cannot connect, check firewall rules, load balancer, DNS, or network routing.

Firewall and network considerations

Ensure ports 80 (HTTP) and 443 (HTTPS) are open at both the cloud/provider level and the host firewall. Ubuntu (ufw):

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload

RHEL / CentOS / Fedora (firewalld):

sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --reload

If either cloud or host firewall blocks these ports, external traffic will not reach your site.

Closing

This article covered NGINX access and error logging, the stub_status endpoint for lightweight runtime metrics, integrating with observability platforms (Prometheus/Grafana, Datadog, etc.), centralized log collection, and safe procedures for validating and reloading configuration. For production systems, centralize logs, export NGINX metrics to a monitoring system, and alert on key signals (status code spikes, latency increases, resource exhaustion) to respond proactively. Further reading and references:

Introduction

Install & Config

Intermediate Config

Security

Performance

Conclusion

Monitoring Troubleshooting

Logging (Access logs)

Error logs

NGINX stub_status (basic runtime metrics)

Monitoring platforms

Collecting and searching logs

Configuration syntax validation and safe reloads

Quick troubleshooting commands

Checking service status

Verifying NGINX is serving requests

Firewall and network considerations

Closing

Watch Video

​Logging (Access logs)

​Error logs

​NGINX stub_status (basic runtime metrics)

​Monitoring platforms

​Collecting and searching logs

​Configuration syntax validation and safe reloads

​Quick troubleshooting commands

​Checking service status

​Verifying NGINX is serving requests

​Firewall and network considerations

​Closing

Watch Video

Logging (Access logs)

Error logs

NGINX stub_status (basic runtime metrics)

Monitoring platforms

Collecting and searching logs

Configuration syntax validation and safe reloads

Quick troubleshooting commands

Checking service status

Verifying NGINX is serving requests

Firewall and network considerations

Closing