Monitoring Troubleshooting

This guide explains practical ways to monitor and troubleshoot NGINX. It covers access and error logs, the built-in stub_status endpoint, integration options with monitoring platforms, and common runtime checks and commands to validate changes safely.

Logging

NGINX produces two primary log streams:

Access logs — record every incoming HTTP request (useful for traffic analysis, response codes, request sizes, user agents, and referers).
Error logs — record server-side errors and configuration/runtime issues.

You can customize the access log format with the log_format directive inside the http block. See the NGINX log module docs for full details: https://nginx.org/en/docs/http/ngx_http_log_module.html Example custom log format:

http {
    log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
}

Choose the variables that match your analysis and monitoring requirements (for example, add $host to capture the requested host header). Common log variables

A presentation slide titled "Log Format Options" that lists common web-server log variables (e.g., remote_addr, remote_user, time_local, request, $status) with short descriptions. The content is shown inside a rounded light-gray box with a KodeKloud copyright.

Variable	Meaning
`$remote_addr`	Client IP address.
`$remote_user`	Authenticated user (empty if not used).
`$time_local`	Local timestamp.
`$request`	Full request line (method, URI, HTTP version).
`$status`	Response status code (200, 301, 404, etc.).
`$body_bytes_sent`	Bytes sent in the response body.
`$http_referer`	Referral URL (if present).
`$http_user_agent`	Browser or client user-agent string.
`$http_x_forwarded_for`	Proxy forwarded IPs (when behind a proxy).

Enable access logs inside a server block (default log files are usually under /var/log/nginx):

server {
    listen 80;
    server_name example.com www.example.com;

    root /var/www/example.com/html;
    index index.html;

    access_log /var/log/nginx/access.log;

    location / {
        try_files $uri $uri/ =404;
    }
}

Tail access logs to watch incoming requests in real time:

cd /var/log/nginx/
tail -f access.log

Example access log entry (trimmed):

192.168.1.1 - - [26/Jan/2025:14:23:35 +0000] "GET /images/logo.png HTTP/1.1" 200 512 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

Troubleshooting with access logs:

Ask a remote user to make a request while you tail -f the access log. If there’s no entry, the request didn’t reach NGINX (DNS, client, proxy, or firewall issue).
Ensure access_log is set in the appropriate http, server, or location block. Custom logs won’t be enabled unless explicitly configured.

Error logs

Error logs capture server-side and runtime issues. They should be quiet during normal operation; frequent errors indicate misconfiguration or application problems. Configure location and verbosity with error_log:

error_log /var/log/nginx/error.log warn;

Check error_log for stack traces, file-not-found errors, permission problems, or upstream failures.

NGINX metrics: stub_status

NGINX provides a lightweight status endpoint through the stub_status module (see https://nginx.org/en/docs/http/ngx_http_stub_status_module.html). It exposes runtime counters for connections and requests. Example server block to enable a local-only /nginx_status endpoint:

server {
    listen 81;
    server_name example.com;

    location /nginx_status {
        stub_status;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

Notes:

access_log off; prevents scrape requests from cluttering logs.
Restricting access (localhost or your monitoring hosts) and using a nonstandard port reduce exposure.

Query locally:

curl localhost:81/nginx_status

Sample output:

Active connections: 309
server accepts handled requests
16630948 16630948 31070465
Reading: 11 Writing: 218 Waiting: 38

Field meanings:

Active connections — current client connections.
accepts — total accepted connections.
handled — accepted and handled connections.
requests — total number of client requests.
Reading/Writing/Waiting — connections reading the request, writing the response, and keep-alive idle connections.

Monitoring platforms and approaches

There are many solutions for metrics and log aggregation. Common choices include Prometheus + Grafana, Datadog, Dynatrace, and New Relic.

A presentation slide titled "Monitoring Tools" displaying the logos and names of Prometheus, Grafana, Datadog, Dynatrace, and New Relic. The logos are arranged inside a rounded rectangle on the slide.

Tool	Type	Typical use
Prometheus + Grafana	Open-source metrics + dashboards	Scrape `stub_status` via an exporter, create dashboards and alerts.
Datadog	SaaS monitoring	Agent collects system and NGINX metrics, ingest logs, dashboards and alerts.
Dynatrace	SaaS APM	Full-stack tracing and infrastructure monitoring.
New Relic	SaaS APM	Application and infrastructure monitoring with log ingestion.

Common approach:

Install an agent on the host (Datadog Agent, Prometheus node_exporter).
Collect system metrics (CPU, memory, disk, network) and NGINX metrics (via stub_status or an NGINX exporter).
Centralize logs and metrics, then build dashboards and alerts (error rate, response codes, connection saturation).

Datadog example

Datadog can ingest both logs and metrics and scrape stub_status for basic NGINX metrics. A typical system dashboard includes CPU, memory, load, disk, network, and NGINX charts.

Screenshot of a DataDog system metrics dashboard. It shows multiple monitoring charts—CPU and memory (including a treemap), load averages, disk latency, network traffic and disk usage—for an nginx host.

Note: NGINX Plus (commercial) exposes additional metrics such as detailed status codes, upstream health, and cache statistics. Open-source NGINX combined with exporters and monitoring tools still provides strong observability for most environments.

Example log excerpts

Error and request lines can reveal missing files, bad routes, or failing backends:

2018/05/16 22:02:02 [error] 1244#1244: *203 open() "/var/www/html/order/2480" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /order/2480 HTTP/1.1"
2018/05/16 22:02:02 [error] 1244#1244: *202 open() "/var/www/html/send/notice" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /send/notice HTTP/1.1"
GET /order/2480 HTTP/1.1
GET /send/notice HTTP/1.1
GET /pass/beanserver/refund/2480 HTTP/1.1
GET /pass/beanserver/payment/ HTTP/1.1
GET /pass/beanserver/payment/7794 HTTP/1.1
GET /pass/beanserver/order/8048 HTTP/1.1
GET /pass/beanserver/order/4943 HTTP/1.1
2018/05/16 22:02:01 [error] 1244#1244: *186 open() "/var/www/html/bad" failed (2: No such file or directory), client: 127.0.0.1, server: _, request: "GET /bad HTTP/1.1"
GET /pass/welcome HTTP/1.1

Configuration testing and reloading

Always test configuration changes before applying them to avoid service disruption. Test configuration syntax:

nginx -t

Successful test example:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Sample failure output for misplaced directives or typos:

nginx: [emerg] unknown directive "worker_connections" in /etc/nginx/nginx.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed
2025/01/29 23:46:03 [emerg] 71439#71439: "server" directive is not allowed here in /etc/nginx/sites-enabled/default:39

Reload without dropping connections after a successful test:

nginx -s reload

Check service status with systemctl:

systemctl status nginx

Example active service output (trimmed):

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2025-01-28 00:18:10 UTC; 1 day 23h ago
     Docs: man:nginx(8)
  Main PID: 2164 (nginx)
    Tasks: 2 (limit: 1130)
   Memory: 24.7M
      CPU: 16.367s

If inactive, start and inspect recent logs:

systemctl start nginx
journalctl -u nginx --since "10 minutes ago"

Do not run two services bound to the same port (for example, Apache and NGINX both listening on port 80). Port conflicts will prevent the web server from starting.

Quick runtime checks

If systemctl reports NGINX as active but users still have problems, test the local HTTP response:

curl --head localhost

Example response:

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 30 Jan 2025 00:10:05 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
ETag: "67982241-264"
Accept-Ranges: bytes

A 200 status shows NGINX is responding locally; next check the application, firewall, DNS, or network path.

Firewalls and connectivity

Ensure cloud-provider security groups and host firewalls allow ports 80 and 443. UFW (Ubuntu) example:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload

firewalld (RHEL/CentOS) example:

sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --reload

Production monitoring best practices

Rotate logs to avoid disk exhaustion (logrotate or your platform’s logging agent).
Centralize metrics and logs (Prometheus, Datadog, ELK/Opensearch) and set alerts for high error rates, CPU spikes, or connection limits.
Limit exposure of sensitive endpoints (for example, restrict /nginx_status to localhost or specific monitoring hosts).
Consider NGINX Plus for advanced metrics and commercial support if you need upstream health, cache analytics, and built-in dashboarding.

Final recommendations

Always validate configuration changes with nginx -t before reloading, restrict sensitive endpoints (e.g., /nginx_status) to trusted hosts, and centralize logs and metrics to simplify troubleshooting and alerting.

This concludes the monitoring and troubleshooting guide for NGINX. For further reading, see:

NGINX documentation: https://nginx.org/en/docs/
Prometheus: https://prometheus.io/
Grafana: https://grafana.com/
Datadog: https://www.datadoghq.com/

Introduction

Install & Config

Intermediate Config

Security

Performance

Conclusion

Monitoring Troubleshooting

Logging

Error logs

NGINX metrics: stub_status

Monitoring platforms and approaches

Datadog example

Example log excerpts

Configuration testing and reloading

Quick runtime checks

Firewalls and connectivity

Production monitoring best practices

Final recommendations

Watch Video

​Logging

​Error logs

​NGINX metrics: stub_status

​Monitoring platforms and approaches

​Datadog example

​Example log excerpts

​Configuration testing and reloading

​Quick runtime checks

​Firewalls and connectivity

​Production monitoring best practices

​Final recommendations

Watch Video

Logging

Error logs

NGINX metrics: stub_status

Monitoring platforms and approaches

Datadog example

Example log excerpts

Configuration testing and reloading

Quick runtime checks

Firewalls and connectivity

Production monitoring best practices

Final recommendations