Guide

Splunk health check: a practical checklist with searches

Updated November 3, 2026 · Ayrix Labs

Short answer: A Splunk health check covers skipped scheduled searches, blocked indexing queues, hosts that stopped sending, timestamp and line-breaking errors, license usage, index retention, certificate expiry and KV store status. Start with the Monitoring Console's Health Check, then run the searches below for detail.

1. Run the built-in checks

In Splunk Enterprise open Settings, Monitoring Console, Health Check and run it. The health icon in Splunk Web's bar shows the splunkd health report; the same data is at | rest /services/server/health/splunkd/details.

2. Skipped scheduled searches

index=_internal sourcetype=scheduler earliest=-24h
| stats count(eval(status="skipped")) as skipped count as total by host
| eval skip_pct=round(skipped/total*100, 1)
| sort - skip_pct

See skipped searches for reasons and fixes.

3. Blocked queues

index=_internal source=*metrics.log* group=queue blocked=true earliest=-24h
| stats count by host, name
| sort - count

See blocked queues for how to read the pipeline.

4. Hosts that stopped sending

| tstats latest(_time) as last_seen where index=* by host
| eval hours_quiet=round((now() - last_seen)/3600, 1)
| where hours_quiet > 24
| sort - hours_quiet

See forwarder not sending data.

5. Timestamp and line-breaking errors

index=_internal sourcetype=splunkd earliest=-24h
    (component=DateParserVerbose OR component=LineBreakingProcessor OR component=AggregatorMiningProcessor)
| stats count by component, data_sourcetype
| sort - count

See data onboarding with props.conf.

6. License usage

Check each day's total against the quota and which sourcetypes grew. The searches are in license usage.

7. Index size and retention

| rest /services/data/indexes
| eval used_pct=round(currentDBSizeMB/maxTotalDataSizeMB*100, 1), retention_days=round(frozenTimePeriodInSecs/86400)
| table splunk_server title currentDBSizeMB maxTotalDataSizeMB used_pct retention_days
| sort - used_pct

An index near maxTotalDataSizeMB freezes its oldest buckets early, so you keep less history than frozenTimePeriodInSecs promises.

8. Errors by component

index=_internal sourcetype=splunkd log_level=ERROR earliest=-24h
| stats count by component
| sort - count

9. Certificates

openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/cacert.pem

Check every certificate your server.conf, inputs.conf, outputs.conf and web.conf name, including the CA chain. A renewed server certificate with an expired CA still fails.

10. KV store

$SPLUNK_HOME/bin/splunk show kvstore-status

See KV store not starting.

How Drew helps

Ask Drew "run a health check on our Splunk" or any single item above. Drew runs the searches through your Splunk MCP Server and says which figures it checked in that turn.

• Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
• Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.

Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.