Guide

Splunk KV store not starting: causes and checks

Updated November 3, 2026 · Ayrix Labs

Short answer: Run splunk show kvstore-status and read mongod.log. Common causes are an expired server certificate or CA in the chain, port 8191 in use or blocked, a full disk, wrong file ownership after running Splunk as root, and a stale lock file after an unclean shutdown.

Check status and the log

$SPLUNK_HOME/bin/splunk show kvstore-status
tail -n 200 $SPLUNK_HOME/var/log/splunk/mongod.log

index=_internal (source=*mongod.log* OR (sourcetype=splunkd component=KVStore*)) earliest=-24h
| stats count latest(_raw) as example by host, source
| sort - count

Expired certificates

The KV store uses the instance's server certificate and CA. Renewing server.pem is not enough when a CA certificate in the chain has expired. Check each certificate in the chain:

openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/cacert.pem
openssl verify -CAfile $SPLUNK_HOME/etc/auth/cacert.pem $SPLUNK_HOME/etc/auth/server.pem

Also check any custom certificates named in server.conf under [sslConfig] and [kvstore]. When the KV store is down, lookups backed by it fail, so apps such as Enterprise Security can look broken while the cause is the certificate.

Other common causes

• Port: the KV store listens on 8191 by default ([kvstore] port in server.conf). Check nothing else holds it and that cluster members can reach each other on it.
• Disk: a full $SPLUNK_DB volume stops mongod.
• Permissions: starting Splunk once as root can leave files in $SPLUNK_DB/kvstore the Splunk user cannot open.
• Lock file: after an unclean shutdown a stale mongod.lock can block start-up.
• Search head cluster: check the KV store member status on each member and the captain before resyncing.

Resyncing or cleaning the KV store can lose data on that member; follow Splunk's documentation for your version and take a backup first.

How Drew helps

Ask Drew "why are KV store lookups failing on our search head?" Drew searches the KV store and mongod logs in _internal and reports the errors it finds, with when they started.

• Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
• Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.

Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.