Guide

Splunk forwarder not sending data: a troubleshooting guide

Updated November 3, 2026 · Ayrix Labs

Short answer: Find quiet hosts with | tstats latest(_time) where index=* by host, then check whether the forwarder is connected (group=tcpin_connections in metrics.log on the indexer, splunk list forward-server on the forwarder), its TcpOutputProc errors, its inputs, whether the target index exists, and whether events are arriving with wrong timestamps.

1. Find hosts that went quiet

| tstats latest(_time) as last_seen where index=* by host
| eval hours_quiet=round((now() - last_seen)/3600, 1)
| where hours_quiet > 4
| sort - hours_quiet

2. Is the forwarder connected?

On the indexers' side:

index=_internal source=*metrics.log* group=tcpin_connections earliest=-24h
| stats latest(_time) as last_connected by hostname, sourceIp, fwdType, version
| eval last_connected=strftime(last_connected, "%F %T")

On the forwarder:

$SPLUNK_HOME/bin/splunk list forward-server
$SPLUNK_HOME/bin/splunk btool outputs list --debug

An inactive forward-server means the forwarder cannot reach the receiver: check the host and port in outputs.conf, that receiving is enabled on the indexer (Settings, Forwarding and receiving; port 9997 by default), firewalls, and TLS settings on both sides.

3. Read the forwarder's own errors

index=_internal host=<forwarder> sourcetype=splunkd (component=TcpOutputProc OR component=TcpOutputFd) log_level!=INFO earliest=-24h
| stats count latest(_raw) as example by component, log_level

If even the forwarder's _internal data is missing, the problem is the connection; read $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder.

4. Are the inputs reading?

$SPLUNK_HOME/bin/splunk list monitor
$SPLUNK_HOME/bin/splunk list inputstatus

Look for files the forwarder cannot read (permissions), paths that no longer match, and files it considers already read (a crcSalt or initCrcLength issue when many files start with the same header).

5. Does the index exist?

index=_internal sourcetype=splunkd "unconfigured/disabled/deleted index" earliest=-24h
| rex "index=(?<missing_index>\S+)"
| stats count by missing_index, host

Events for an index that does not exist on the indexers are dropped, or go to the lastChanceIndex if one is set in indexes.conf.

6. Is the data there with the wrong time?

index=* host=<forwarder> earliest=0 latest=+10y _index_earliest=-1h
| eval lag_s=_indextime - _time
| stats count min(lag_s) max(lag_s) by sourcetype

Data indexed recently but stamped in the past or future is not missing; its timestamps are wrong. Fix TIME_FORMAT, TIME_PREFIX or TZ in props.conf (see props.conf).

How Drew helps

Ask Drew "which hosts stopped sending in the last day?" Drew runs the searches above against your Splunk and answers with a table of quiet hosts and when each was last seen.

• Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
• Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.

Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.