Guide

Splunk blocked queues: how to find the bottleneck

Updated · Ayrix Labs

Short answer: Search index=_internal source=*metrics.log* group=queue for blocked=true and fill percentage by queue name. A block backs up every queue before it, so the last blocked queue in the order parsing, aggregation, typing, indexing (then tcpout) is the bottleneck.

Queue fill over time

index=_internal source=*metrics.log* group=queue host=<indexer> earliest=-4h
| eval fill_pct=round(current_size_kb/max_size_kb*100)
| timechart span=5m perc90(fill_pct) by name

Which queue blocks

index=_internal source=*metrics.log* group=queue blocked=true earliest=-24h
| stats count by host, name
| sort - count

Read it from the end

Last full queueUsual cause
indexqueueStorage cannot keep up: slow or full disk, or too much data per indexer
typingqueueHeavy regex at index time: TRANSFORMS and SEDCMD on high-volume sourcetypes
aggqueueTimestamp extraction and line merging: missing TIME_FORMAT, SHOULD_LINEMERGE = true
parsingqueueLine breaking and character set work, or everything downstream is blocked
tcpout (on forwarders)A destination is down or slow and the output blocks

Blocked outputs on forwarders

When outputs.conf clones data to two output groups and one goes down, the shared queue fills and data to the healthy group stops too, unless the failing group may drop events. dropEventsOnQueueFull in that group's outputs.conf stanza sets how long to wait before dropping; it trades blocking for data loss to that destination, so set it deliberately.

index=_internal sourcetype=splunkd component=TcpOutputProc (log_level=WARN OR log_level=ERROR) earliest=-4h
| stats count latest(_raw) as example by host

How Drew helps

Ask Drew "are any indexing queues blocking?" Drew runs the metrics.log searches above and tells you which hosts and queues block, and when.

  • Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
  • Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.

Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.

Questions

How do I know which Splunk queue is the bottleneck?

A blocked queue fills every queue before it, so find the last queue in the pipeline (parsing, aggregation, typing, indexing) that shows blocked=true in metrics.log; that one is the bottleneck.