Guide

Splunk skipped searches: how to find the cause and fix it

Updated · Ayrix Labs

Short answer: Search index=_internal sourcetype=scheduler status=skipped | stats count by savedsearch_name, app, reason. Most skips come from the concurrency limit for scheduled searches being reached, or from a search still running when its next run is due; fix them by spreading schedules, using schedule windows and making slow searches faster.

Which searches are skipped, and why

index=_internal sourcetype=scheduler status=skipped earliest=-24h
| stats count by savedsearch_name, app, user, reason
| sort - count

Skip ratio per search head

index=_internal sourcetype=scheduler earliest=-24h
| stats count(eval(status="skipped")) as skipped count as total by host
| eval skip_pct=round(skipped/total*100, 1)

Common reasons

Reason containsWhat it meansFix
maximum number of concurrent historical scheduled searches on this instance (or cluster) has been reachedToo many scheduled searches want to run at the same momentSpread cron schedules (not everything at minute 0), add schedule_window, lower the frequency of low-value searches, or add search capacity
maximum number of concurrent running jobs for this historical scheduled searchThe previous run of the same search is still runningMake the search faster or run it less often
The maximum disk usage quota for this user has been reachedThe owner's search artifacts filled their disk quotaShorten dispatch.ttl, clean old jobs, or raise the role's quota

Find the collision minutes

index=_internal sourcetype=scheduler earliest=-24h
| eval minute=strftime(scheduled_time, "%M")
| stats count(eval(status="skipped")) as skipped count as runs by minute
| sort - skipped

If most skips land on minutes like 00, 15, 30 and 45, move some searches to other minutes or give them a schedule_window so the scheduler can shift them.

Find the slow searches

index=_internal sourcetype=scheduler status=success earliest=-24h
| stats avg(run_time) as avg_s max(run_time) as max_s count by savedsearch_name, app
| sort - avg_s

A search whose run time is close to its interval will skip itself. Narrow its time range, use tstats or an accelerated data model, or summarize.

Settings that control it

In savedsearches.conf: schedule_window, schedule_priority and realtime_schedule. In limits.conf under [scheduler]: max_searches_perc (the share of search slots the scheduler may use). Raising limits only helps if the hardware has headroom; check CPU first.

How Drew helps

Ask Drew "which searches were skipped yesterday and why?" Drew runs the scheduler searches above on your deployment and names the searches, owners, apps and reasons.

  • Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
  • Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.

Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.

Questions

Why are my Splunk scheduled searches being skipped?

Usually because the concurrency limit for scheduled searches was reached, or because the previous run of the same search was still running. The reason field in index=_internal sourcetype=scheduler status=skipped says which.

What is a normal skip ratio in Splunk?

Ideally close to zero. Any steady skipping means alerts and reports are silently missing runs, so investigate the top searches by skip count.