Guide
Splunk skipped searches: how to find the cause and fix it
Updated · Ayrix Labs
Short answer: Search index=_internal sourcetype=scheduler status=skipped | stats count by savedsearch_name, app, reason. Most skips come from the concurrency limit for scheduled searches being reached, or from a search still running when its next run is due; fix them by spreading schedules, using schedule windows and making slow searches faster.
Which searches are skipped, and why
index=_internal sourcetype=scheduler status=skipped earliest=-24h
| stats count by savedsearch_name, app, user, reason
| sort - countSkip ratio per search head
index=_internal sourcetype=scheduler earliest=-24h
| stats count(eval(status="skipped")) as skipped count as total by host
| eval skip_pct=round(skipped/total*100, 1)Common reasons
| Reason contains | What it means | Fix |
|---|---|---|
| maximum number of concurrent historical scheduled searches on this instance (or cluster) has been reached | Too many scheduled searches want to run at the same moment | Spread cron schedules (not everything at minute 0), add schedule_window, lower the frequency of low-value searches, or add search capacity |
| maximum number of concurrent running jobs for this historical scheduled search | The previous run of the same search is still running | Make the search faster or run it less often |
| The maximum disk usage quota for this user has been reached | The owner's search artifacts filled their disk quota | Shorten dispatch.ttl, clean old jobs, or raise the role's quota |
Find the collision minutes
index=_internal sourcetype=scheduler earliest=-24h
| eval minute=strftime(scheduled_time, "%M")
| stats count(eval(status="skipped")) as skipped count as runs by minute
| sort - skippedIf most skips land on minutes like 00, 15, 30 and 45, move some searches to other minutes or give them a schedule_window so the scheduler can shift them.
Find the slow searches
index=_internal sourcetype=scheduler status=success earliest=-24h
| stats avg(run_time) as avg_s max(run_time) as max_s count by savedsearch_name, app
| sort - avg_sA search whose run time is close to its interval will skip itself. Narrow its time range, use tstats or an accelerated data model, or summarize.
Settings that control it
In savedsearches.conf: schedule_window, schedule_priority and realtime_schedule. In limits.conf under [scheduler]: max_searches_perc (the share of search slots the scheduler may use). Raising limits only helps if the hardware has headroom; check CPU first.
How Drew helps
Ask Drew "which searches were skipped yesterday and why?" Drew runs the scheduler searches above on your deployment and names the searches, owners, apps and reasons.
- Answers from your own Splunk. Ask in plain English in Slack or the web app. Drew runs the SPL searches through your Splunk MCP Server and answers with text, tables or charts.
- Says whether each answer was checked. Every answer that states figures is marked Verified (a search this turn returned the data), Likely (figures from an earlier search, with when) or Not checked. The label is decided by code from what actually ran, not by the model.
Drew is taking a small group of MSPs, Splunk teams and consultants into early access. Request access or email hello@rundrew.com.
Questions
Why are my Splunk scheduled searches being skipped?
Usually because the concurrency limit for scheduled searches was reached, or because the previous run of the same search was still running. The reason field in index=_internal sourcetype=scheduler status=skipped says which.
What is a normal skip ratio in Splunk?
Ideally close to zero. Any steady skipping means alerts and reports are silently missing runs, so investigate the top searches by skip count.