Skip to content
Muhammet Şafak
tr

Postgres never turned the partial index into a generic plan: forty executions, forty custom plans

Does Postgres switch a partial-index query to a generic plan on its own inside a prepared statement — or is the 1,635-fold cliff something you have to opt into?

Finding

Postgres declines. On the partial index all forty executions used a custom plan — the counter reads 40/0. The reason it declines is the disaster itself: a generic plan cannot use the partial index, so its estimated cost comes out high and the planner does not choose it. The composite index switches at the sixth execution exactly as documented (5/35) and loses nothing by it. So the 1,635-fold cliff is real but fenced: reaching it takes writing `plan_cache_mode = force_generic_plan`.

Partial index, generic plans
0 of 40 executions
Composite index, switch
execution 6
Partial, first 5 → last 5
0.36 → 0.20 ms
What reaching the cliff takes
force_generic_plan

Method

The same harness and the same 10-million-dead-plus-5,000-live table, across three index strategies. For each, one prepared statement was executed forty times in a single psql session and `pg_prepared_statements` was queried after every execution — the counters are session-local, which is why this measurement cannot be made from pgbench. The decision is not inferred from timings but read from the counters: `custom_plans` and `generic_plans` say directly what the planner chose on that execution. Timings arrive in pairs (the statement, then the counter query) and the parser keeps only the first. psql's aligned output pads the counter value with spaces, so output was switched to unaligned mode — the first run produced an empty series because of exactly that.

High confidence Repeated runs, controlled environment, raw data published.
Measured on

measured 25 days ago

Published

Environment

Postgres
17-alpine · plan_cache_mode at its default (auto)
Table
10M dead + 5,000 live rows
Measurement
one session · one prepared statement · 40 executions
Source of the decision
pg_prepared_statements.custom_plans / generic_plans
Hardware
Apple M4 Pro · 12 cores · 24 GB · Docker Desktop 29.7.2

Technologies

PostgreSQL SQL Docker

To reproduce

EXECUTIONS=40 ./bench/plan-switch.sh

The partial index measurement pinned plan_cache_mode to each extreme and found a factor of 1,635 between them: 11,752 tps on a custom plan, 7 on a generic one. What it did not measure is the setting everyone actually runs — auto.

That record wrote a sentence there: “the profile degrades as it warms up… it does not show up in staging.” It was a reasonable inference and it turned out to be wrong.

The decision was read, not guessed

Postgres runs a prepared statement on a custom plan for its first five executions, then compares the generic plan’s estimated cost against the average of the custom ones, and switches only if the generic looks cheaper. pg_prepared_statements keeps that decision as a counter, so nothing has to be inferred from timings.

Strategy First generic plan Final counter (custom/generic) First 5 (ms) Last 5 (ms)
partial the switch was declined never 40 / 0 0.36 0.20
composite execution 6 5 / 35 0.40 0.21
(status) never 40 / 0 0.98 0.78
One session, one prepared statement, forty executions. The counters were read after every one.

The composite index switches where the documentation says it will, right after the fifth:

#5  0.266 ms   custom=5  generic=0
#6  0.275 ms   custom=5  generic=1   ← switched
#7  0.185 ms   custom=5  generic=2

On the partial index it never happens. Forty executions, forty custom plans.

It declines because of the disaster itself

The planner does not choose the generic plan because it estimates its cost correctly. A generic plan does not know what $1 is; to use the partial index it would have to prove $1 = 'pending', it cannot, so that plan falls back to a sequential scan and its estimate goes through the roof. The average custom plan sits far below it, and the comparison comes out in favour of custom every time.

The real trade is smaller, and elsewhere

The protection has a price: a query on the partial index is re-planned on every execution. Forty executions, forty plans. The composite one reuses its plan from the sixth onwards.

At this scale the difference is unmeasurable — 0.20 ms against 0.21 ms. But planning is cheap on a cheap query, not on a many-table join or a long IN list. The partial index’s hidden costs are bloat (see the endurance measurement) and constant re-planning; both are small in this workload, and neither is guaranteed to stay small in another.

It corrects the earlier record

The “degrades as it warms up” paragraph in the partial index record went too far, and a note pointing at this finding has been added there. A measurement correcting its own publication is what this section’s contract asks for: if a number is published, so is the number that refutes it.

Share:

Other records

All records

The partial index grew three hundred and five times in fifteen minutes — and autovacuum never ran

Under sustained churn, does a partial index stay small on a queue table, and do the default autovacuum settings keep up with it?

Finding

With the live set holding steady at five thousand rows the partial index went from 0.125 MB to 38.2 MB — three hundred and five times. Its smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. The composite index bloated less in proportion (42%) and more in absolute terms (+126 MB), and while bloating it stopped fitting in memory: its latency went from 0.52 ms to 61 seconds and its backlog climbed to 126,000. Fifteen minutes produced 1.75 million dead rows and autovacuum **did not run once** — the default threshold scales with the whole table (50 + 0.2 × 10 million ≈ 2 million) while the churn happens in a tiny subset.

measured 25 days ago

High confidence

A partial index makes a queue table forty-one times smaller — for as long as the planner picks it

On a Postgres queue table with millions of dead rows, what does a partial index buy, and when does the planner refuse to use it?

Finding

At 10 million dead rows a partial index sustains 11,537 claims per second where the same table without one manages 7. Against a composite index the throughput difference is small (6.9%) but the size difference is not: 7.6 MB against 310.4 MB, and the partial one does not grow with the table because it indexes only the 5,000 live rows. None of that is the real finding. The moment the planner switches a prepared statement to a generic plan the partial index stops being used at all — 11,752 tps becomes 7, and 0.68 ms becomes 1.1 seconds. A factor of 1,635. The composite index is untouched under the same conditions.

measured 25 days ago

High confidence
Service & load Measurement

I measured PHP and Go on the same OAuth2 API: no gap at 10,000 writes, a real one at 50,000 reads

The same API verifies an OAuth2 token on every request and then writes to or reads from PostgreSQL. On four cores, how much CPU do PHP-FPM, FrankenPHP worker mode and Go need for 10,000 writes and 50,000 reads a second?

Finding

At 10,000 writes a second all three candidates hit the target in five runs out of five, and none had a p99 above 2.5 ms: at this load the language is not a capacity line item. At 50,000 reads a second only Go held the target on four cores (p99 6.45 ms); PHP-FPM stopped at 23,528 and FrankenPHP at 22,859. CPU per read request is 64 microseconds for Go, 117 for FrankenPHP and 168 for PHP-FPM. Sized by instance, 50,000 reads take 3.4 cores in Go and 8.2–8.8 cores for the two PHP candidates. FrankenPHP's CPU saving does not turn into capacity: at saturation it leaves about one of its four cores idle.

measured today

Medium confidence

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind