hypopg is a Postgres Pro extension, adding support for hypothetical indexes.
A hypothetical, or virtual, index is an index that does not really exist, and therefore does not cost CPU, disk, or any resource to create. They are useful to find out whether specific indexes can increase the performance for problematic queries, since you can discover if Postgres Pro will use these indexes or not without having to spend resources to create them.
The hypopg extension is provided with
Postgres Pro as a separate pre-built package
hypopg-ent-16 (for the detailed installation
instructions, see Chapter 16). Once you have
Postgres Pro installed, create the
hypopg extension:
CREATE EXTENSION hypopg;
hypopg is now available. You can check easily if the extension is present using psql:
\dx
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+-------------------------------------
hypopg | 1.4.1 | public | Hypothetical indexes for Postgres Pro
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(2 rows)
As you can see, hypopg is installed.
hypopg is useful if you want to check if some index would help one or multiple queries. Therefore, you should already know what are the queries you need to optimize, and ideas on which indexes you want to try.
Also, the hypothetical indexes that hypopg will create are not stored in any catalog, but in your connection private memory. Therefore, it won't bloat any table and won't impact any concurrent connection.
Moreover, since the hypothetical indexes don't really exist,
hypopg makes sure they will only be using a
simple EXPLAIN statement (without the
ANALYZE option).
The following access methods are supported:
btree
brin
hash
bloom (requires the
bloom extension to be installed)
Using hypopg requires some knowledge on the
EXPLAIN command.
hypopg_create_index()
#For clarity, let's see how it works with a very simple test case:
CREATE TABLE hypo (id integer, val text) ; INSERT INTO hypo SELECT i, 'line ' || i FROM generate_series(1, 100000) i ; VACUUM ANALYZE hypo ;
This table doesn't have any index. Let's assume we want to check if an index would help a simple query. First, let's see how it behaves:
EXPLAIN SELECT val FROM hypo WHERE id = 1;
QUERY PLAN
--------------------------------------------------------
Seq Scan on hypo (cost=0.00..1791.00 rows=1 width=14)
Filter: (id = 1)
(2 rows)
A plain sequential scan is used, since no index exists on the
table. A simple btree index on the id
column should help this query. Let's check with
hypopg. The function
hypopg_create_index() will accept any standard
CREATE INDEX
statement(s) (any other statement passed to this function will be
ignored), and create a hypothetical index for each:
SELECT * FROM hypopg_create_index('CREATE INDEX ON hypo (id)') ;
indexrelid | indexname
------------+----------------------
18284 | <18284>btree_hypo_id
(1 row)
The function returns two columns:
the object identifier of the hypothetical index
the generated hypothetical index name
We can run the EXPLAIN again to see if
Postgres Pro would use this index:
EXPLAIN SELECT val FROM hypo WHERE id = 1;
QUERY PLAN
----------------------------------------------------------------------------------
Index Scan using <18284>btree_hypo_id on hypo (cost=0.04..8.06 rows=1 width=10)
Index Cond: (id = 1)
(2 rows)
Yes, Postgres Pro would use such an index. Just to be sure, let's check that the hypothetical index won't be used to actually run the query:
EXPLAIN ANALYZE SELECT val FROM hypo WHERE id = 1;
QUERY PLAN
---------------------------------------------------------------------------------------------------
Seq Scan on hypo (cost=0.00..1791.00 rows=1 width=10) (actual time=0.046..46.390 rows=1 loops=1)
Filter: (id = 1)
Rows Removed by Filter: 99999
Planning time: 0.160 ms
Execution time: 46.460 ms
(5 rows)
That's all you need to create hypothetical indexes and see if Postgres Pro would use such indexes.
The hypopg_list_indexes view
lists all hypothetical indexes that have been created.
SELECT * FROM hypopg_list_indexes ;
indexrelid | index_name | schema_name | table_name | am_name
------------+-----------------------+-------------+------------+---------
18284 | <18284>btree_hypo_id | public | hypo | btree
(1 row)
hypopg()
#
hypopg() lists all hypothetical indexes that have
been created with the same format as pg_index.
SELECT * FROM hypopg() ;
indexname | indexrelid | indrelid | innatts | indisunique | indkey | indcollation | indclass | indoption | indexprs | indpred | amid
----------------------+------------+----------+---------+-------------+--------+--------------+----------+-----------+----------+---------+------
<18284>btree_hypo_id | 13543 | 18122 | 1 | f | 1 | 0 | 1978 | <NULL> | <NULL> | <NULL> | 403
(1 row)
hypopg_get_indexdef(oid)
#
hypopg_get_indexdef(oid) lists the
CREATE INDEX statements that would recreate a stored
hypothetical index.
SELECT index_name, hypopg_get_indexdef(indexrelid) FROM hypopg_list_indexes ;
index_name | hypopg_get_indexdef
-----------------------+----------------------------------------------
<18284>btree_hypo_id | CREATE INDEX ON public.hypo USING btree (id)
(1 row)
hypopg_relation_size(oid)
#
hypopg_relation_size(oid) estimates how big a
hypothetical index would be:
SELECT index_name, pg_size_pretty(hypopg_relation_size(indexrelid))
FROM hypopg_list_indexes ;
index_name | pg_size_pretty
-----------------------+----------------
<18284>btree_hypo_id | 2544 kB
(1 row)
hypopg_drop_index(oid)
#
hypopg_drop_index(oid) removes the given
hypothetical index.
hypopg_reset()
#
hypopg_reset() removes all hypothetical indexes.
hypopg_hide_index(oid)
#
You can hide both existing and hypothetical indexes
hypothetically. If you want to test it as described in the
documentation, you should first use
hypopg_reset() to clear the
effects of any other hypothetical indexes.
As a simple case, let's consider two indexes:
SELECT hypopg_reset(); CREATE INDEX ON hypo(id); CREATE INDEX ON hypo(id, val);
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
----------------------------------------------------------------------------------
Index Only Scan using hypo_id_val_idx on hypo (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)
The query plan is using the hypo_id_val_idx index
now.
hypopg_hide_index(oid) allows you to hide an index
in the EXPLAIN output by using its OID. It returns
true if the index was successfully hidden, and false otherwise.
SELECT hypopg_hide_index('hypo_id_val_idx'::REGCLASS);
hypopg_hide_index
-------------------
t
(1 row)
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
-------------------------------------------------------------------------
Index Scan using hypo_id_idx on hypo (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)
As an example, let's assume that the query plan is currently using
the hypo_id_val_idx index. To
continue testing, use the
hypopg_hide_index(oid) function
to hide another index.
SELECT hypopg_hide_index('hypo_id_idx'::REGCLASS);
hypopg_hide_index
-------------------
t
(1 row)
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
-------------------------------------------------------
Seq Scan on hypo (cost=0.00..180.00 rows=1 width=13)
Filter: (id = 1)
(2 rows)
hypopg_unhide_index(oid)
#
hypopg_unhide_index(oid) restores a previously
hidden index in the EXPLAIN output by using its OID.
It returns true if the index was successfully restored, and false
otherwise.
SELECT hypopg_unhide_index('hypo_id_idx'::regclass);
hypopg_unhide_index
-------------------
t
(1 row)
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
-------------------------------------------------------------------------
Index Scan using hypo_id_idx on hypo (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)
hypopg_unhide_all_index(oid)
#
hypopg_unhide_all_index() restores all hidden
indexes and returns void.
hypopg_hidden_indexes()
#
hypopg_hidden_indexes() returns a list of OIDs
for all hidden indexes.
SELECT * FROM hypopg_hidden_indexes(); indexid --------- 526604 (1 rows)
The hypopg_hidden_indexes view returns a
formatted list of all hidden indexes.
SELECT * FROM hypopg_hidden_indexes;
indexrelid | index_name | schema_name | table_name | am_name | is_hypo
-------------+----------------------+-------------+------------+---------+---------
526604 | hypo_id_val_idx | public | hypo | btree | f
(1 rows)
Hypothetical indexes can be hidden as well.
SELECT hypopg_create_index('CREATE INDEX ON hypo(id)');
hypopg_create_index
------------------------------
(12659,<12659>btree_hypo_id)
(1 row)
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
------------------------------------------------------------------------------------
Index Scan using "<12659>btree_hypo_id" on hypo (cost=0.04..8.05 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)
Now that the hypothetical index is being used, we can try hiding it to see the change:
SELECT hypopg_hide_index(12659);
hypopg_hide_index
-------------------
t
(1 row)
EXPLAIN SELECT * FROM hypo WHERE id = 1;
QUERY PLAN
-------------------------------------------------------------------------
Index Scan using hypo_id_idx on hypo (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)
SELECT * FROM hypopg_hidden_indexes;
indexrelid | index_name | schema_name | table_name | am_name | is_hypo
-------------+----------------------+-------------+------------+---------+---------
12659 | <12659>btree_hypo_id | public | hypo | btree | t
526604 | hypo_id_val_idx | public | hypo | btree | f
(2 rows)
If a hypothetical index has been hidden, it will be
automatically unhidden when it is deleted using
hypopg_drop_index(oid) or
hypopg_reset().
SELECT hypopg_drop_index(12659);
SELECT * FROM hypopg_hidden_indexes;
indexrelid | index_name | schema_name | table_name | am_name | is_hypo
-------------+----------------------+-------------+------------+---------+---------
526604 | hypo_id_val_idx | public | hypo | btree | f
(2 rows)
The following configuration parameters (GUCs) are available, and can be changed interactively:
hypopg.enabled
#
Defaults to on. Use this parameter to globally enable
or disable hypopg. When
hypopg is disabled, no hypothetical index
will be used, but the defined hypothetical indexes won't be removed.
hypopg.use_real_oids
#
Defaults to off. By default,
hypopg won't use "real"
object identifiers, but instead borrow ones from the ~ 14000 / 16384
(respectively the lowest unused OID less then
FirstNormalObjectId and
FirstNormalObjectId) range, which are reserved by
Postgres Pro for future usage in future
releases. This doesn't cause any problem, as the free range is
dynamically computed the first time a connection uses
hypopg, and has the advantage to work on
a standby server. But the drawback is that you can't have more than
approximately 2500 hypothetical indexes at the same time, and creating
a new hypothetical index will become very slow once more than the
maximum number of objects has been created until
hypopg_reset() is called.
If those drawbacks are problematic, you can enable this parameter. hypopg will then ask for a real object identifier, which will need to obtain more locks and won't work on a standby, but will allow using the full range of object identifiers.
Note that switching this parameter doesn't require resetting the entries, both can coexist at the same time.