SQL order of execution
Introduction
- In
SQLin general andPostgreSQLin particular, the order of writing keywords in code (Lexical Order) is completely different from the order in which theDatabase Engineactually executes the statement (Logical Execution Order). - A query with full command structures like this will be parsed and executed by the
Optimizerin a strict logical order from root (data source) to top (displayed results).
[ WITH [RECURSIVE] cte_name AS (...) ]
SELECT [ DISTINCT [ON (...)] ] columns
FROM source_table [ TABLESAMPLE ... ]
[ JOIN_clauses ]
[ WHERE raw_filter_conditions ]
[ GROUP BY { column | ROLLUP | CUBE | GROUPING SETS } ]
[ HAVING group_filter_conditions ]
[ WINDOW window_name AS (...) ]
[ { UNION | INTERSECT | EXCEPT } [ALL | DISTINCT] other_select_statement ]
[ ORDER BY sort_order ]
[ LIMIT limit_count ] [ OFFSET offset_count ]
[ FOR UPDATE / FOR SHARE [ SKIP LOCKED ] ];
Below is the execution order from the first step to the last step:
WITH / WITH RECURSIVE- This is the way to use
CTE (Common Table Expressions)to execute statements inside theWITHblock first to prepare temporary data tables - If it is
RECURSIVE, it will run a loop to generate data until the stop condition is met.
- This is the way to use
FROMandTABLESAMPLE- Identify tables to retrieve data
- If
TABLESAMPLEis present,Postgreswill only extract a random sample amount (for example10%of table rows) for further processing, instead of the whole table.
JOINclauses- Used to match data to create an expanded
Cartesianproduct table, combining secondary tables into the source table based on operators (INNER, LEFT, RIGHT, FULL JOIN) and conditions after the ON keyword - At the end of this step, we have a temporary "super table" containing all columns of participating tables.
- Used to match data to create an expanded
WHERE- Filters raw data by iterating through each row to keep rows that satisfy the
WHEREcondition, while non-satisfying rows are eliminated - Note that aggregate calculation functions (such as
SUM, COUNT) cannot be used at this point because data has not been grouped yet.
- Filters raw data by iterating through each row to keep rows that satisfy the
GROUP BY- The remaining rows after filtering will be grouped based on specified columns.
- Super functions like
ROLLUPorCUBEwill automatically generate subtotal and grand total rows at this step.
HAVING- Filters data after grouping based on aggregate functions
- After groups are formed, the
HAVINGcondition will be applied to eliminate groups that do not satisfy the condition.
WINDOW- Uses a combination of
Aggregate Function(such asROW_NUMBER(), RANK(), SUM(),...) andWindow Function Clause (OVER)to compute aggregates based on 1 column - Unlike
GROUP BY, this step computes based on partitions while keeping the original number of rows without collapsing rows.
- Uses a combination of
SELECT and DISTINCT [ON]: Column extraction and deduplication- At this point, the
Databasecalculates expressions and setsAliasbased on the specified column list - If
DISTINCTis present, it eliminates duplicate rows - With
DISTINCT ON (...), it keeps the first unique row for each set of duplicate values.
- At this point, the
- Set operations
UNION / INTERSECT / EXCEPT: Union, intersection and difference of statements.- If you have multiple
SELECTstatements connected by set operators, the Database takes the complete result of this statement to merge, intersect or subtract from the result of anotherSELECTstatement - If
ALLis used, it keeps everything, ifDISTINCTis used (default), it scans to filter duplicates again.
- If you have multiple
ORDER BY: Display sorting- Final data is sorted according to specified columns or expressions.
- Because this step runs after
SELECT, Aliases defined in theSELECTstep can be used for sorting.
LIMITandOFFSET- Slices results, commonly used for data pagination.
- Skips a number of initial rows (
OFFSET) and keeps only the maximum requested number of rows (LIMIT) - This is performed at the very end to ensure returning only the exact amount of data needed by the client.
FOR UPDATE / FOR SHARE: Concurrency Control- After determining the exact final result rows to return to the user, Postgres locks those rows in memory for the
Transaction - If
SKIP LOCKEDis used, it automatically skips rows currently locked by another session to avoid blocking. - In addition, there are other
Row level lockswith different lock levels
- After determining the exact final result rows to return to the user, Postgres locks those rows in memory for the
Detail
First, create Tables and Indexes as follows:
CREATE TABLE categories (
category_id INT PRIMARY KEY,
category_name VARCHAR(100) NOT NULL,
parent_id INT REFERENCES categories(category_id)
);
CREATE TABLE products (
product_id INT PRIMARY KEY,
product_name VARCHAR(200) NOT NULL,
category_id INT REFERENCES categories(category_id),
price NUMERIC(12, 2) NOT NULL,
stock_quantity INT NOT NULL DEFAULT 0
);
CREATE TABLE orders (
order_id INT PRIMARY KEY,
customer_id INT NOT NULL,
order_date TIMESTAMPTZ NOT NULL DEFAULT NOW(),
status VARCHAR(30) NOT NULL
);
CREATE TABLE order_items (
item_id INT PRIMARY KEY,
order_id INT REFERENCES orders(order_id),
product_id INT REFERENCES products(product_id),
quantity INT NOT NULL,
unit_price NUMERIC(12, 2) NOT NULL
);
Next is a comprehensive example describing the execution order:
WITH RECURSIVE category_tree AS (
SELECT category_id, category_name, parent_id
FROM categories
WHERE category_id = 1
UNION ALL
SELECT c.category_id, c.category_name, c.parent_id
FROM categories c
JOIN category_tree ct ON c.parent_id = ct.category_id
),
target_orders AS (
SELECT order_id, customer_id, order_date, status
FROM orders
WHERE status = 'PENDING'
)
SELECT
p.product_id,
p.product_name,
DATE(o.order_date) AS sales_date,
SUM(oi.quantity) AS total_quantity_sold,
SUM(oi.quantity * oi.unit_price) AS total_revenue,
SUM(SUM(oi.quantity * oi.unit_price)) OVER w_daily AS daily_total_revenue,
DENSE_RANK() OVER w_rank AS product_sales_rank
FROM order_items oi
JOIN target_orders o ON oi.order_id = o.order_id
JOIN products p ON oi.product_id = p.product_id
JOIN category_tree ct ON p.category_id = ct.category_id
WHERE oi.unit_price >= 1000.00
GROUP BY p.product_id, p.product_name, DATE(o.order_date)
HAVING SUM(oi.quantity) >= 1
WINDOW
w_daily AS (PARTITION BY DATE(o.order_date)),
w_rank AS (PARTITION BY DATE(o.order_date) ORDER BY SUM(oi.quantity * oi.unit_price) DESC)
ORDER BY p.product_id, DATE(o.order_date), total_revenue DESC
LIMIT 100 OFFSET 0
The output will be as follows:
[
{
"Plan": {
"Node Type": "Limit",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 100.00,
"Actual Loops": 1,
"Disabled": false,
"Plans": [
{
"Node Type": "Recursive Union",
"Parent Relationship": "InitPlan",
"Subplan Name": "CTE category_tree",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 100.00,
"Actual Loops": 1,
"Disabled": false,
"Storage": "Memory",
"Maximum Storage": 35,
"Plans": [
{
"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Relation Name": "categories",
"Alias": "categories",
"Actual Rows": 1.00,
"Actual Loops": 1,
"Disabled": false,
"Filter": "(category_id = 1)",
"Rows Removed by Filter": 99
},
{
"Node Type": "Hash Join",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Async Capable": false,
"Join Type": "Inner",
"Actual Rows": 11.00,
"Actual Loops": 9,
"Disabled": false,
"Inner Unique": false,
"Hash Cond": "(c.parent_id = ct_1.category_id)",
"Plans": [
{
"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Relation Name": "categories",
"Alias": "c",
"Actual Rows": 100.00,
"Actual Loops": 9,
"Disabled": false
},
{
"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 11.11,
"Actual Loops": 9,
"Disabled": false,
"Hash Buckets": 1024,
"Original Hash Buckets": 1024,
"Hash Batches": 1,
"Original Hash Batches": 1,
"Peak Memory Usage": 9,
"Plans": [
{
"Node Type": "WorkTable Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"CTE Name": "category_tree",
"Alias": "ct_1",
"Actual Rows": 11.11,
"Actual Loops": 9,
"Disabled": false
}
]
}
]
}
]
},
{
"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 100.00,
"Actual Loops": 1,
"Disabled": false,
"Sort Key": ["p.product_id", "(date(orders.order_date))", "(sum(((oi.quantity)::numeric * oi.unit_price))) DESC"],
"Sort Method": "top-N heapsort",
"Sort Space Used": 47,
"Sort Space Type": "Memory",
"Plans": [
{
"Node Type": "WindowAgg",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 1882.00,
"Actual Loops": 1,
"Disabled": false,
"Window": "w_daily AS (PARTITION BY (date(orders.order_date)))",
"Storage": "Memory",
"Maximum Storage": 19,
"Plans": [
{
"Node Type": "WindowAgg",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 1882.00,
"Actual Loops": 1,
"Disabled": false,
"Window": "w_rank AS (PARTITION BY (date(orders.order_date)) ORDER BY (sum(((oi.quantity)::numeric * oi.unit_price))) ROWS UNBOUNDED PRECEDING)",
"Storage": "Memory",
"Maximum Storage": 17,
"Plans": [
{
"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 1882.00,
"Actual Loops": 1,
"Disabled": false,
"Sort Key": ["(date(orders.order_date))", "(sum(((oi.quantity)::numeric * oi.unit_price))) DESC"],
"Sort Method": "quicksort",
"Sort Space Used": 151,
"Sort Space Type": "Memory",
"Plans": [
{
"Node Type": "Aggregate",
"Strategy": "Hashed",
"Partial Mode": "Simple",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 1882.00,
"Actual Loops": 1,
"Disabled": false,
"Group Key": ["p.product_id", "date(orders.order_date)"],
"Filter": "(sum(oi.quantity) >= 1)",
"HashAgg Batches": 1,
"Peak Memory Usage": 1105,
"Disk Usage": 0,
"Rows Removed by Filter": 0,
"Plans": [
{
"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Join Type": "Inner",
"Actual Rows": 1918.00,
"Actual Loops": 1,
"Disabled": false,
"Inner Unique": false,
"Hash Cond": "(p.category_id = ct.category_id)",
"Plans": [
{
"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Join Type": "Inner",
"Actual Rows": 1918.00,
"Actual Loops": 1,
"Disabled": false,
"Inner Unique": true,
"Hash Cond": "(oi.product_id = p.product_id)",
"Plans": [
{
"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Join Type": "Inner",
"Actual Rows": 1918.00,
"Actual Loops": 1,
"Disabled": false,
"Inner Unique": true,
"Hash Cond": "(oi.order_id = orders.order_id)",
"Plans": [
{
"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Relation Name": "order_items",
"Alias": "oi",
"Actual Rows": 6038.00,
"Actual Loops": 1,
"Disabled": false,
"Filter": "(unit_price >= 1000.00)",
"Rows Removed by Filter": 3962
},
{
"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 323.00,
"Actual Loops": 1,
"Disabled": false,
"Hash Buckets": 1024,
"Original Hash Buckets": 1024,
"Hash Batches": 1,
"Original Hash Batches": 1,
"Peak Memory Usage": 22,
"Plans": [
{
"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Relation Name": "orders",
"Alias": "orders",
"Actual Rows": 323.00,
"Actual Loops": 1,
"Disabled": false,
"Filter": "((status)::text = 'PENDING'::text)",
"Rows Removed by Filter": 677
}
]
}
]
},
{
"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 500.00,
"Actual Loops": 1,
"Disabled": false,
"Hash Buckets": 1024,
"Original Hash Buckets": 1024,
"Hash Batches": 1,
"Original Hash Batches": 1,
"Peak Memory Usage": 34,
"Plans": [
{
"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"Relation Name": "products",
"Alias": "p",
"Actual Rows": 500.00,
"Actual Loops": 1,
"Disabled": false
}
]
}
]
},
{
"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Async Capable": false,
"Actual Rows": 100.00,
"Actual Loops": 1,
"Disabled": false,
"Hash Buckets": 1024,
"Original Hash Buckets": 1024,
"Hash Batches": 1,
"Original Hash Batches": 1,
"Peak Memory Usage": 12,
"Plans": [
{
"Node Type": "CTE Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Async Capable": false,
"CTE Name": "category_tree",
"Alias": "ct",
"Actual Rows": 100.00,
"Actual Loops": 1,
"Disabled": false,
"Storage": "Memory",
"Maximum Storage": 21
}
]
}
]
}
]
}
]
}
]
}
]
}
]
}
]
},
"Triggers": [
]
}
]
This query is used to calculate statistics and rank sales of high-value products (>1000) by day, executing each step in order of the Bottom-Up data flow:
- Uses recursion to retrieve all products with
category_id = 1and products belonging to its sub-categories, saving the result in aCTEnamedcategory_tree target_ordersstores orders withstatus=PENDING- Data Preparation
- Filters high-priced order details from
order_itemsaccording to the conditionoi.unit_price >= 1000.00, resulting in"Actual Rows": 6038.00 - Loads table
productsaccording to the conditionoi.product_id = p.product_id, resulting in"Actual Rows": 500.00 - Loads
CTE category_treeaccording to the conditionp.category_id = ct.category_id, resulting in"Actual Rows": 100.00
- Filters high-priced order details from
- Uses consecutive
Hash Joinoperations as follows:- Joins
order_itemswithorderswith condition"Hash Cond": "(oi.order_id = orders.order_id)" - Joins the result above with
productswith condition"Hash Cond": "(oi.product_id = p.product_id)" - Joins the result above with
category_treewith condition"Hash Cond": "(p.category_id = ct.category_id)" - The output consists of 1,918 rows.
- Joins
- Groups data into
1,882rows by product and date, calculating totals withHAVING SUM(oi.quantity) >= 1. - Sorting and
Window Functionscomputation- Sorting for Window Functions is
(PARTITION BY DATE(o.order_date))andPARTITION BY DATE(o.order_date) ORDER BY SUM(...) DESC - Calculates product sales ranking for
product_sales_rankviaDENSE_RANK() OVER w_rank - Calculates daily total revenue for
daily_total_revenueviaSUM(SUM(oi.quantity * oi.unit_price)) OVER w_daily
- Sorting for Window Functions is
- Sorts via
ORDER BY p.product_id, DATE(o.order_date), total_revenue DESC - Performs pagination via
LIMIT 100 OFFSET 0
Happy coding!
Comments
Post a Comment