SQL order of execution

Introduction

  • In SQL in general and PostgreSQL in particular, the order of writing keywords in code (Lexical Order) is completely different from the order in which the Database Engine actually executes the statement (Logical Execution Order).
  • A query with full command structures like this will be parsed and executed by the Optimizer in a strict logical order from root (data source) to top (displayed results).
[ WITH [RECURSIVE] cte_name AS (...) ]
SELECT [ DISTINCT [ON (...)] ] columns
FROM source_table [ TABLESAMPLE ... ]
[ JOIN_clauses ]
[ WHERE raw_filter_conditions ]
[ GROUP BY { column | ROLLUP | CUBE | GROUPING SETS } ]
[ HAVING group_filter_conditions ]
[ WINDOW window_name AS (...) ]
[ { UNION | INTERSECT | EXCEPT } [ALL | DISTINCT] other_select_statement ]
[ ORDER BY sort_order ]
[ LIMIT limit_count ] [ OFFSET offset_count ]
[ FOR UPDATE / FOR SHARE [ SKIP LOCKED ] ];

Below is the execution order from the first step to the last step:

  • WITH / WITH RECURSIVE
    • This is the way to use CTE (Common Table Expressions) to execute statements inside the WITH block first to prepare temporary data tables
    • If it is RECURSIVE, it will run a loop to generate data until the stop condition is met.
  • FROM and TABLESAMPLE
    • Identify tables to retrieve data
    • If TABLESAMPLE is present, Postgres will only extract a random sample amount (for example 10% of table rows) for further processing, instead of the whole table.
  • JOIN clauses
    • Used to match data to create an expanded Cartesian product table, combining secondary tables into the source table based on operators (INNER, LEFT, RIGHT, FULL JOIN) and conditions after the ON keyword
    • At the end of this step, we have a temporary "super table" containing all columns of participating tables.
  • WHERE
    • Filters raw data by iterating through each row to keep rows that satisfy the WHERE condition, while non-satisfying rows are eliminated
    • Note that aggregate calculation functions (such as SUM, COUNT) cannot be used at this point because data has not been grouped yet.
  • GROUP BY
    • The remaining rows after filtering will be grouped based on specified columns.
    • Super functions like ROLLUP or CUBE will automatically generate subtotal and grand total rows at this step.
  • HAVING
    • Filters data after grouping based on aggregate functions
    • After groups are formed, the HAVING condition will be applied to eliminate groups that do not satisfy the condition.
  • WINDOW
    • Uses a combination of Aggregate Function (such as ROW_NUMBER(), RANK(), SUM(),...) and Window Function Clause (OVER) to compute aggregates based on 1 column
    • Unlike GROUP BY, this step computes based on partitions while keeping the original number of rows without collapsing rows.
  • SELECT and DISTINCT [ON]: Column extraction and deduplication
    • At this point, the Database calculates expressions and sets Alias based on the specified column list
    • If DISTINCT is present, it eliminates duplicate rows
    • With DISTINCT ON (...), it keeps the first unique row for each set of duplicate values.
  • Set operations UNION / INTERSECT / EXCEPT: Union, intersection and difference of statements.
    • If you have multiple SELECT statements connected by set operators, the Database takes the complete result of this statement to merge, intersect or subtract from the result of another SELECT statement
    • If ALL is used, it keeps everything, if DISTINCT is used (default), it scans to filter duplicates again.
  • ORDER BY: Display sorting
    • Final data is sorted according to specified columns or expressions.
    • Because this step runs after SELECT, Aliases defined in the SELECT step can be used for sorting.
  • LIMIT and OFFSET
    • Slices results, commonly used for data pagination.
    • Skips a number of initial rows (OFFSET) and keeps only the maximum requested number of rows (LIMIT)
    • This is performed at the very end to ensure returning only the exact amount of data needed by the client.
  • FOR UPDATE / FOR SHARE: Concurrency Control
    • After determining the exact final result rows to return to the user, Postgres locks those rows in memory for the Transaction
    • If SKIP LOCKED is used, it automatically skips rows currently locked by another session to avoid blocking.
    • In addition, there are other Row level locks with different lock levels

Detail

First, create Tables and Indexes as follows:

CREATE TABLE categories (
    category_id INT PRIMARY KEY,
    category_name VARCHAR(100) NOT NULL,
    parent_id INT REFERENCES categories(category_id)
);

CREATE TABLE products (
    product_id INT PRIMARY KEY,
    product_name VARCHAR(200) NOT NULL,
    category_id INT REFERENCES categories(category_id),
    price NUMERIC(12, 2) NOT NULL,
    stock_quantity INT NOT NULL DEFAULT 0
);

CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT NOT NULL,
    order_date TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    status VARCHAR(30) NOT NULL 
);

CREATE TABLE order_items (
    item_id INT PRIMARY KEY,
    order_id INT REFERENCES orders(order_id),
    product_id INT REFERENCES products(product_id),
    quantity INT NOT NULL,
    unit_price NUMERIC(12, 2) NOT NULL
);

Next is a comprehensive example describing the execution order:

WITH RECURSIVE category_tree AS (
    SELECT category_id, category_name, parent_id
    FROM categories
    WHERE category_id = 1
    
    UNION ALL
    
    SELECT c.category_id, c.category_name, c.parent_id
    FROM categories c
    JOIN category_tree ct ON c.parent_id = ct.category_id
),
target_orders AS (
    SELECT order_id, customer_id, order_date, status
    FROM orders
    WHERE status = 'PENDING'
)
SELECT 
    p.product_id,
    p.product_name,
    DATE(o.order_date) AS sales_date,
    SUM(oi.quantity) AS total_quantity_sold,
    SUM(oi.quantity * oi.unit_price) AS total_revenue,
    SUM(SUM(oi.quantity * oi.unit_price)) OVER w_daily AS daily_total_revenue,
    DENSE_RANK() OVER w_rank AS product_sales_rank
FROM order_items oi
JOIN target_orders o ON oi.order_id = o.order_id
JOIN products p ON oi.product_id = p.product_id
JOIN category_tree ct ON p.category_id = ct.category_id
WHERE oi.unit_price >= 1000.00
GROUP BY p.product_id, p.product_name, DATE(o.order_date)
HAVING SUM(oi.quantity) >= 1
WINDOW 
    w_daily AS (PARTITION BY DATE(o.order_date)),
    w_rank AS (PARTITION BY DATE(o.order_date) ORDER BY SUM(oi.quantity * oi.unit_price) DESC)
ORDER BY p.product_id, DATE(o.order_date), total_revenue DESC
LIMIT 100 OFFSET 0





The output will be as follows:

[
  {
    "Plan": {
      "Node Type": "Limit",
      "Parallel Aware": false,
      "Async Capable": false,
      "Actual Rows": 100.00,
      "Actual Loops": 1,
      "Disabled": false,
      "Plans": [
        {
          "Node Type": "Recursive Union",
          "Parent Relationship": "InitPlan",
          "Subplan Name": "CTE category_tree",
          "Parallel Aware": false,
          "Async Capable": false,
          "Actual Rows": 100.00,
          "Actual Loops": 1,
          "Disabled": false,
          "Storage": "Memory",
          "Maximum Storage": 35,
          "Plans": [
            {
              "Node Type": "Seq Scan",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Async Capable": false,
              "Relation Name": "categories",
              "Alias": "categories",
              "Actual Rows": 1.00,
              "Actual Loops": 1,
              "Disabled": false,
              "Filter": "(category_id = 1)",
              "Rows Removed by Filter": 99
            },
            {
              "Node Type": "Hash Join",
              "Parent Relationship": "Inner",
              "Parallel Aware": false,
              "Async Capable": false,
              "Join Type": "Inner",
              "Actual Rows": 11.00,
              "Actual Loops": 9,
              "Disabled": false,
              "Inner Unique": false,
              "Hash Cond": "(c.parent_id = ct_1.category_id)",
              "Plans": [
                {
                  "Node Type": "Seq Scan",
                  "Parent Relationship": "Outer",
                  "Parallel Aware": false,
                  "Async Capable": false,
                  "Relation Name": "categories",
                  "Alias": "c",
                  "Actual Rows": 100.00,
                  "Actual Loops": 9,
                  "Disabled": false
                },
                {
                  "Node Type": "Hash",
                  "Parent Relationship": "Inner",
                  "Parallel Aware": false,
                  "Async Capable": false,
                  "Actual Rows": 11.11,
                  "Actual Loops": 9,
                  "Disabled": false,
                  "Hash Buckets": 1024,
                  "Original Hash Buckets": 1024,
                  "Hash Batches": 1,
                  "Original Hash Batches": 1,
                  "Peak Memory Usage": 9,
                  "Plans": [
                    {
                      "Node Type": "WorkTable Scan",
                      "Parent Relationship": "Outer",
                      "Parallel Aware": false,
                      "Async Capable": false,
                      "CTE Name": "category_tree",
                      "Alias": "ct_1",
                      "Actual Rows": 11.11,
                      "Actual Loops": 9,
                      "Disabled": false
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "Node Type": "Sort",
          "Parent Relationship": "Outer",
          "Parallel Aware": false,
          "Async Capable": false,
          "Actual Rows": 100.00,
          "Actual Loops": 1,
          "Disabled": false,
          "Sort Key": ["p.product_id", "(date(orders.order_date))", "(sum(((oi.quantity)::numeric * oi.unit_price))) DESC"],
          "Sort Method": "top-N heapsort",
          "Sort Space Used": 47,
          "Sort Space Type": "Memory",
          "Plans": [
            {
              "Node Type": "WindowAgg",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Async Capable": false,
              "Actual Rows": 1882.00,
              "Actual Loops": 1,
              "Disabled": false,
              "Window": "w_daily AS (PARTITION BY (date(orders.order_date)))",
              "Storage": "Memory",
              "Maximum Storage": 19,
              "Plans": [
                {
                  "Node Type": "WindowAgg",
                  "Parent Relationship": "Outer",
                  "Parallel Aware": false,
                  "Async Capable": false,
                  "Actual Rows": 1882.00,
                  "Actual Loops": 1,
                  "Disabled": false,
                  "Window": "w_rank AS (PARTITION BY (date(orders.order_date)) ORDER BY (sum(((oi.quantity)::numeric * oi.unit_price))) ROWS UNBOUNDED PRECEDING)",
                  "Storage": "Memory",
                  "Maximum Storage": 17,
                  "Plans": [
                    {
                      "Node Type": "Sort",
                      "Parent Relationship": "Outer",
                      "Parallel Aware": false,
                      "Async Capable": false,
                      "Actual Rows": 1882.00,
                      "Actual Loops": 1,
                      "Disabled": false,
                      "Sort Key": ["(date(orders.order_date))", "(sum(((oi.quantity)::numeric * oi.unit_price))) DESC"],
                      "Sort Method": "quicksort",
                      "Sort Space Used": 151,
                      "Sort Space Type": "Memory",
                      "Plans": [
                        {
                          "Node Type": "Aggregate",
                          "Strategy": "Hashed",
                          "Partial Mode": "Simple",
                          "Parent Relationship": "Outer",
                          "Parallel Aware": false,
                          "Async Capable": false,
                          "Actual Rows": 1882.00,
                          "Actual Loops": 1,
                          "Disabled": false,
                          "Group Key": ["p.product_id", "date(orders.order_date)"],
                          "Filter": "(sum(oi.quantity) >= 1)",
                          "HashAgg Batches": 1,
                          "Peak Memory Usage": 1105,
                          "Disk Usage": 0,
                          "Rows Removed by Filter": 0,
                          "Plans": [
                            {
                              "Node Type": "Hash Join",
                              "Parent Relationship": "Outer",
                              "Parallel Aware": false,
                              "Async Capable": false,
                              "Join Type": "Inner",
                              "Actual Rows": 1918.00,
                              "Actual Loops": 1,
                              "Disabled": false,
                              "Inner Unique": false,
                              "Hash Cond": "(p.category_id = ct.category_id)",
                              "Plans": [
                                {
                                  "Node Type": "Hash Join",
                                  "Parent Relationship": "Outer",
                                  "Parallel Aware": false,
                                  "Async Capable": false,
                                  "Join Type": "Inner",
                                  "Actual Rows": 1918.00,
                                  "Actual Loops": 1,
                                  "Disabled": false,
                                  "Inner Unique": true,
                                  "Hash Cond": "(oi.product_id = p.product_id)",
                                  "Plans": [
                                    {
                                      "Node Type": "Hash Join",
                                      "Parent Relationship": "Outer",
                                      "Parallel Aware": false,
                                      "Async Capable": false,
                                      "Join Type": "Inner",
                                      "Actual Rows": 1918.00,
                                      "Actual Loops": 1,
                                      "Disabled": false,
                                      "Inner Unique": true,
                                      "Hash Cond": "(oi.order_id = orders.order_id)",
                                      "Plans": [
                                        {
                                          "Node Type": "Seq Scan",
                                          "Parent Relationship": "Outer",
                                          "Parallel Aware": false,
                                          "Async Capable": false,
                                          "Relation Name": "order_items",
                                          "Alias": "oi",
                                          "Actual Rows": 6038.00,
                                          "Actual Loops": 1,
                                          "Disabled": false,
                                          "Filter": "(unit_price >= 1000.00)",
                                          "Rows Removed by Filter": 3962
                                        },
                                        {
                                          "Node Type": "Hash",
                                          "Parent Relationship": "Inner",
                                          "Parallel Aware": false,
                                          "Async Capable": false,
                                          "Actual Rows": 323.00,
                                          "Actual Loops": 1,
                                          "Disabled": false,
                                          "Hash Buckets": 1024,
                                          "Original Hash Buckets": 1024,
                                          "Hash Batches": 1,
                                          "Original Hash Batches": 1,
                                          "Peak Memory Usage": 22,
                                          "Plans": [
                                            {
                                              "Node Type": "Seq Scan",
                                              "Parent Relationship": "Outer",
                                              "Parallel Aware": false,
                                              "Async Capable": false,
                                              "Relation Name": "orders",
                                              "Alias": "orders",
                                              "Actual Rows": 323.00,
                                              "Actual Loops": 1,
                                              "Disabled": false,
                                              "Filter": "((status)::text = 'PENDING'::text)",
                                              "Rows Removed by Filter": 677
                                            }
                                          ]
                                        }
                                      ]
                                    },
                                    {
                                      "Node Type": "Hash",
                                      "Parent Relationship": "Inner",
                                      "Parallel Aware": false,
                                      "Async Capable": false,
                                      "Actual Rows": 500.00,
                                      "Actual Loops": 1,
                                      "Disabled": false,
                                      "Hash Buckets": 1024,
                                      "Original Hash Buckets": 1024,
                                      "Hash Batches": 1,
                                      "Original Hash Batches": 1,
                                      "Peak Memory Usage": 34,
                                      "Plans": [
                                        {
                                          "Node Type": "Seq Scan",
                                          "Parent Relationship": "Outer",
                                          "Parallel Aware": false,
                                          "Async Capable": false,
                                          "Relation Name": "products",
                                          "Alias": "p",
                                          "Actual Rows": 500.00,
                                          "Actual Loops": 1,
                                          "Disabled": false
                                        }
                                      ]
                                    }
                                  ]
                                },
                                {
                                  "Node Type": "Hash",
                                  "Parent Relationship": "Inner",
                                  "Parallel Aware": false,
                                  "Async Capable": false,
                                  "Actual Rows": 100.00,
                                  "Actual Loops": 1,
                                  "Disabled": false,
                                  "Hash Buckets": 1024,
                                  "Original Hash Buckets": 1024,
                                  "Hash Batches": 1,
                                  "Original Hash Batches": 1,
                                  "Peak Memory Usage": 12,
                                  "Plans": [
                                    {
                                      "Node Type": "CTE Scan",
                                      "Parent Relationship": "Outer",
                                      "Parallel Aware": false,
                                      "Async Capable": false,
                                      "CTE Name": "category_tree",
                                      "Alias": "ct",
                                      "Actual Rows": 100.00,
                                      "Actual Loops": 1,
                                      "Disabled": false,
                                      "Storage": "Memory",
                                      "Maximum Storage": 21
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    "Triggers": [
    ]
  }
]

This query is used to calculate statistics and rank sales of high-value products (>1000) by day, executing each step in order of the Bottom-Up data flow:

  • Uses recursion to retrieve all products with category_id = 1 and products belonging to its sub-categories, saving the result in a CTE named category_tree
  • target_orders stores orders with status=PENDING
  • Data Preparation
    • Filters high-priced order details from order_items according to the condition oi.unit_price >= 1000.00, resulting in "Actual Rows": 6038.00
    • Loads table products according to the condition oi.product_id = p.product_id, resulting in "Actual Rows": 500.00
    • Loads CTE category_tree according to the condition p.category_id = ct.category_id, resulting in "Actual Rows": 100.00
  • Uses consecutive Hash Join operations as follows:
    • Joins order_items with orders with condition "Hash Cond": "(oi.order_id = orders.order_id)"
    • Joins the result above with products with condition "Hash Cond": "(oi.product_id = p.product_id)"
    • Joins the result above with category_tree with condition "Hash Cond": "(p.category_id = ct.category_id)"
    • The output consists of 1,918 rows.
  • Groups data into 1,882 rows by product and date, calculating totals with HAVING SUM(oi.quantity) >= 1.
  • Sorting and Window Functions computation
    • Sorting for Window Functions is (PARTITION BY DATE(o.order_date)) and PARTITION BY DATE(o.order_date) ORDER BY SUM(...) DESC
    • Calculates product sales ranking for product_sales_rank via DENSE_RANK() OVER w_rank
    • Calculates daily total revenue for daily_total_revenue via SUM(SUM(oi.quantity * oi.unit_price)) OVER w_daily
  • Sorts via ORDER BY p.product_id, DATE(o.order_date), total_revenue DESC
  • Performs pagination via LIMIT 100 OFFSET 0

Happy coding!

See more articles here.

Comments

Popular posts from this blog

All Practice Series

Kubernetes Deployment for Zero Downtime

Deploying a NodeJS Server on Google Kubernetes Engine

Sitemap

React Practice Series

Docker Practice Series

Helm for beginer - Deploy nginx to Google Kubernetes Engine

A Handy Guide to Using Dynamic Import in JavaScript

DevOps Practice Series

Kubernetes Practice Series