Feature request: volume anomalies pre-aggregate data before computing statistics

[garfieldthesam]

I need to run volume anomaly tests on very large tables. However, I cannot performantly do so because the compiled query does not pre-aggregate the row count data. For example this is what the first CTE looks like for a test run on my Databricks cluster:

with monitored_table as (
  select
    *
  from
    <very large table>
  where
    <where predicates>
),

For large tables (especially ones that are very wide), this is prohibitively expensive for 2 reasons:

• A select * fetches all table columns, which isn't necessary in principle for creating a time series; this is especially costly for columnar databases
• An aggregated time series can be generated from the table, providing a much smaller dataset for later CTEs in the compiled query to apply their transformations/filters to

As a result we've had to create a cumbersome system where we create derived data quality metrics tables summarizing the large table's metrics each day, and then run elementary column tests on those.

I'd like to request a rearchitecture of how the volume anomalies code works to improve performance for large tables.

[garfieldthesam]

Closing and turning this into an issue instead