DRILL-8537: Bump Calcite to Version 1.42

[cgivre]

DRILL-8537: Bump Calcite to Version 1.42

Description

I am attempting a new approach and instead of bumping Calcite from 1.34 -> 1.40, I'm going to try this one version at a time and see how far we get.

After reaching a known-good 1.38 (all unit tests passing), the remaining bump was done in a single step from 1.38 → 1.42 (skipping the 1.39 regressions, which are resolved by 1.42). Avatica was bumped 1.23 → 1.28 to match Calcite 1.42.

Current Status:

• Update to 1.35
• Update to 1.36
• Update to 1.37
• Update to 1.38
• Update to 1.42 (Avatica 1.28)

Significant Changes in Calcite 1.35

• Calcite 1.35 adds a literal_agg function which allows literals in aggregate queries.
• Fix bugs in the ElasticSearch plugin which prevented certain configuration parameters from being passed to ElasticSearch.
• Fix bugs introduced in Calcite 1.34 relating to certain date arithmetic functions
• Update issues in Drill relating to Calcite's new VARDECIMAL handling.

Significant Changes in Calcite 1.36

There are no significant changes in Calcite 1.36.

Significant Changes in Calcite 1.37

• SqlRandFunction Removed: The dedicated SqlRandFunction class was removed in Calcite 1.37. The RAND function is now implemented as a SqlBasicFunction in SqlStdOperatorTable.
• INTERVAL Type Precision Calculation Changed: Calcite 1.37 changed how it calculates INTERVAL_PRECISION and COLUMN_SIZE for interval types, reporting values based on the full ISO-8601 representation rather than just the specified field precision.
• Expression Simplification Performance Regression: Performance Issue: Calcite 1.37 exhibits exponential planning complexity when processing large IN clauses with expressions (not just literals), particularly during RexNode simplification and OR expression expansion.
• Lambda Expressions: Calcite 1.37 adds lambda expression support. Drill does not support this as yet, but the planning is there.
• Apache Arrow: Calcite now has an adapter for Apache Arrow. If there is interest it would be possible to create a plugin for Drill which could read Arrow data.

Significant Changes in Calcite 1.38

• Implemented SQL standard EXCLUDE clause for window functions - All 4 modes supported (NO OTHERS, CURRENT ROW, GROUP, TIES)
• Enhanced type checking compatibility - Created DrillSqlToRelConverter for graceful handling of Calcite 1.38's strict type checking
• Fixed STDDEV/VAR aggregate infinite loop - Disabled problematic aggregate reduction causing RexChecker recursion
• Resolved DECIMAL precision handling - Updated tests for Calcite 1.38's new DECIMAL type derivation (default precision 19, multiplication scale changes)
• Fixed CONCAT operator precision - Updated VARCHAR precision from 85 to 120 to match Calcite 1.38 type inference
• Upgraded ClickHouse test infrastructure - Updated to ClickHouse 23.8 for DECIMAL CAST syntax compatibility
• JoinPushTransitivePredicatesRule remains disabled - Kept disabled due to CALCITE-6432 infinite loop bug (fixed in Calcite 1.40)
• 2 partition pruning optimization tests degraded - Queries produce correct results but may scan more partitions than optimal in some scenarios
• Calcite now supports ASOF joins, but Drill does not yet support that yet.

Significant Changes in Calcite 1.39 – 1.42

Bumped directly from 1.38 to 1.42 (with Avatica 1.23 → 1.28). The notable breaking changes and how Drill adapts to them:

Schema resolution rewritten to a Lookup API (CALCITE-6029, 1.39). CalciteSchema no longer exposes getImplicitSubSchema / getImplicitTable; sub-schema and table resolution now go through subSchemas() / tables() Lookups. DynamicSchema / DynamicRootSchema were rewritten accordingly:

• Storage-plugin schemas are still loaded lazily (and by alias) via an overridden subSchemas(). Both get and getIgnoreCase trigger the lazy load, so single-identifier multi-level names such as `cp.default` / `dfs.tmp` keep working.
• Table lookups remain exact/case-sensitive (avoiding directory-listing scans) and temporary tables are resolved through an overridden tables().
• AbstractSchema (the base for every storage-plugin schema) also overrides subSchemas(). Calcite 1.42's default case-insensitive getIgnoreCase matches only against getSubSchemaNames(), but several plugins expose sub-schemas lazily via getSubSchema(name) without enumerating them (Cassandra keyspaces, Mongo databases, …). The override keeps the default name-based match and adds a lazy getSubSchema(name) fallback, so multi-level references like cassandra.test_keyspace.employee resolve again (these regressed to Object 'test_keyspace' not found within 'cassandra'). Enumerated schemas are unaffected since the name-based match wins first.

RexVisitor.visitNodeAndFieldIndex (1.41). Implemented in JdbcExpressionCheck (the only direct RexVisitor implementation); all other visitors extend RexVisitorImpl and inherit the default.

Type-system methods made final (1.42). RelDataTypeSystem.getMaxNumericPrecision() / getMaxNumericScale() are now final; removed Drill's overrides — the logic already lives in getMaxPrecision / getMaxScale (which Drill overrides to 38 for DECIMAL).

Window output column naming (1.42). Calcite no longer names window-function output columns w<group>$.... WindowPrule now selects each group's output fields positionally (after the input fields) instead of by name prefix; the old name-based filter silently dropped all window columns, producing "field sizes are not equal".

TRIM grammar (1.42). CalciteResource.illegalFromEmpty() was removed; Parser.jj raises a ParseException for TRIM(FROM x) while preserving Drill's TRIM(<flag> <chars>) extension.

Avatica 1.28. Cursor.Accessor gained unsigned accessors (getUByte/getUShort/getUInt/getULong, returning jOOU types) and AvaticaSite.get gained a signed parameter. Implemented the unsigned accessors in AvaticaDrillSqlAccessor (Drill has only signed types, so they mirror the signed getters) and pass signed=true.

Stronger plan-time constant reduction. Calcite 1.42 reduces more aggressively and builds Sargs during predicate inference, which surfaced two issues in DrillConstExecutor:

• DATE + INTERVAL YEAR is typed DATE by Calcite but computed as a TIMESTAMP by Drill; folding it to a TIMESTAMP literal then landing in a DATE Sarg failed with TimestampString cannot be cast to DateString. The executor now emits a DATE literal whenever Calcite types the expression as DATE.
• Some CHAR constants make Drill's interpreter return no value; the executor now leaves such expressions unfolded instead of NPE-ing.

ROW() in a constant context (1.42). DrillOptiq's ROW handling built field-name literals via getRexBuilder(), which is null when converting a standalone (constant-folded) expression; it now builds the Drill string literal directly.

LIKE ... ESCAPE when the escape character is a wildcard. Calcite 1.42's RexSimplify.simplifyLike mishandles a LIKE whose ESCAPE character is also a wildcard ('%'/'_') — it collapses the escaped wildcard and produces an altered pattern. Filter-expression reduction is skipped when the condition contains such a LIKE, preserving the original pattern.

Constant VALUES collation. Calcite 1.42 derives collations for constant/single-row VALUES and propagates them. Drill applies ordering in a later physical phase and has no logical collation-conversion rules, so this caused CannotPlanException: ... sort=[...]. Derived collations are stripped before logical Volcano planning; a Sort's own collation (an explicit ORDER BY) is preserved.

GROUP BY by alias. Calcite 1.42 stopped expanding GROUP BY items that reference a SELECT alias (it still does for HAVING), so GROUP BY <alias> failed validation even though DrillConformance.isGroupByAlias() is true. A pre-validation GroupByAliasRewriter restores the behavior.

BOOLEAN in arithmetic. Calcite 1.42 coerces BOOLEAN to a numeric type in arithmetic, emitting casts (e.g. castINT(BIT)) that Drill did not implement. Added Bit -> Int/BigInt/Float4/Float8 cast functions (true -> 1, false -> 0).

ORDER BY validation performance (wide queries). Calcite 1.42's validateOrderList calls isAggregate(select) once for every ORDER BY item (via OrderExpressionExpander → nthSelectItem → expandSelectItem), and each call rescans the entire SELECT list with AggFinder. For wide queries this is O(orderByItems × selectListSize) — TestLargeFileCompilation's 2500-column projection ordered by 500 columns took >200s to validate and timed out. Since isAggregate depends only on GROUP BY / HAVING / the SELECT list (none of which change during ORDER BY expansion), DrillSqlValidator now memoizes it per SELECT node by identity. Planning a 1600×320 query drops from ~127s to ~2.5s.

Constant propagation into projections over ANY columns. Under schema-on-read a scan column is typed ANY and = is an implicit-cast comparison, not a type-strict equality. Calcite 1.42 propagates equality constants from pulled-up predicates more aggressively, so a query like SELECT float ... WHERE float = '1.2' rewrote the projected ANY float column into the compared VARCHAR literal and returned the string "1.2" instead of the double 1.2. Both project reduce rules (DrillReduceExpressionsRule and ReduceAndSimplifyExpressionsRules) now strip ANY-column equality predicates before simplifying, so the column keeps its real (FLOAT8) type.

Degenerate cartesian joins in set-op-in-WITH tests. Some TPC-DS-derived tests (TestSetOp.testIntersectWith, TestUnionAll.testUnionAllInWith) self-join on a constant literal column (year_total = year_total), which is a genuine cartesian join. Calcite 1.42 correctly folds the constant join key and identifies the cartesian (earlier versions kept a redundant equi-join), so those queries now require Drill's non-scalar nested-loop join — the tests enable planner.enable_nljoin_for_scalar_only=false for the affected queries.

Other behavioral notes (test baselines updated):

• A decimal-union widening that overflows the maximum precision of 38 now resolves by reducing scale (e.g. DECIMAL(38,4)) rather than dropping integer digits (DECIMAL(38,6)).

Documentation

No user facing changes.

Testing

Ran existing unit tests.

[cgivre]

Major Changes

1. Function Type Inference

EXTRACT Function

Problem: EXTRACT(SECOND) was returning BIGINT instead of DOUBLE, losing fractional seconds
Solution:

• Created DrillCalciteSqlExtractWrapper with custom type inference
• Updated DrillConvertletTable.extractConvertlet() to use TypeInferenceUtils.getSqlTypeNameForTimeUnit()
• Returns DOUBLE for SECOND, BIGINT for other time units

Files Modified:

• DrillCalciteSqlExtractWrapper.java (new)
• DrillConvertletTable.java
• DrillOperatorTable.java
• TestFunctionsWithTypeExpoQueries.java

TIMESTAMPDIFF Function

Problem: Type mismatch between validation (BIGINT) and conversion (INTEGER)
Solution:

• Created DrillCalciteSqlTimestampDiffWrapper for consistent BIGINT return type
• Updated DrillConvertletTable.timestampDiffConvertlet() to use BIGINT
• Both validation and conversion now consistently return BIGINT

Files Modified:

• DrillCalciteSqlTimestampDiffWrapper.java (new)
• DrillConvertletTable.java
• DrillOperatorTable.java

TIMESTAMPADD Function

Problem: Calcite 1.35 was adding precision to DATE types, causing assertion errors
Solution:

• Created DrillCalciteSqlTimestampAddWrapper with proper type logic
• Only adds precision to TIMESTAMP and TIME types, not DATE
• Updated DrillConvertletTable.timestampAddConvertlet() to skip precision for DATE

Files Modified:

• DrillCalciteSqlTimestampAddWrapper.java (new)
• DrillConvertletTable.java
• DrillOperatorTable.java

2. Function Registration & Resolution

Vararg Functions (CONCAT, COALESCE, etc.)

Problem: Function resolution failures for functions with variable arguments
Solution: Enhanced LocalFunctionRegistry with sophisticated vararg matching logic

Files Modified:

• LocalFunctionRegistry.java (+57 lines)

Niladic Special Functions (CURRENT_DATE, SESSION_USER, etc.)

Problem: Special functions not properly recognized in Calcite 1.35
Solution:

• Created SpecialFunctionRewriter to handle niladic function transformations
• Added explicit registration in DrillOperatorTable

Files Modified:

• SpecialFunctionRewriter.java (+84 lines, new)
• CountFunctionRewriter.java (+52 lines, new)
• CharToVarcharRewriter.java (+61 lines, new)
• DrillOperatorTable.java

3. COUNT(*) Handling

Problem: COUNT(*) type inference changed in Calcite 1.35
Solution: Created LiteralAggFunction for proper literal aggregate handling

Files Modified:

• LiteralAggFunction.java (+192 lines, new)
• DrillAggregateRel.java
• AggPrelBase.java

4. Aggregate Cost Estimation

Problem: Deprecated RelOptCost constructor removed
Solution: Updated to use withAggCallCount() builder pattern

Files Modified:

• DrillAggregateRel.java
• AggPrelBase.java
• DrillReduceAggregatesRule.java

5. TIMESTAMPADD Implementation

Problem: Complete signature change in Calcite 1.35
Solution:

• Reimplemented TIMESTAMPADD function with new interval arithmetic
• Added template-based code generation

Files Modified:

• DateIntervalFunc.tdd
• TimestampAddFunction.java (+203 lines, new)
• DrillConvertletTable.java

6. Complex Writer Functions

Problem: FLATTEN, CONVERT_FROM, CONVERT_TO require ProjectRecordBatch context
Solution:

• Modified DrillConstExecutor to skip constant folding for these functions
• Updated dummy function implementations for Calcite 1.35 compatibility

Files Modified:

• DrillConstExecutor.java
• DummyConvertFrom.java
• DummyConvertTo.java
• DummyFlatten.java

7. FLATTEN in Aggregates Validation

Problem: FLATTEN only validated in COUNT, allowed in other aggregates
Solution: Changed validation from SqlCountAggFunction to SqlAggFunction to catch ALL aggregate types

Files Modified:

• UnsupportedOperatorsVisitor.java

8. Error Handling & Validation

Prepared Statement Errors

Problem: Parse errors wrapped differently in RPC layer, appearing as SYSTEM instead of VALIDATION
Solution:

• Added exception unwrapping in SqlConverter.parse()
• Updated test expectations to match Calcite 1.35 behavior

Files Modified:

• SqlConverter.java
• TestPreparedStatementProvider.java

Invalid CAST Operations

Problem: Calcite 1.35 correctly rejects semantically invalid CAST(DATE as TIME)
Solution: Removed invalid test case with explanatory comment

Files Modified:

• TestParquetFilterPushDownForDateTimeCasts.java

---

Test Updates

Core Module Tests

• TestFunctionsQuery.java - Updated for DOUBLE return from EXTRACT(SECOND)
• TestFunctionsWithTypeExpoQueries.java - Fixed type expectations (FLOAT8 for SECOND)
• TestTimestampAddDiffFunctions.java - Updated for new type inference
• TestCountStar.java - Fixed test infrastructure (changed to PlanTestBase)
• TestParquetFilterPushDownForDateTimeCasts.java - Removed invalid cast test
• TestAggregateFunctions.java - Updated aggregate type expectations
• TestLiteralAggFunction.java (+241 lines, new) - Comprehensive literal aggregate testing

JDBC Storage Plugin Tests

• TestJdbcPluginWithMySQLIT.java - Updated SQRT to return DOUBLE instead of BigDecimal
• TestJdbcPluginWithMSSQL.java - Fixed schema expectations (BIGINT for COUNT, REQUIRED types)
• TestJdbcPluginWithPostgres.java - Fixed schema expectations (BIGINT for COUNT, REQUIRED types)

---

Key Behavioral Changes

Type Inference

1. EXTRACT(SECOND) now returns DOUBLE (was BIGINT) - supports fractional seconds
2. TIMESTAMPDIFF returns BIGINT consistently
3. COUNT(*) returns BIGINT (was INT in some contexts)
4. SQRT and math functions consistently return DOUBLE
5. Literal expressions and aggregates are REQUIRED (not OPTIONAL/nullable)

Function Resolution

1. Vararg functions (CONCAT, COALESCE) now match multiple signatures
2. Niladic functions (CURRENT_DATE, etc.) properly transformed with parentheses
3. Special functions (COUNT(*), FLATTEN, etc.) have dedicated handling

Validation

1. Stricter type checking between validation and conversion phases
2. Invalid casts (DATE→TIME) now properly rejected
3. FLATTEN properly rejected in ALL aggregate functions (not just COUNT)

---

Files Created

Core Engine

1. DrillCalciteSqlExtractWrapper.java - Custom EXTRACT type inference
2. DrillCalciteSqlTimestampAddWrapper.java - Custom TIMESTAMPADD type inference
3. DrillCalciteSqlTimestampDiffWrapper.java - Custom TIMESTAMPDIFF type inference
4. SpecialFunctionRewriter.java - Niladic function rewriting
5. CountFunctionRewriter.java - COUNT(*) rewriting
6. CharToVarcharRewriter.java - CHAR to VARCHAR conversion
7. LiteralAggFunction.java - Literal aggregate handling
8. TimestampAddFunction.java - New TIMESTAMPADD implementation

Tests

1. TestLiteralAggFunction.java - Literal aggregate testing
2. TestCountStar.java - COUNT(*) functionality testing

Migration Notes for Developers

If you use EXTRACT(SECOND):

• Before: Returned BIGINT (lost fractional seconds)
• After: Returns DOUBLE (preserves fractional seconds like 45.123)
• Action: Update code expecting integer values to handle doubles

If you use COUNT(*):

• Before: Could return INT in some contexts
• After: Consistently returns BIGINT
• Action: Update code expecting INT to handle BIGINT

If you use SQRT or math functions:

• Before: Might return DECIMAL/BigDecimal in some contexts
• After: Consistently returns DOUBLE
• Action: Update type expectations in tests/code

If you cast DATE to TIME:

• Before: Allowed (but semantically meaningless)
• After: Properly rejected with validation error
• Action: Remove invalid casts, use proper conversions

---

Compatibility

Backward Compatibility

• Query Results: Mostly compatible, but type changes may affect downstream applications
• API: Fully compatible
• Storage Format: Fully compatible

Breaking Changes

1. EXTRACT(SECOND) return type changed from BIGINT → DOUBLE
2. COUNT(*) return type changed from INT → BIGINT in some contexts
3. Math function return types more strictly DOUBLE (not DECIMAL)
4. Invalid DATE→TIME casts now rejected