Skip to content

v0.67.0 - LazyFrame redesigned as Expression-only filter optimizer (2026-04-05)

What Changed?

This release redesigns LazyFrame to specialize in Expression-based filtering with built-in predicate optimization. The old LazyFrame API (select, withColumn, rename methods) has been removed in favor of a focused, filter-only implementation. Users should migrate to using BareFrame directly for non-filter transformations, and LazyFrame.from(df).filter(expr).collect() for optimized filtering.

What's New

LazyFrame: Expression-Only Filter Engine

What it does: LazyFrame now accepts only Expression instances for filtering, enabling:

  • Predicate fusion: multiple filters are automatically combined via AND logic
  • Short-circuit evaluation: evaluation stops immediately when a condition fails
  • Foundation for future optimizations: predicate reordering by selectivity, pushdown analysis

How to use it:

TypeScript
const df = BareFrame.fromColumns({
  name: ["Alice", "Bob", "Charlie"],
  age: [25, 17, 30],
});

// Old API (deprecated) - REMOVED
// df.lazy().filter(...).select(...).collect()

// New API
const result = LazyFrame.from(df)
  .filter(new Expression("age").ge(18))
  .filter(new Expression("name").startsWith("A"))
  .collect();
// name  | age
// Alice | 25

Code example:

TypeScript
// Multiple filters fused into a single AND predicate
const lazy = LazyFrame.from(df)
  .filter(new Expression("age").ge(18))
  .filter(new Expression("status").eq("active"));

// Internally optimized:
// combined = age >= 18 AND status = 'active'
const result = lazy.collect();

Added

  • New LazyFrame implementation in src/torch/lazyframe.ts with Expression-only API
  • Static factory method: LazyFrame.from(df) for creating lazy wrappers
  • Predicate fusion engine combining multiple Expression filters via AND logic
  • 22 comprehensive tests covering filter operations, complex conditions, and edge cases
  • Support for complex Expression conditions: AND, OR, NOT operations

Changed

  • LazyFrame now exported from src/torch/lazyframe.ts instead of src/torch/frame.ts
  • BareFrame.lazy() deprecated in favor of LazyFrame.from()
  • LazyFrame.filter() now accepts Expression instances instead of RowPredicate functions
  • Old transformation methods removed: select(), withColumn(), rename(), collect() (use BareFrame directly)

Fixed

  • Removed circular dependency issue by separating LazyFrame into independent module

Is It Safe to Upgrade?

  • Breaking Changes: Yes (LazyFrame API completely redesigned)
  • Backward Compatible: No

Migration Guide:

TypeScript
// OLD: Using df.lazy() with multiple transformations
const result = df.lazy()
  .filter(row => row.get("age") > 18)
  .select(["name", "age"])
  .collect();

// NEW: Use BareFrame directly for transformations
const result = df
  .filter(row => row.get("age") > 18)
  .select(["name", "age"]);

// NEW: Use LazyFrame.from() only for optimized filtering
const result = LazyFrame.from(df)
  .filter(new Expression("age").gt(18))
  .collect();

Release Details

  • Date: 2026-04-05
  • Version: v0.67.0
  • gaslamp: 126
  • pilotlamp: 84
  • Files Changed: 5 (src/torch/lazyframe.ts, src/torch/frame.ts, src/torch/index.ts, __tests__/v2/frame.test.ts, __tests__/v2/lazyframe.test.ts)
  • Commits:
    • e9197270: feat: add new LazyFrame implementation (Expression-only, filter-focused)
    • 8d6d2b2d: feat: replace old LazyFrame with Expression-only implementation
    • 7a7b1b8c: refactor: remove deprecated LazyFrame implementation from frame.ts (Closes #159)
    • 36451a05: test: remove deprecated LazyFrame test cases
    • 9382a277: test: remove DataFrame.lazy test suite
    • 15683a03: test: add comprehensive tests for new LazyFrame implementation

Known Issues

None at this time.

Next Steps

  • Future versions will implement predicate reordering optimization based on selectivity analysis
  • Pushdown optimization for multi-step filter chains
  • Integration with GAS-specific performance profiling tools