1.44.0
Python Polars 1.44.0
Added 12
- Support integer fixed-array dot products
- SQL correlated in
- Support more join_types in join_where
- Support Iceberg schema evolution
- Support Iceberg table with V3 deletion vectors in native scan_iceberg
- Support Iceberg snapshot properties
- Expose sinked_paths_callback on sink_parquet as unstable parameter
- Add infer_schema_files to CSV inference hint
- Add native fixed-array dot
- Support Iceberg object storage paths
- Add adaptive HTTP rate-limiter for cloud IO
- Add struct.drop()
Changed 2
- Relax default CloudRetryConfig for rate limit stability
- Improve performance of when/then/otherwise by masking out unevaluated elements
Fixed 3
- Fix assertion error or incorrect data load from parquet written by fastparquet
- Fix OOC spilling of categoricals
- Raise on overflow in Decimal sum instead of silently wrapping
Deprecated 3
- Deprecate rechunk parameter for all read/scan functions
- Deprecate Expr.rechunk()
- Deprecate struct.rename_fields() with an incorrect number of fields
From Polars
⚠️ Deprecations
- Deprecate
rechunkparameter for all read/scan functions (#28063) - Deprecate
Expr.rechunk()(#28692) - Deprecate
struct.rename_fields()with an incorrect number of fields (#28672)
🚀 Performance improvements
- Add private env var toggle for HTTP rate limit (#28882)
- Relax default
CloudRetryConfigfor rate limit stability (#28885) - Elide generic cross join on subquery decorrelation in equality predicates (#28876)
- cache CTE's in SQL layer (#28864)
- Allow CSPE if manual caches are set (#28859)
- Lower unqualified join predicates to inner joins in SQL (#28854)
- Rechunk
maskfor DataFramefilter(#28762) - Improve performance of when/then/otherwise by masking out unevaluated elements (#28498)
- Drop unused projections in filter on streaming engine (#28713)
- Retain Parquet
Partialmetadata on filter (#28737)
✨ Enhancements
- Support integer fixed-array dot products (#28829)
- SQL correlated in (#28927)
- Support more join_types in
join_where(#28880) - Support Iceberg schema evolution (#28794)
- SQL conformance (#28494)
- Introduce
RemoteEngineand a common base class for all engines (#28800) - Support Iceberg table with V3 deletion vectors in native
scan_iceberg(#28772) - Support Iceberg snapshot properties (#28793)
- Expose
sinked_paths_callbackonsink_parquetas unstable parameter (#28814) - Add
infer_schema_filesto CSV inference hint (#28809) - Add native fixed-array dot (#28504)
- Export query metrics to polars_cloud (#28757)
- Support Iceberg object storage paths (#28634)
- More CSEE simplification (#28731)
- Improve error in Series constructor with name in context (#28743)
- Add adaptive HTTP rate-limiter for cloud IO (#28591)
- Add
struct.drop()(#28666) - Improve error message when CSV name de-duplication fails (#28658)
- Always keep first metadata per source for Parquet (#28661)
- Improve plan-time row estimates for multi-file parquet scans (#28380)
- Expose array and plugin function views in Python visitor (#28635)
🐞 Bug fixes
- SQL handle quantified comparisons against a subquery (#28929)
- Fix SQL subquery qualified lowering and cache row-index (#28926)
- Clippy-stable failures (#28919)
- Use uuidv7 for cloud observed queries (#28910)
- Unsoundness in AnyValue::into_static for structs (#28889)
- Invalidate suborders on projection (#28866)
- Ensure SQL function registry is available in isolated subqueries (#28881)
- Handle null in Expr.is_not_nan (#28883)
- Ensure eager
unpivotcolumn selection aligns with the lazy engine (#28846) - Fix assertion error / incorrect data load from parquet written by fastparquet (#28656)
- Fix projection pushdown incorrect
select()height (#28751) - Restore
monitoringas an engine-level parameter (#28821) - Bincode serialization of IR hints (#28837)
- Retain datetime inference across CSV fallback (#28663)
- Avoid chunk fragmentation on bool agg
minandmax(#28789) - Merge join with coalesce and empty suffix (#28799)
- Wrong function name call in test (#28801)
- Preserve row mapping in nested over expressions (#28767)
- Restore CI compatibility with NumPy 2.5.2 (#28782)
- Do not deduplicate inherently nondeterministic plan nodes (#28768)
- Fix OOC spilling of categoricals (#28761)
- Correct partitionable filter handling for pivot (#28749)
- Small typo in docstring (#28763)
- Avoid chunk fragmentation on bool agg
anyandall(#28754) - Ignore nans in (min|max)_horizontal (#28710)
- Correct function display for str.replace_all (#28742)
- Exclude debug tests from nodebug streaming run (#28747)
- Convert predicate series dtype during prefiltered parquet reading (#28711)
- Do not remove nested caches when performing predicate pushdown past a cache node (#28740)
- Correct handling of nulls at the start of business day functions (#28703)
- Do not panic when exceeding Parquet metadata scan cache (#28721)
- Remove cryptographic hashing from CSPE (#28716)
- Unsoundness in rayon block_on (#28709)
- Check for (shallow) IR node equality on top of hashing in CSPE (#28506)
- Don't hand IO plugins a predicate they cannot evaluate (#28643)
- Raise on overflow in Decimal sum instead of silently wrapping (#28688)
- Preserve nulls when importing Arrow maps (#28680)
- Propagate null
bycolumn values inrolling_*_by(#27367) - Fix self-referencing
fieldinstruct.with_fieldswithover(#28678) - Fix Arrow buffer offset for
Utf8andBinary(#28662) - Clamp group-by slice offset (#28579)
- Release the GIL while resolving the schema in
__arrow_c_stream__(#28642) - Fix data corruption on Series from nested Arrow
LargeList(#28632)
📖 Documentation
- Interlink join docstrings (#28907)
- Migration guide for 2.0 release (#28788)
- Clarify
read_database_urirequirements for SQLAlchemy (#28366) - Update engine tags (#28852)
- Clarify "that that side" comment in asof_join (#28847)
- Update mage.ai documentation link (#28813)
- Add engine tags (#28708)
- Document that arg_extremum functions return any index on ties (#28691)
- Fix Polars Cloud API reference link (#28693)
🛠️ Other improvements
- Bump runtime toolchains (#28924)
- Add min-publish-age for cargo (#28906)
- Tighten join invariants DSL (#28913)
- Bump Rust toolchain (#28900)
- Ensure that predicates are pushed down to the scans (#28857)
- Clarify "that that side" comment in asof_join (#28847)
- Restore
monitoringas an engine-level parameter (#28821) - Remove workaround for inconsistent CSE (#28834)
- Introduce
RemoteEngineand a common base class for all engines (#28800) - Add row count and file byte size to sinked paths callback (#28522)
- Use canonical expression IDs in CSEE (#28786)
- Enforce 100% type coverage in CI with Pyrefly (#28795)
- Attribute annotations for
CredentialProviderAWSandCredentialProviderAzure(#28790) - Attribute annotations for
ScanCastOptions(#27906) - Disable credentials in upload chunk size tests (#28745)
- Improve backtrace path formatting (#28732)
- Exclude debug tests from nodebug streaming run (#28747)
- Mark
test_grouped_agg_parametricas slow (and add param ids) (#28715) - Fix first
expectintest_group_by_arg_max_boolean_26978(#28690) - Fix the name of
struct.drop()inrename_fields()deprecation message (#28687) - Add
dist=loadgroupto the pytest config (#28676) - Enable a lot of
may_fail_auto_streamingtests (#28675) - Re-enable
test_extension()for streaming engine (#28611) - Update
test_hive_join_rewrite_semi_jointest to work with streaming engine (#28610)
Thank you to all our contributors for making this release possible! @0guban0v, @JakubValtar, @Joosboy, @Kevin-Patyk, @MatteoPossamai, @NicoOhR, @TNieuwdorp, @VedantMadane, @aarushkandukoori, @alexander-beedie, @azimafroozeh, @borchero, @carnarez, @dancsi, @dependabot[bot], @dsprenkels, @gautamvarmadatla, @jonasdedden, @jorenham, @kdn36, @lzcmian, @malhotrashivam, @matthewbayer, @mikhail5555, @nameexhaustion, @orlp, @ritchie46, @subotac and dependabot[bot]