26 Pull requests opened by 22 people

• [SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite

  #50989 opened May 22, 2025

• [SPARK-52281][SQL] Change `ALTER TABLE ALTER COLUMN TYPE STRING` not to apply default collation if original data type was instance of `StringType`

  #51001 opened May 23, 2025

• [SPARK-51987][SQL] DSv2 expressions in column defaults on write

  #51002 opened May 23, 2025

• [SPARK-52283][SQL] Declarative Pipelines `DataflowGraph` creation and resolution

  #51003 opened May 23, 2025

• [SPARK-52171] [SS] StateDataSource join implementation for state v3

  #51004 opened May 24, 2025

• [SPARK-52288][PS] Avoid INVALID_ARRAY_INDEX in `split`/`rsplit` when ANSI mode is on

  #51006 opened May 24, 2025

• Increase report interval of spaming logs to 10 seconds

  #51012 opened May 25, 2025

• [SPARK-48660][SQL] Fix explain result for CreateTableAsSelect

  #51013 opened May 25, 2025

• [SPARK-52306][SQL] Add support of ALTER TABLE ... UNSET SERDEPROPERTIES

  #51017 opened May 26, 2025

• [SPARK-52307][PYTHON][CONNECT] Support Scalar Arrow Iterator UDF

  #51018 opened May 26, 2025

• [SPARK-52308][CORE] Fix the bug that the actual configs of the driver are overwritten

  #51019 opened May 26, 2025

• [SPARK-52311][SQL] Fix bug with multiple self-references with UnresolvedSubqueries in rCTEs

  #51022 opened May 26, 2025

• [SPARK-44223][CORE] Remove LevelDB support

  #51027 opened May 27, 2025

• [SPARK-52088][CORE] Refactor the `ClosureCleaner#clean` and avoid using the `-Djdk.reflect.useDirectMethodHandle=false`

  #51028 opened May 27, 2025

• [SPARK-52326][SQL] Add partitions related ExternalCatalogEvent and post them in corresponding operations

  #51030 opened May 27, 2025

• [SPARK-52312][SQL] Ignore V2WriteCommand when caching DataFrame

  #51032 opened May 27, 2025

• [SPARK-48353][FOLLOW UP][SQL] Exception handling improvements

  #51034 opened May 27, 2025

• [SPARK-52334][CORE][K8S] update all files, jars, and pyFiles to reference the working directory after they are downloaded

  #51037 opened May 28, 2025

• [SPARK-52335][CONNET][SQL] Unify the 'invalid bucket count' error for both Connect and Classic

  #51039 opened May 28, 2025

• [SPARK-52311][SQL] Redefine UnionLoop output to not be duplicated if the anchor output is duplicated

  #51041 opened May 28, 2025

• Added s3 offloading technique for RDD

  #51042 opened May 28, 2025

• [SPARK-52339][SQL] Fix comparison of `InMemoryFileIndex` instances

  #51043 opened May 28, 2025

• Move `DefineSqlGraphElements` to `PipelineCommand`

  #51044 opened May 28, 2025

• [SPARK-52327][Core] Glob based provider for history server

  #51045 opened May 29, 2025

• [SPARK-51831][SQL] Column pruning with existsJoin for Datasource V2

  #51046 opened May 29, 2025

• [SPARK-52345][SQL] Fix NULL behavior in scripting conditions

  #51047 opened May 29, 2025

22 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-52187][SQL] Introduce Join pushdown for DSv2

  #50921 commented on May 28, 2025 • 14 new comments

• [SPARK-52235][SQL] Add implicit cast to DefaultValue V2 Expressions passed to DSV2

  #50959 commented on May 29, 2025 • 7 new comments

• [SPARK-52249][PS] Enable divide-by-zero for truediv with ANSI enabled

  #50972 commented on May 29, 2025 • 6 new comments

• [SPARK-52232][SQL] Fix non-deterministic queries to produce different results at every step

  #50957 commented on May 29, 2025 • 6 new comments

• [SPARK-52065][SQL] Produce another plan tree with output columns (name, data type, nullability) in plan change logging

  #50852 commented on May 25, 2025 • 4 new comments

• [SPARK-52226] [SQL] Fix unusual equality checks in three operators

  #50949 commented on May 22, 2025 • 2 new comments

• [SPARK-52218][SQL] Make current datetime functions evaluable again

  #50936 commented on May 28, 2025 • 2 new comments

• [SPARK-52238][PYTHON] Python client for Declarative Pipelines

  #50963 commented on May 28, 2025 • 1 new comment

• [SPARK-52032][SQL] Fix orc filter pushdown with null-safe equality operator

  #50932 commented on May 22, 2025 • 1 new comment

• [SPARK-52082][PYTHON][DOCS] Improve ExtractPythonUDF docs

  #50867 commented on May 28, 2025 • 1 new comment

• Enable -Xsource:3 compiler flag

  #50474 commented on May 29, 2025 • 1 new comment

• [SPARK-51069][SQL] Add big-endian support to UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl

  #49773 commented on May 27, 2025 • 1 new comment

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 commented on May 29, 2025 • 0 new comments

• [SPARK-51919][PYTHON] Allow overwriting statically registered Python Data Source

  #50716 commented on May 28, 2025 • 0 new comments

• Test rockdbsjni 10

  #50688 commented on May 26, 2025 • 0 new comments

• [SPARK-52212][PYTHON][INFRA] Upgrade linter image to python 3.11

  #50931 commented on May 28, 2025 • 0 new comments

• [SPARK-51847][PYTHON] Extend PySpark testing framework util functions with basic data tests

  #50644 commented on May 22, 2025 • 0 new comments

• [SPARK-47547] BloomFilter fpp degradation

  #50933 commented on May 27, 2025 • 0 new comments

• [SPARK-51534][Core] Improve parquet read performance by reducing unnecessary operations when converting parquet to row

  #50297 commented on May 26, 2025 • 0 new comments

• [SPARK-50903][CONNECT] Cache logical plans after analysis

  #49584 commented on May 27, 2025 • 0 new comments

• [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`

  #49005 commented on May 23, 2025 • 0 new comments

• [SPARK-49386][SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill

  #47856 commented on May 26, 2025 • 0 new comments