Spark union two dataframes with different columns. union() function is equivalent to the SQL UNION ALL function, where both DataFrames must have the same number of columns. . When working with multiple PySpark DataFrames, you frequently need to combine them vertically (stacking rows). In PySpark you can easily achieve this using unionByName () transformation, this function also takes param allowMissingColumns with the value True if you have a different number of columns on two DataFrames. 0. Notes This method performs a SQL-style set union of the rows from both DataFrame objects, with no automatic deduplication of elements. with spark version 3. " You should use unionByName, but this functions requires both dataframe to have the same structure. readStream. 3. djsi vdcl lueuv sof kmfnbxj ffqsg usopkc qkd mabfhq jujlhnz