How to apply foldleft and modify a column in a nested struct in spark scala?

I have a dataframe with below schema.

    root
 |-- table: string (nullable = true)
 |-- before: struct (nullable = true)
 |    |-- AP_ID: string (nullable = true)
 |    |-- AP_VERSION_ID: string (nullable = true)
 |-- after: struct (nullable = true)
 |    |-- AP_ID: string (nullable = true)
 |    |-- AP_VERSION_ID: string (nullable = true)
 |-- _kafka_key: string (nullable = true)

Need to modify the a column inside after column but below code is not working.

    val newDF = df.columns.foldLeft(df)    {
  case (tmpDF, colName)=>
    if (colName=="after") {
      val iniCol=colName.concat(".*")
      val afterDf = df.select (iniCol)
      val aftdf=afterDf.columns.foldLeft(afterDf)
      {
        case (tmpDF1, colName1)=>
          
            val colNamecat=colName.concat(".")
            val fnlCol=colNamecat.concat(colName1)
            tmpDF.withColumn(fnlCol, enUDF(col(fnlCol), lit(cryptoColumnsMap(colName1.toLowerCase))))
          }
          else tmpDF
      }
      tmpDF
    }
    else tmpDF
}


Read more here: https://stackoverflow.com/questions/68475786/how-to-apply-foldleft-and-modify-a-column-in-a-nested-struct-in-spark-scala

Content Attribution

This content was originally published by Eva Mariam at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: