How to convert all the contents of a list to individual strings to pass as a parameter in dataframe.select()?

My requirement is to pass the list of all column names as a parameter in dataframe.select(columns) -


To achieve the aforementioned:

  1. First of all I am using dataframe.columns to get me the list of all the columns. Which I am storing as list1=(dataframe.columns)
  2. Secondly I am using .join to convert my list of columns in strings like - (strList1 = "','".join(list1)) which gives me the following output

col1','col2','col3','col4

As you see there is some missing single quotes at the start & end of the string

  1. To rectify strList1 i use f-string formatting as follows - strList2 = f"{strList1}'"which gives me the following output

'col1','col2','col3','col4'


Coming to the main problem:

When you pass strList2 as a parameter in dataframe.select(strList2) gives me the following error

Py4JJavaError: An error occurred while calling o5813.select. : org.apache.spark.sql.AnalysisException: cannot resolve 'backquotecol1','col2','col3','col4'backquote' given input columns: [ col1, col2, db.table.col4, db.table.col3];; 'Project ['col1','col2','col3','col4']

Note - There is a backquote before col1 and after col4

Kindly help as i don't understand why the select function isn't taking a proper string value / variable (strList2)



Read more here: https://stackoverflow.com/questions/64894644/how-to-convert-all-the-contents-of-a-list-to-individual-strings-to-pass-as-a-par

Content Attribution

This content was originally published by Mohseen Mulla at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: