How to convert all the contents of a list to individual strings to pass as a parameter in

My requirement is to pass the list of all column names as a parameter in -

To achieve the aforementioned:

  1. First of all I am using dataframe.columns to get me the list of all the columns. Which I am storing as list1=(dataframe.columns)
  2. Secondly I am using .join to convert my list of columns in strings like - (strList1 = "','".join(list1)) which gives me the following output


As you see there is some missing single quotes at the start & end of the string

  1. To rectify strList1 i use f-string formatting as follows - strList2 = f"{strList1}'"which gives me the following output


Coming to the main problem:

When you pass strList2 as a parameter in gives me the following error

Py4JJavaError: An error occurred while calling : org.apache.spark.sql.AnalysisException: cannot resolve 'backquotecol1','col2','col3','col4'backquote' given input columns: [ col1, col2, db.table.col4, db.table.col3];; 'Project ['col1','col2','col3','col4']

Note - There is a backquote before col1 and after col4

Kindly help as i don't understand why the select function isn't taking a proper string value / variable (strList2)

Read more here:

Content Attribution

This content was originally published by Mohseen Mulla at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: