Can we set remove column names from s3 partition path and set path to values?

I am just curious, for Spark using Glue sinkFormat, is it possible to save the file as "2021/05/05/filename.parquet" and not as "year=2021/month=05/day=05/filename.parquet". I tried to play with 'writepath' but it works at record level and I believe it will break Spark's ability to save partitioned files.



Read more here: https://stackoverflow.com/questions/67395809/can-we-set-remove-column-names-from-s3-partition-path-and-set-path-to-values

Content Attribution

This content was originally published by Charmee Lee at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: