Remove columns with same values from a large csv file

I have a large csv file with thousands of columns. I want to;

  1. Obtain the list of all columns with same values
  2. Remove all column obtained in step 1

Sample input

Sample Input

Sample output

Columns with same values: F1 F2

Sample Output

I have implemented python based solutions which works fine for small files but are too slow for large files (more than 8 GB).

A solution in any programming language (but robust and fast) will be appreciated.



Read more here: https://stackoverflow.com/questions/64945968/remove-columns-with-same-values-from-a-large-csv-file

Content Attribution

This content was originally published by Ahmad at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: