bigdata

Create big text file for testing

I’m performing python ETL on a file.
But I only have a small template. The real file will have more than 20gb.
How to reproduce this small file into a big one? can only repeat lines.

0|18033552000161|032021|PAR_200|21659151780|0|0|C-200|0…

Imagine you had a large CSV file – let’s say 1 billion rows and 100 GB in size.
You want each row in the file to become a document in elastic search.
You can’t load the file into memory – it has to be streamed or chunked.
The time taken is…