how to enable memory mapping while reading feather file in c++

While reading the same feather file in python and in c++,in python, the function pyarrow.feather.read_table() performs expectionally well then the API i am using for c++. When i investigated further, i found out that the main difference is because in python the read_table() API uses a flag named, memory_map(set true by default). When i disable this flag, c++ APIs perform just better than read_table() in python. Now as it suggests,c++ is not using memory_mapping by default, but to improve the performance i want to use it. Please suggest me ways to use memory mapping for c++ i as didn't find any APIs of interest in the documentation available for c++. The code i am using is -

#!/usr/bin/env python3
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.feather
import time

for i in range(20):
    start_time = time.time()

    table = pyarrow.feather.read_table('data'+str(i + 1)+'.feather')
    end_time = time.time()
    print("Time taken to read file is : ",(end_time - start_time)*1000,"ms")

C++ code is -

void read_feather_to_table(std::string path,std::shared_ptr<arrow::Table> *feather_table){
    std::shared_ptr <arrow::io::RandomAccessFile> input_file = file_system.OpenInputFile(path).ValueOrDie();
    std::shared_ptr <arrow::ipc::feather::Reader> feather_reader = arrow::ipc::feather::Reader::Open(input_file).ValueOrDie();
    auto t1 = std::chrono::high_resolution_clock::now();
    arrow::Status temp_status = feather_reader -> Read(feather_table);
    auto t2 = std::chrono::high_resolution_clock::now();
    auto ms_int = std::chrono::duration_cast<std::chrono::milliseconds>(t2 - t1);
    std::cout << "Time taken to read file is : "<< ms_int.count()<< "ms\n";
    return;
}


Read more here: https://stackoverflow.com/questions/67932844/how-to-enable-memory-mapping-while-reading-feather-file-in-c

Content Attribution

This content was originally published by Sourabh Kulhari at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: