Reading CSV files

Arrow provides preliminary support for reading data from CSV files. The features currently offered are the following:

  • multi-threaded or single-threaded reading
  • automatic decompression of input files (based on the filename extension, such as my_data.csv.gz)
  • fetching column names from the first row in the CSV file
  • column-wise type inference and conversion to one of null, int64, float64, timestamp[s], string or binary data
  • detecting various spellings of null values such as NaN or #N/A

Usage

CSV reading functionality is available through the pyarrow.csv module. In many cases, you will simply call the read_csv() function with the file path you want to read from:

>>> from pyarrow import csv
>>> fn = 'tips.csv.gz'
>>> table = csv.read_csv(fn)
>>> table
pyarrow.Table
total_bill: double
tip: double
sex: string
smoker: string
day: string
time: string
size: int64
>>> len(table)
244
>>> df = table.to_pandas()
>>> df.head()
   total_bill   tip     sex smoker  day    time  size
0       16.99  1.01  Female     No  Sun  Dinner     2
1       10.34  1.66    Male     No  Sun  Dinner     3
2       21.01  3.50    Male     No  Sun  Dinner     3
3       23.68  3.31    Male     No  Sun  Dinner     2
4       24.59  3.61  Female     No  Sun  Dinner     4

Customized parsing

To alter the default parsing settings in case of reading CSV files with an unusual structure, you should create a ParseOptions instance and pass it to read_csv().

Customized conversion

To alter how CSV data is converted to Arrow types and data, you should create a ConvertOptions instance and pass it to read_csv().

Performance

Due to the structure of CSV files, one cannot expect the same levels of performance as when reading dedicated binary formats like Parquet. Nevertheless, Arrow strives to reduce the overhead of reading CSV files.

Performance options can be controlled through the ReadOptions class. Multi-threaded reading is the default for highest performance, distributing the workload efficiently over all available cores.

Note

The number of threads to use concurrently is automatically inferred by Arrow and can be inspected using the cpu_count() function.