How to open multiple csv files from a folder in python?

I want to open multiple csv files in python, collate them and have python create a new file with the data from the multiple files reorganised...

Is there a way for me to read all the files from a single directory on my desktop and read them in python like this?

Thanks a lot

2 Answers

If you a have a directory containing your csv files, and they all have the extension .csv, then you could use, for example, glob and pandas to read them all in and concatenate them into one csv file. For example, say you have a directory, like this:

csvfiles/one.csv csvfiles/two.csv 

where one.csv contains:

name,age Keith,23 Jane,25 

and two.csv contains:

name,age Kylie,35 Jake,42 

Then you could do the following in Python (you will need to install pandas with, e.g., pip install pandas):

import glob import os import pandas as pd # the path to your csv file directory mycsvdir = 'csvdir' # get all the csv files in that directory (assuming they have the extension .csv) csvfiles = glob.glob(os.path.join(mycsvdir, '*.csv')) # loop through the files and read them in with pandas dataframes = [] # a list to hold all the individual pandas DataFrames for csvfile in csvfiles: df = pd.read_csv(csvfile) dataframes.append(df) # concatenate them all together result = pd.concat(dataframes, ignore_index=True) # print out to a new csv file result.to_csv('all.csv') 

Note that the output csv file will have an additional column at the front containing the index of the row. To avoid this you could instead use:

result.to_csv('all.csv', index=False) 

You can see the documentation for the to_csv() method here.

Hope that helps.

Here is a very simple way to do what you want to do.

import pandas as pd import glob, os os.chdir("C:\\your_path\\") results = pd.DataFrame([]) for counter, file in enumerate(glob.glob("1*")): namedf = pd.read_csv(file, skiprows=0, usecols=[1,2,3]) results = results.append(namedf) results.to_csv('C:\\your_path\\combinedfile.csv') 

Notice this part: glob("1*")

This will look only for files that start with '1' in the name (1, 10, 100, etc). If you want everything, change it to this: glob("*")

Sometimes it's necessary to merge all CSV files into a single CSV file, and sometimes you just want to merge some files that match a certain naming convention. It's nice to have this feature!