I have a list of files that I pass into a for loop and do a whole bunch of functions. Whats the easiest way to parallelize this? Not sure I could find this exact thing anywhere and I think my current implementation is incorrect because I only saw one file being run. From some reading I've done, I think this should be a perfectly parallel case.
Old code is something like this:
import pandas as pd filenames = ['file1.csv', 'file2.csv', 'file3.csv', 'file4.csv'] for file in filenames: file1 = pd.read_csv(file) print('running ' + str(file)) a = function1(file1) b = function2(a) c = function3(b) for d in range(1,6): e = function4(c, d) c.to_csv('output.csv') (incorrectly) Parallelized code
import pandas as pd from multiprocessing import Pool filenames = ['file1.csv', 'file2.csv', 'file3.csv', 'file4.csv'] def multip(filenames): file1 = pd.read_csv(file) print('running ' + str(file)) a = function1(file1) b = function2(a) c = function3(b) for d in range(1,6): e = function4(c, d) c.to_csv('output.csv') if __name__ == '__main__' pool = Pool(processes=4) runstuff = pool.map(multip(filenames)) What I (think) I want to do is have one file be computed per core (maybe per process?). I also did
multiprocessing.cpu_count() and got 8 (I have a quad so its probably taking into account threads). Since I have around 10 files total, if I can put one file per process to speed things up that would be great! I would hope the remaining 2 files would find a process after the processes from the first round complete as well.
Edit: for further clarity, the functions (i.e. function1, function2 etc) also feed into other functions (i.e function1a, function1b) inside their respective files. I call function 1 using an import statement.
I get the following error:
OSError: Expected file path name or file-like object, got <class 'list'> type Apparently doesn't like being passed a list but i don't want to do filenames[0] in the if statement because that only runs one file
1 Answer
import multiprocessing names = ['file1.csv', 'file2.csv'] def multip(name): [do stuff here] if __name__ == '__main__': #use one less process to be a little more stable p = multiprocessing.Pool(processes = multiprocessing.cpu_count()-1) #timing it... start = time.time() for file in names: p.apply_async(multip, [file]) p.close() p.join() print("Complete") end = time.time() print('total time (s)= ' + str(end-start)) EDIT: Swap out the if__name__== '____main___' for this one. This runs all the files:
if __name__ == '__main__': p = Pool(processes = len(names)) start = time.time() async_result = p.map_async(multip, names) p.close() p.join() print("Complete") end = time.time() print('total time (s)= ' + str(end-start)) 
