What is the most efficient way to loop through dataframes with pandas?
dataframe, for-loop, pandas, performance, python
Solution
The newest versions of pandas now include a built-in function for iterating over rows.
for index, row in df.iterrows():
# do some logic here
Or, if you want it faster use `itertuples()`
But, unutbu's suggestion to use numpy functions to avoid iterating over rows will produce the fastest code.
Problem
I want to perform my own complex operations on financial data in dataframes in a sequential manner. For example I am using the following MSFT CSV file taken from Yahoo Finance: ``` Date,Open,High,Low,Close,Volume,Adj Close 2011-10-19,27.37,27.47,27.01,27.13,42880000,27.13 2011-10-18,26.94,27.40,26.80,27.31,52487900,27.31 2011-10-17,27.11,27.42,26.85,26.98,39433400,26.98 2011-10-14,27.31,27.50,27.02,27.27,50947700,27.27 .... ``` I then do the following: ``` #!/usr/bin/env python from pandas import * df = read_csv('table.csv') for i, row in enumerate(df.values): date = df.index[i] open, high, low, close, adjclose = row #now perform analysis on open/close based on date, etc.. ``` Is that the most efficient way? Given the focus on speed in pandas, I would assume there must be some special function to iterate through the values in a manner that one also retrieves the index (possibly through a generator to be memory efficient)? `df.iteritems` unfortunately only iterates column by column.