Pandas DataFrame concat vs append
pandas, python
Solution
So what are you doing is with append and concat is almost equivalent. The difference is the empty DataFrame. For some reason this causes a big slowdown, not sure exactly why, will have to look at some point. Below is a recreation of basically what you did.
I almost always use concat (though in this case they are equivalent, except for the empty frame); if you don't use the empty frame they will be the same speed.
In [17]: df1 = pd.DataFrame(dict(A = range(10000)),index=pd.date_range('20130101',periods=10000,freq='s'))
In [18]: df1
Out[18]:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 10000 entries, 2013-01-01 00:00:00 to 2013-01-01 02:46:39
Freq: S
Data columns (total 1 columns):
A 10000 non-null values
dtypes: int64(1)
In [19]: df4 = pd.DataFrame()
The concat
In [20]: %timeit pd.concat([df1,df2,df3])
1000 loops, best of 3: 270 us per loop
This is equavalent of your append
In [21]: %timeit pd.concat([df4,df1,df2,df3])
10 loops, best of
3: 56.8 ms per loop
Problem
I have a list of 4 pandas dataframes containing a day of tick data that I want to merge into a single data frame. I cannot understand the behavior of concat on my timestamps. See details below: ``` data [<class 'pandas.core.frame.DataFrame'> DatetimeIndex: 35228 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-03-28 18:59:20.357000+02:00 Data columns: Price 4040 non-null values Volume 4040 non-null values BidQty 35228 non-null values BidPrice 35228 non-null values AskPrice 35228 non-null values AskQty 35228 non-null values dtypes: float64(6), <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 33088 entries, 2013-04-01 00:03:17.047000+02:00 to 2013-04-01 18:59:58.175000+02:00 Data columns: Price 3969 non-null values Volume 3969 non-null values BidQty 33088 non-null values BidPrice 33088 non-null values AskPrice 33088 non-null values AskQty 33088 non-null values dtypes: float64(6), <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 50740 entries, 2013-04-02 00:03:27.470000+02:00 to 2013-04-02 18:59:58.172000+02:00 Data columns: Price 7326 non-null values Volume 7326 non-null values BidQty 50740 non-null values BidPrice 50740 non-null values AskPrice 50740 non-null values AskQty 50740 non-null values dtypes: float64(6), <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 60799 entries, 2013-04-03 00:03:06.994000+02:00 to 2013-04-03 18:59:58.180000+02:00 Data columns: Price 8258 non-null values Volume 8258 non-null values BidQty 60799 non-null values BidPrice 60799 non-null values AskPrice 60799 non-null values AskQty 60799 non-null values dtypes: float64(6)] ``` Using `append` I get: ``` pd.DataFrame().append(data) <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 179855 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-04-03 18:59:58.180000+02:00 Data columns: AskPrice 179855 non-null values AskQty 179855 non-null values BidPrice 179855 non-null values BidQty 179855 non-null values Price 23593 non-null values Volume 23593 non-null values dtypes: float64(6) ``` Using `concat` I get: ``` pd.concat(data) <class 'pandas.core.frame.DataFrame'> DatetimeIndex: 179855 entries, 2013-03-27 22:00:07.089000+02:00 to 2013-04-03 16:59:58.180000+02:00 Data columns: Price 23593 non-null values Volume 23593 non-null values BidQty 179855 non-null values BidPrice 179855 non-null values AskPrice 179855 non-null values AskQty 179855 non-null values dtypes: float64(6) ``` Notice how the index changes when using `concat`. Why is that happening and how would I go about using `concat` to reproduce the results obtained using `append`? (Since `concat` seems so much faster; 24.6 ms per loop vs 3.02 s per loop)