Pandas Dataframe Concat Is Giving Unwanted Na/nan Columns

Instead of this example where it is horizontal After Pandas Dataframe pd.concat I get NaNs, I'm trying vertical: import pandas a=[['Date', 'letters', 'numbers', 'mixed'], ['1/2/201

Solution 1:

I would create a dataframe df_a with the correct columns directly.

With a little refactoring of your code, it gives

import pandas
a=[['Date', 'letters', 'numbers', 'mixed'], \
   ['1/2/2014', 'a', '6', 'z1'],\
   ['1/2/2014', 'a', '3', 'z1'],\
   ['1/3/2014', 'c', '1', 'x3']]
df = pandas.DataFrame.from_records(a[1:],columns=a[0])
df['new'] = df['Date'] + ' ' + df['letters']

n = len(df.columns)
b = [['NA'] * n]
df_a = pandas.DataFrame.from_records(b,columns=df.columns)
df_b = pandas.concat([df,df_a])

It gives

       Date letters numbers mixed         new
01/2/2014       a       6    z1  1/2/2014 a
11/2/2014       a       3    z1  1/2/2014 a
21/3/2014c1    x3  1/3/2014c0NANANANANA

Eventually:

df_b = pandas.concat([df,df_a]).reset_index(drop=True)

It gives

       Date letters numbers mixed         new
01/2/2014       a       6    z1  1/2/2014 a
11/2/2014       a       3    z1  1/2/2014 a
21/3/2014c1    x3  1/3/2014c3NANANANANA

Solution 2:

If you are using latest versions, this gives you what you want

df.ix[len(df), :]='NA'

EDIT: OR if you want concat, when you define df_a, use columns of df as columns

df_a = pandas.DataFrame.from_records(b,columns=df.columns)