How To Extract Non Na Values In A List Or Dict From A Pandas Dataframe

I have a df like this, df, AAA BBB CCC 0 4 10 100 1 5 20 50 2 6 30 -30 3 7 40 -50 df_mask = pd.DataFrame({'AAA' : [True] * 4, 'BBB' : [False] * 4,'CCC' : [

Solution 1:

Let's use agg here:

v = df.where(df_mask).agg(lambda x: x.dropna().to_dict())

On older versions, apply does the same thing (albeit a bit slower).

v = df.where(df_mask).apply(lambda x: x.dropna().to_dict())

And now, filter out rows with empty dictionaries for the final step:

res = v[v.str.len() > 0].to_dict()

print(res)
{'AAA': {0:4.0, 1:5.0, 2:6.0, 3:7.0}, 'CCC': {0:100.0, 2:-30.0}}

---

Another apply-free option is a dict-comprehension:

v=df.where(df_mask)res= {k :v[k].dropna().to_dict()forkindf} 

print(res)
{'AAA': {0:4, 1:5, 2:6, 3:7}, 'BBB': {}, 'CCC': {0:100.0, 2:-30.0}}

Note that this (slightly) simpler solution retains keys with empty values.

Solution 2:

You can iterate df's columns and apply dropnaSerieswise

{col: df[col].dropna().values for col indf}

Which yields

{'AAA': array([4, 5, 6, 7]),
 'BBB': array([], dtype=float64),
 'CCC': array([ 100.,  -30.])}

You can filter out empty arrays such as 'BBB' with

{key: valfor key, valin ddict.items() ifval}