Append List Of Python Dictionaries To A File Without Loading It

Suppose I need to have a database file consisting of a list of dictionaries: file: [ {'name':'Joe','data':[1,2,3,4,5]}, { ... }, ... ] I n

Solution 1:

You can use json to dump the dicts, one per line. Now each line is a single json dict that you've written. You loose the outer list, but you can add records with a simple append to the existing file.

import json
import os

defappend_record(record):
    withopen('my_file', 'a') as f:
        json.dump(record, f)
        f.write(os.linesep)

# demonstrate a program writing multiple recordsfor i inrange(10):
    my_dict = {'number':i}
    append_record(my_dict)

The list can be assembled later

withopen('my_file') as f:
    my_list = [json.loads(line) for line in f]

The file looks like

{"number":0}{"number":1}{"number":2}{"number":3}{"number":4}{"number":5}{"number":6}{"number":7}{"number":8}{"number":9}

Solution 2:

If you are looking to not actually load the file, going about this with json is not really the right approach. You could use a memory mapped file… and never actually load the file to memory -- a memmap array can open the file and build an array "on-disk" without loading anything into memory.

Create a memory-mapped array of dicts:

>>>import numpy as np>>>a = np.memmap('mydict.dat', dtype=object, mode='w+', shape=(4,))>>>a[0] = {'name':"Joe", 'data':[1,2,3,4]}>>>a[1] = {'name':"Guido", 'data':[1,3,3,5]}>>>a[2] = {'name':"Fernando", 'data':[4,2,6,9]}>>>a[3] = {'name':"Jill", 'data':[9,1,9,0]}>>>a.flush()>>>del a

Now read the array, without loading the file:

>>>a = np.memmap('mydict.dat', dtype=object, mode='r')

The contents of the file are loaded into memory when the list is created, but that's not required -- you can work with the array on-disk without loading it.

>>> a.tolist()
[{'data': [1, 2, 3, 4], 'name': 'Joe'}, {'data': [1, 3, 3, 5], 'name': 'Guido'}, {'data': [4, 2, 6, 9], 'name': 'Fernando'}, {'data': [9, 1, 9, 0], 'name': 'Jill'}]

It takes a negligible amount of time (e.g. nanoseconds) to create a memory-mapped array that can index a file regardless of size (e.g. 100 GB) of the file.

Solution 3:

Using the same approach as user3500511...

Suppose we have two lists of dictionaries (dicts, dicts2). The dicts are converted to json formatted strings. Dicts is saved to a new file - test.json. Test.json is reopened and the string objects are formatted with the proper delimiters. With the reformatted objects, dict2 can be appended and the file still maintains the proper structure for a JSON object.

import json

dicts = [{ "name": "Stephen", "Number": 1 }
         ,{ "name": "Glinda", "Number": 2 }
         ,{ "name": "Elphaba", "Number": 3 }
         ,{ "name": "Nessa", "Number": 4 }]

dicts2= [{ "name": "Dorothy", "Number": 5 }
         ,{ "name": "Fiyero", "Number": 6 }]


f = open("test.json","w")
f.write(json.dumps(dicts))
f.close()

f2 = open("test.json","r+")
f2.seek(-1,2)
f2.write(json.dumps(dicts2).replace('[',',',1))
f2.close()

f3 = open('test.json','r')
f3.read()