Syncronizing data between two django servers

Question

I have a central Django server containing all of my information in a database. I want to have a second Django server that contains a subset of that information in a second database. I need a bulletproof way to selectively sync data between the two. - The secondary Django will need to pull its subset of data from the primary at certain times. The subset will have to be filtered by certain fields. - The secondary Django will have to occasionally push its data to the primary. - Ideally, the two-way sync would keep the most recently modified objects for each model. I was thinking something along the lines of having using TimeStampedModel (from django-extensions) or adding my own DateTimeField(auto_now=True) so that every object stores its last modified time. Then, maybe a mechanism to dump the data from one DB and load it in to the other such that only the more recently modified objects are kept. Possibilities I am considering are django's dumpdata, django-extensions dumpscript, django-te…

Accepted Answer

Here is my solution, which fits all of my requirements:

- Implement natural keys and unique constraints on all models

- Allows for a unique way to refer to each object without using primary key IDs

- Sublcass each model from TimeStampedModel in django-extensions

- Adds automatically updated `created` and `modified` fields

Create a Django management command for exporting, which filters a subset of data and serializes it with natural keys

```
baz = Baz.objects.filter(foo=bar)
yaz = Yaz.objects.filter(foo=bar)

objects = [baz, yaz]
flat_objects = list(itertools.chain.from_iterable(objects))

data = serializers.serialize("json", flat_objects, indent=3, use_natural_keys=True)
print(data)
```

Create a Django management command for importing, which reads in the serialized file and iterates through the objects as follows:

- If the object does not exist in the database (by natural key), create it

- If the object exists, check the `modified` timestamps

- If the imported object is newer, update the fields

- If the imported object is older, do not update (but print a warning)

Code sample:

```
# Open the file
with open(args[0]) as data_file:
    json_str = data_file.read()

# Deserialize and iterate
for obj in serializers.deserialize("json", json_str, indent=3, use_natural_keys=True):

# Get model info
    model_class = obj.object.__class__
    natural_key = obj.object.natural_key()
    manager = model_class._default_manager

# Delete PK value
    obj.object.pk = None

try:
        # Get the existing object
        existing_obj = model_class.objects.get_by_natural_key(*natural_key)

# Check the timestamps
        date_existing = existing_obj.modified
        date_imported = obj.object.modified
        if date_imported > date_existing:

# Update fields
            for field in obj.object._meta.fields:
                if field.editable and not field.primary_key:
                    imported_val = getattr(obj.object, field.name)
                    existing_val = getattr(existing_obj, field.name)
                    if existing_val != imported_val:
                        setattr(existing_obj, field.name, imported_val)

except ObjectDoesNotExist:
        obj.save()
```

The workflow for this is to first call `python manage.py exportTool > data.json`, then on another django instance (or the same), call `python manage.py importTool data.json`.

Syncronizing data between two django servers

Solution

Problem