Remove duplicates in a Django query

django, sql

Solution

This query will not give you duplicates - ie, it will give you all the rows in the database, ordered by email.

However, I presume what you mean is that you have duplicate data within your database. Adding `distinct()` here won't help, because even if you have only one field, you also have an automatic `id` field - so the combination of id+email is not unique.

Assuming you only need one field, `email_address`, de-duplicated, you can do this:

email_list = Email.objects.values_list('email', flat=True).distinct()

However, you should really fix the root problem, and remove the duplicate data from your database.

Example, deleting duplicate Emails by email field:

for email in Email.objects.values_list('email', flat=True).distinct():
    Email.objects.filter(pk__in=Email.objects.filter(email=email).values_list('id', flat=True)[1:]).delete()

Or books by name:

for name in Book.objects.values_list('name', flat=True).distinct(): 
    Book.objects.filter(pk__in=Artwork.objects.filter(name=name).values_list('id', flat=True)[3:]).delete()

Problem

Is there a simple way to remove duplicates in the following basic query: ``` email_list = Emails.objects.order_by('email') ``` I tried using `duplicate()` but it was not working. What is the exact syntax for doing this query without duplicates?

Original source

Related problems