This tutorial explains various methods to remove duplicates from a list in Python.
In examples below, we are going to take a list that contains duplicate entries. Then we will generate another without any duplicate entries in it, and in this process, we will also keep the order of the element present in the list.
Example
Input List = [ s, t, e, c, h, i, e, s ]
Output List = [ s, t, e, c, h, i ]
Using Dictionary
# Python program to remove duplicate from list
# Using dictionary
# Create a list containing duplicate element
listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
print('List with duplicate: ',listOfNums)
# Create dictionary using list items as key
# Dictionary will automatically remove any duplicate entry
listOfNums = dict.fromkeys(listOfNums)
# Convert dictionary to list
listOfNums = list(listOfNums)
# Print list
print('List after removing duplicate: ',listOfNums)
Output:
List with duplicate: [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
List after removing duplicate: [10, 2, 45, 3, 5, 7, 8]
One Line Code:
listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
listOfNums = list(dict.fromkeys(listOfNums))
print('List after removing duplicate: ',listOfNums)
- In above example, we convert the list to dictionary using list items like keys.
- This will automatically remove all duplicates because dictionary cannot have duplicate keys.
- Once all the duplicate entry is removed, we have converted dictionary back to list.
Note: Using dictionary order of elements present in the list is retained
Using Custom Function
# Python program to remove duplicate from list
# Using custom function
def removeduplicate(listOfNums):
# Create empty list
listNoDuplicate = []
# Iteration every element in the list
for x in listOfNums:
# Check if element is present in final list 'listNoDuplicate'
# Append element of the original list to 'listNoDuplicate' if not present in 'listNoDuplicate
if x not in listNoDuplicate:
listNoDuplicate.append(x)
return listNoDuplicate
# Driver Code
listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
print('List without Duplicate',removeduplicate(listOfNums))
In the above example, we have created an empty list ‘listNoDuplicate', append element of listOfNums in 'listNoDuplicate' if not present.
Note: The order of elements present in the list remains the same
Using set()
# Python program to remove duplicate from list
# Using set()
# Create a list containing a duplicate element
listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
print('List with duplicate: ',listOfNums)
# Using set()
# This will remove duplicate from list
listOfNums = set(listOfNums)
# Convert set to list
listOfNums = list(listOfNums)
# Print list
print('List after removing duplicate: ',listOfNums)
Output:
List with duplicate: [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
List after removing duplicate: [2, 3, 5, 7, 8, 10, 45]
Single line Code:
listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
listOfNums = list(set(listOfNums))
print('List after removing duplicate: ',listOfNums)
Note: Set method does not retain the order of elements present in the list
Remove duplicates from a list of lists
Example:
# Python program to remove duplicate from list of lists
import itertools
listorignal = [[1, 2], [4], [3, 4, 5], [1, 2], [3], [0]]
print("Original List", listorignal)
listorignal.sort()
new_list = list(listorignal for listorignal,_ in itertools.groupby(listorignal))
print("New List", new_list)
Output:
Original List [[1, 2], [4], [3, 4, 5], [1, 2], [3], [0]]
New List [[0], [1, 2], [3], [3, 4, 5], [4]]