Online Tutorials & Training Materials | STechies.com
Register Login

Python Remove Duplicates from List

|| 0

Python Remove Duplicates from List
Stechies

This tutorial explains various methods to remove duplicates from a list in Python.

In examples below, we are going to take a list that contains duplicate entries. Then we will generate another without any duplicate entries in it, and in this process, we will also keep the order of the element present in the list.

Example

Input List = [ s, t, e, c, h, i, e, s ]

Output List = [ s, t, e, c, h, i ]

Using Dictionary

# Python program to remove duplicate from list

# Using dictionary

# Create a list containing duplicate element

listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

print('List with duplicate: ',listOfNums)

# Create dictionary using list items as key

# Dictionary will automatically remove any duplicate entry

listOfNums = dict.fromkeys(listOfNums)

# Convert dictionary to list

listOfNums = list(listOfNums)

# Print list

print('List after removing duplicate: ',listOfNums)

Output:

List with duplicate: [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

List after removing duplicate: [10, 2, 45, 3, 5, 7, 8]

One Line Code:

listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

listOfNums = list(dict.fromkeys(listOfNums))

print('List after removing duplicate: ',listOfNums)
  • In above example, we convert the list to dictionary using list items like keys.
  • This will automatically remove all duplicates because dictionary cannot have duplicate keys.
  • Once all the duplicate entry is removed, we have converted dictionary back to list.

Note: Using dictionary order of elements present in the list is retained

Using Custom Function

# Python program to remove duplicate from list

# Using custom function

def removeduplicate(listOfNums):

# Create empty list

listNoDuplicate = []

# Iteration every element in the list

for x in listOfNums:

# Check if element is present in final list 'listNoDuplicate'

# Append element of the original list to 'listNoDuplicate' if not present in 'listNoDuplicate

if x not in listNoDuplicate:

listNoDuplicate.append(x)

return listNoDuplicate

# Driver Code

listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

print('List without Duplicate',removeduplicate(listOfNums))

In the above example, we have created an empty list ‘listNoDuplicate', append element of listOfNums in 'listNoDuplicate' if not present.

Note: The order of elements present in the list remains the same

Using set()

# Python program to remove duplicate from list

# Using set()

# Create a list containing a duplicate element

listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

print('List with duplicate: ',listOfNums)

# Using set()

# This will remove duplicate from list

listOfNums = set(listOfNums)

# Convert set to list

listOfNums = list(listOfNums)

# Print list

print('List after removing duplicate: ',listOfNums)

Output:

List with duplicate: [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

List after removing duplicate: [2, 3, 5, 7, 8, 10, 45]

Single line Code:

listOfNums = [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

listOfNums = list(set(listOfNums))

print('List after removing duplicate: ',listOfNums)

Note: Set method does not retain the order of elements present in the list

Remove duplicates from a list of lists

Example:

# Python program to remove duplicate from list of lists

import itertools

listorignal = [[1, 2], [4], [3, 4, 5], [1, 2], [3], [0]]

print("Original List", listorignal)

listorignal.sort()

new_list = list(listorignal for listorignal,_ in itertools.groupby(listorignal))

print("New List", new_list)

Output:

Original List [[1, 2], [4], [3, 4, 5], [1, 2], [3], [0]]

New List [[0], [1, 2], [3], [3, 4, 5], [4]]