In this tutorial we will learn what is Python List Comprehension and how to use it in Python programs using programming examples:
Professional programmers are constantly looking for ways to write code that is more readable and often faster at the same time.
The good news is that, when it has to do with Python lists, there is a way to achieve both readability and speed with List comprehensions.
=> Visit Here To Learn Python From Scratch
Table of Contents:
What Is Python List Comprehension
List comprehensions provide a more concise way to create lists in situations where map() and filter() and/or nested loops would currently be used (more on this later).
Based on this documentation, List comprehensions become a part of the Python language with release 2.0.
Syntax:
[expression for value in iterable [if condition]]
Where both expression and condition depend on value, and the bracket enclosing the if clause simply tells us that is it optional.
To summarize, List comprehension portrays two main powerful characteristics: Readability, Speed.
Readability
Example 1: Say we want to create a new list by adding 10 to each item of some other list.
Solution1.1: The most familiar way is to use a traditional for loop as below:
def multiply_by_10(list_items):
new_list_items = [] # create a new empty list
for item in list_items:
# for each item, multiply by 10 and append to the new list.
new_list_items.append(item*10)
return new_list_items
if __name__ == '__main__':
list_items = [2,3,7,9]
print("Result: {}".format(multiply_by_10(list_items)))
Output
Solution1.2: Now, using list comprehension, we have:
def multiply_by_10(list_items):
# list comprehension.
return [item*10 for item in list_items]
if __name__ == '__main__':
list_items = [2,3,7,9]
print("Result: {}".format(multiply_by_10(list_items)))
Output
We see from the solutions above that, although someone with a bit of knowledge in Python will understand solution 1, we will all agree that solution 2 is shorter, more readable, and with an intent that is explicit.
Speed
List comprehension is arguably faster than normal for loops. Though it may not be noticeable for simple computations, when a program begins to compute complex operations, we will then begin to see the benefit of using this syntactic form.
Example 2: Let’s investigate this claim with an iterable of 10,000 items.
Now let’s time the method used in the solution1.1 above.
import time
def speed_ex2_sol1(list_items):
start = time.time() # start time
new_list_items = [] # create a new empty list
for item in list_items:
# for each item, multiply by 10 and append to the new list.
new_list_items.append(item*10)
end = time.time() # end time
return new_list_items, end-start
if __name__ == '__main__':
# create an iterable of 10000 items
list_items = range(0, 10000)
# print just the time it took to compute
print("Time: {}".format(speed_ex2_sol1(list_items)[1]))
Output
Now, let’s time the method used in solution1.2 above.
import time
def speed_ex2_sol2(list_items):
start = time.time() # start time
# list comprehension
new_list_items = [item*10 for item in list_items]
end = time.time() # end time
return new_list_items, end-start
if __name__ == '__main__':
# create an iterable of 10000 items
list_items = range(0, 10000)
# print just the time it took to compute
print("Time: {}".format(speed_ex2_sol2(list_items)[1]))
Output
Comparing the time taken to compute both for solution 1.1 and solution 2.2 above, we see that solution 1.2 is shorter than solution 1.1 by about 0.000409 seconds. Though the seconds obtained above may vary, one thing is for sure i.e. the method using list comprehension will always be shorter for this example.
List Comprehension Vs Map/Filter
In this section, we will compare the readability and speed between list comprehension and map/filter functions.
Readability
Prior to the introduction of list comprehension(or listcomps as many pythonistas call it), the map/filter functions composition was commonly used to filter and map functions and items. However, since the introduction of listcomps, this is prefered over map() and filter().
Below, let’s see how it looks like to use a map/filter composition and a listcomp.
Example 3: We have a string ‘2459a09b’ and we want to extract all integer literals, and use int() to cast them into integers.
Solution 3.1: Using map/filter composition
In order to properly use this composition, we should understand what we are trying to do. In the map/filter composition, the map() and filter() functions can be used in any order and one nested in another based on the problem we are trying to solve.
In this case, we have the following steps:
- Filter-in all integer literals from the string.
- Execute a specific function for each item in the filtered iterable.
From the above steps, we understand that the filter() function will be nested into the map() function.
>> str = '2459a09b' >>> list(map(int, filter(lambda s: ord(s) >=49 and ord(s) = 57, <str))) [2, 4, 5, 9, 9]
In the filter() function, we used the lambda expression to check for each item if it is an integer literal using the ord() function.
Our filter() function will return a list of all items that meet the requirements, which are then passed to the map() function where each item will be mapped to the int() function for conversion.
Solution 3.2: Using Listcomps
>>> str = '2459a09b' >>> [int(s) for s in str if ord(s) >=49 and ord(s) <= 57] [2, 4, 5, 9, 9]
We can see how straightforward and easy listcomps are. We can read it in plain english as:
For each item in str, cast it into integer using int() if the item meets specification of being an integer literal.
Speed
It is often arguably said that listcomps are faster than map() and the noticeable condition that makes listcomps microscopically faster than map() is when a lambda function is involved.
However, this is not always the case. To sum up, listcomps are not always faster than map(), this may depend on the machine that runs the code.
Let’s consider the example below.
#1) Without lambda expression
Open an editor and paste the following code:
import timeit
def compare_speed():
# time code for map()
map_time = timeit.timeit("str='43028473'; map(int, str)")
# time code for listcomps
listcomps_time = timeit.timeit("str='43028473'; [int(x) for x in str]")
# print results
print("MAP: 1000000 loops, best of 3: {:.3f} usec per loop".format(map_time))
print("LISTCOMPS: 1000000 loops, best of 3: {:.3f} usec per loop".format(listcomps_time))
if __name__ == '__main__':
compare_speed()
Output
The above code compares the following code snippet.
For map()
str = '43028473' map(int, str)
For listcomps
str= '43028473' [int(x) for x in str]
These basically do the same thing, iterate through the string and for each item, convert it to an integer.
We see from the output above that the map() approach(0.115 usec per loop) is faster than the listcomp approach(1.226 usec per loop).
#2) With Lambda Expression
Open your editor and paste the following code:
import timeit
def compare_speed_lambda():
# time code for map()
map_time = timeit.timeit("str='43028473'; map(lambda s: int(s), str)")
# time code for listcomps
listcomps_time = timeit.timeit("str='43028473'; [int(x) for x in str]")
# print results
print("MAP: 1000000 loops, best of 3: {:.3f} usec per loop".format(map_time))
print("LISTCOMPS: 1000000 loops, best of 3: {:.3f} usec per loop".format(listcomps_time))
if __name__ == '__main__':
compare_speed_lambda()
Output
We see from above that for both cases, the map() approach seems faster. However, this may not be the case on other machines.
Nested List Comprehension Vs Nested Loops
This is also a section that isn’t certain. As we will see below, some nested loops are preferred to nested list comprehensions and vice versa.
Nested list comprehension has the following syntax:
[ expression for value1 in iterable1 [if condition1]
for value2 in iterable2 [if condition2]....
for valueN in iterableN [if conditionN] ]
Readability
In most cases, when it comes to readability, Pythonistas prefer nested loops to nested list comprehension.
Let’s consider the example below:
Example 4: The functions below flatten a nested list for all inner lists with size equal to 3.
Open an editor, paste this code and name it as nested_listcomps_vs_loop.py
def nested_loop(nested_list):
new_list = [] # create new list to hold the result
for inner_list in nested_list:
if len(inner_list) == 3: # check for inner list equal to 3
for item in inner_list:
new_list.append(item)
return new_list
def nested_listcomps(nested_list):
return [item for inner_list in nested_list if len(inner_list) ==3
for item in inner_list]
if __name__ == '__main__':
nested_list = [[3,4,2],[4,9,3,0],[0,1,3]] # nested list to be flatten
print("Nested Loop: {}".format(nested_loop(nested_list)))
print("Nested Listcomps: {}".format(nested_listcomps(nested_list)))
Output
Many pythonistas prefer the nested loop approach as it is easy to follow its pattern, especially for newbies. However, most programmers who have been coding for many years will prefer the nested listcomps approach as it is shorter, can fit in a lesser line of code and as we will see later, it is faster.
Speed
In addition to being shorter and more compact, nested listcomps are microscopically faster.
Example 5: Let’s measure the speed of example 4 above.
Open an editor, paste the code below and save it in the same directory where you saved nested_listcomps_vs_loop.py defined in example4.
import timeit
def nestedLoop_time():
# set up the code. Import function to be evaluated
SETUP_CODE='''
from nested_listcomps_vs_loop import nested_loop'''
# Run the code to be tested
TEST_CODE='''
nested_list = [[3,4,2],[4,9,3,0],[0,1,3]]
nested_loop(nested_list)'''
# call the timeit() which will run 10000 times repeatedly(3)
times = timeit.repeat(
setup=SETUP_CODE,
stmt=TEST_CODE,
repeat=3,
number=10000
)
print("Nested Loop time: {}".format(min(times)))
def nestedListcomps_time():
SETUP_CODE='''
from nested_listcomps_vs_loop import nested_listcomps'''
TEST_CODE='''
nested_list = [[3,4,2],[4,9,3,0],[0,1,3]]
nested_listcomps(nested_list)'''
times = timeit.repeat(
setup=SETUP_CODE,
stmt=TEST_CODE,
repeat=3,
number=10000
)
print("Nested Listcomps time: {}".format(min(times)))
if __name__ == '__main__':
nestedLoop_time()
nestedListcomps_time()
Output
The code above uses timeit() to time the execution of our two functions and we can clearly see that the nested listcomps approach is faster.
We have seen that for the sake of readability, we should consider using traditional for loops, especially when:
- Nesting is more than 2 levels deep.
- The logic is too complex.
Nested listcomps can be considered if the speed needs to be leveraged.
But there is a saying:
“Programs must be written for people to read, and only incidentally for machines to execute”.
Common Mistakes On Python List Comprehensions
Python List comprehensions have been around for long and have been used in many projects. Though they are powerful and faster, they are rarely used in code where the readability is leveraged.
Newbies turn to use list comprehensions as it is shorter and it makes them feel more of a programmer than the rest. However, below are some common mistakes that newbies turn to make while using listcomps.
Positioning Of for Loops And If Statements
This is usually a problem when nesting is more than 2 levels deep as many for loops and optional if statements are needed.
Example 6: Consider the code that flattens a 3 level deep nested list and ignores negative numbers.
def flatten_3_level_deep(nested_list):
new_list = []
for inner_1st in nested_list: # first level
for inner_2nd in inner_1st: # second level
for item in inner_2nd: # third level
if item >=0: # check condition, ignore -ve numbers
new_list.append(item)
return new_list
if __name__ == '__main__':
nested_list = [[[3,2,-6],[3,0,1],[-1,9,8]]]
print("FLATTEN LIST: ", flatten_3_level_deep(nested_list))
Output
The mistake that is mostly done is to wrongly order each for loop in a listcomp. Most newbies will do it this way:
[item for item in inner_2nd if item >= 0
for inner_2nd in inner_1st
for inner_1st in nested_list]
The first for loop in the above code is correct for lists. However, for nested lists, things become different. Different in that, we start ordering the for loops from the outermost levels.
Example 7: Right way to order for loops and optional if statements in a listcomp.
def listcomps(nested_list):
return [item for inner_1st in nested_list
for inner_2nd in inner_1st
for item in inner_2nd if item >= 0]
if __name__ == '__main__':
nested_list = [[[3,2,-6],[3,0,1],[-1,9,8]]]
print("Listcomps: ", listcomps(nested_list))
Output
Using The Continuation Line Escape ( / )
Most of the time, we are used to printing long texts that usually cause us to use the continuation line escape(\) so that it is readable and well formatted and when we come in a listcomp, we turn to do the same.
Example 8: Consider the listcomps() function in example 7 above. We may feel the need to put / after each line as below:
return [item for inner_1st in nested_list /
for inner_2nd in inner_1st /
for item in inner_2nd if item >= 0]
Though the above code will run successfully, in Python code, line breaks are ignored inside pairs of []. So we can build multi-line listcomps without using the ugly line continuation escape( \).
Assigning Before Returning
In the cases where the result of a listcomp needs to be returned from a function, there is really no need to assign it first to a variable if it won’t be further used in that function. This is something that we often do and don’t even realize that it is useless.
Example 9: The listcomps() function in example 7 returns the generated list directly without first assigning it to a variable.
When Not To Use A List Comprehension
Listcomps are often not recommended due to readability and time wastage(memory). We have already seen many reasons as to why we should use traditional for loops to leverage readability above.
In this section, we shall see where using listcomps can be a waste of time and memory.
#1) When Initializing Other Sequences (Tuple, Array)
Example 10: Initialize a tuple from a listcomp
>>> str = "48387730384" >>> tuple([int(s) for s in str]) (4, 8, 3, 8, 7, 7, 3, 0, 3, 8, 4)
The above code uses a listcomp to iterate through a string of integers and cast each item into an integer. It then uses the tuple() to create a tuple.
The problem with this is that the listcomp first creates a list, then the list is used in tuple() to initialize a tuple.
In this case, a generator expression should be used instead. A Generator expression uses the same syntax as listcomps, but is enclosed in parenthesis rather than brackets. It saves memory as it yields items one by one using the iterator protocol Unlike the listcomp that creates a new list.
Example 11: Initialize a tuple from a generator expression.
>>> str = "48387730384" >>> tuple([int(s) for s in str]) (4, 8, 3, 8, 7, 7, 3, 0, 3, 8, 4)
#2) When Dealing With Large Data
In most cases, we have been using listcomps to compute small data. So, we haven’t seen the trouble that this can cause while dealing with large data.
While dealing with small data like a range of thousands, a listcomp can handle it admirably.
>>> sum([i**2 for i in range(2000)]) 2664667000
However, when dealing with millions and above, we are better off using a lazy function like map() or a generator expression.
>>> sum(i**2 for i in range(2000000)) 2666664666667000000 >>>
They both operate lazily i.e. they return an iterator in which we can request the next value or iterate through until we hit the end. On the other hand, a listcomp will compute and save a list of this big data in memory which may slow-down our machine or cause it to crash.
Frequently Asked Questions
Q #1) Are List Comprehensions faster in Python?
Answer: List comprehensions are microscopically faster than traditional Python loops. However, List comprehensions are arguably faster than the map and filter functions i.e. they may be faster in some machines and also depending on the level of complexity.
Q #2) Why do we use List Comprehension in Python?
Answer: From the name, List comprehensions are a comprehensive, elegant and more compact way to generate lists from iterables based on some operations. Moreover, they are also faster in most cases.
However, it is not recommended to use List comprehensions while dealing with more than 2 levels deep nestings and also when the logic for generating a list is too complex.
Q #3) What kind of operations can a Python List Comprehension do?
Answer: Basically, a list comprehension generates a list from operations on an iterable. These operations are applied to each item of that iterable. So, whatever operation that can be carried out on items of the given iterable will be supported in a list comprehension.
Q #4) Why is it called List Comprehension?
Answer: The comprehension part in a list comprehension tells us that it is a comprehensive way to generate list from operations on an iterable.
Conclusion
In this tutorial, we discussed what list comprehension is all about, and how it is compared to map/filter composition. We also analyzed nested list comprehension versus nested loop under readability and speed.
Towards the end of this tutorial, we discussed some common mistakes on Python list comprehension and when not to use a list comprehension.
=> Check ALL Python Tutorials Here