Python list comprehension

I think  list comprehension is one of  the advanced feature that make python more userfriendly and efficient . Do you know what is list comprehension ? how can be use it ? .This post will help you to learn some basic ideas about list comprehension in  python.

What is a list comprehension?

Mainly list comprehension  feature is indroduced in python version 2.0 . It  is an elegent way  to define and  create lists in python. Using list comprehension we can create  new list from an existing list  and each element in the new list  is the result of some operations applied to each element in the old list. List comprehension is not a complicated idea it fit a for loop,if statement and assignment all are in one line . Let me explain concept behind list comprehension using some examples.

Here is a simple program which create a list of  squares of even numbers in the range of 10

>>>L=[]

>>>for x in range(10):

   if x%2 ==0:

     L.append(x**2)
>>>L
[0,4,16,36,64]

Now i am going to write same program using list comprehension technique

>>L=[x**2  for x in range(10) if x %2==0]
>>L
[0,4,16,36,64]

above two codes are identical, you can see that list comprehension is more good and  you get to see more of the code in one view. Another important factor is that list comprehension is the complete substitute for lambda() functions as well as the functions map(),filter() ,reduce().

Basic syntax  is:

[expr for  item1 in  seq1 for item2 in seq2 … for itemx in seqx if condition]

• the list comprehension starts with a ‘[‘ and ‘]’, to help you remember that the result is going to be a list
• there’s an expression based on the variable used for each element in the old list
• the word ‘for’ followed by the variable name to use followed by the word ‘in'
• the old list

Here are some more examples using list comprehension

Cross product of two sets

>>>colours=["red","blue","yellow"]

>>things=["house'","car","tree"]

>>>coloured_things=[(x,y) x in colours for y in things]

print coloured_things

[('red', 'house'), ('red', 'car'), ('red', 'tree'), ('blue', 'house'), ('blue', 'car'), ('blue', 'tree'), ('yellow', 'house'), ('yellow', 'car'), ('yellow', 'tree')]

Following list comprehension create a pythagorean triples

>>>[(x,y,z) for x in range(1,30) for y in range(x,30) for z in range(y,30) if x**2 + y**2 == z**2]
[(3, 4, 5), (5, 12, 13), (6, 8, 10), (7, 24, 25), (8, 15, 17), (9, 12, 15), (10, 24, 26), (12, 16, 20), (15, 20, 25), (20, 21, 29)]

Calculation of prime numbers between 1 and 100

>>>noprimes = [j for i in range(2, 8) for j in range(i*2, 100, i)]
>>>primes = [x for x in range(2, 100) if x not in noprimes]
>>>print primes
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]
>>>

List comprehension and generator comprehension

If you don’t know about generators in python please go to this link https://neethutv.wordpress.com/2012/01/11/generators-an-advanced-concept-in-python/ and come back to this article. Generator  comprehensions are introduced with python 2.6. They are simply generator expressions with paranthesis .The syntax and way of working of generator comprehension is same way as list comprehension , but most important difference is that generator comprehension return a generator object instead of list. Generator comprehensions  always have to written in paranthesis.  Here some examples of generator comprehensions

>>>x = (x **2 for x in range())
>>>x
<generator object <genexpr> at 0xb76acb44>

>>>for i in x:

  print i

0,1,4,9,16

List comprehension is one of the simple and useful feature of python.  It is  faster than equivalent for loop, but map(),filter() ,reduce() are generally faster than list comprehension. One thing to note that list comprehension create a new list. The new list is rebound the same name.

l = [ foo(i) for i in l ]

but a new list is created ,any other name that refer to original list will not changed. This is a rarely problem so to avoid this type of problems just use slice assignment instead of normal assignment.

l[:] = [foo(i) for i in l]

I think list comprehension is quite easy to use and can make code that manipulates lists shorter and more readble.

Generators- An advanced concept in python

I have always heard about python generators , now i got a chance to study about this generators . I think generators are one of important and advanced  feature  in python .What are these generators ? A generator is a function that produce a sequence of result instead of a single value while maintainig the state of function. Generators are very useful and simplfy our code also  improve the performance. Most typical use of generator is to define iterator  ,iterators are  fundamental part of python and they are used in the  ‘for’ statement. To defining a generator we must know about iterator and its working because generators are powerful tool to create iterators. In this article i explaining about concepts of generators how it improve efficiency of python.

What is iterators ?

 We are familiar with iterations, because  they are fundamental concept in python .When we create a list we can read its items one by one .This process is called iteration. Python allow many type of iterations such as  iterating over list ,dictionary ,file etc. But we don’t know what happen behind a for loop. The for statement call  iter() function and this function return an object that have  a .next() method which can access element one at a time and it raise StopIteration exception  when there is no element and tell to for loop to terminate. This is the  basic working of iterator. When we read values using iteration all values are stored in memory and we can read  as our wish .

What is a generator ?

Generators are iterable but we can only read values once. Its because they do not store all the values in memory. We can also define generator as a function that remember the point in the function body where it last returned.

Eg :

def countdown(n):
    while n > 0:
    yield n
    n -= 1

Calling a generator function create and return a generator object.

 >>>x=countdown(3)
 >>>print x
 <generator object at 0x58490>

It does not start running the function ,instead it create a generator object .Also the function not really exit  and it goes to a suspended  state . Then for loop try to loop over this generated object ,  then function resume from its suspended state and run until the next yield statement and return that as the next item. This procedure will happen until the function exits and at which point the generator raises StopIteration , and the loop exits. In the above function named countdown(3) which print first 3 and then generator forgot 3 and then print 2 , that also forgot and last print 1. So that improved the performance of code because less memory is used . We can only once read values using generator object. If we want to read again all these values then we need to create again a generator object .

Let us look about importance of yield statement  in a generator function

Yield statement

Yield statement is an important part of a generating function  which have following specifications,

• Only used in body of generating functions
• Using a yield statement in a function definition causes that definition to create a generating function instead of normal function
• Yield statement resume where the function left of
• When a yield statement is executed state of a generator function is frozen and next time .next() function invoked the function will resume

Now we can try to run a for loop over the generator object of above generating function.

>>> x.next()

3

>>>x.next()

2

>>>x.next()

1

>>>x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

Next we want to look about generated expressions
Generated expressions

  Generated expressons same as list comprhensions in python . But   some differences are there which are follows,

• Generator function do not construct a list
•  Only useful purpose is iteration
• Once consumed ,then can’nt reused
• Instead of bracket parentheses is used

eg:

 >>>a=[1,2,3,4]
 >>>>c=(2*x for x in a)
<generator object at 0x58760>
 >>>for i in a:
print i
2
4
6
8
>>>sum(i*i for i in range(10))
285

So  generator  function  make python more effective and  efficient and i  hope that this article will help you to learn about some informations about generators  in python ,

Introduction to unit testing in python

In computer programming unit testing is a method by which individual units of source code are tested to determine if they are fit for use. As a dynamic language python it is easier to test than other languages .This  article will help you to learn about unit testing in python .

Unit testing in python mainly using the python standard library testing framework called unittest. Unit tests are for testing components of our code ,usually classes or functions.We can start from the standard libarary module unittest 

The unittest module

Python unit test sometimes referred to as “PyUnit” is a python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in turn, a Java version of Kent’s Smalltalk testing framework. unittest support test for automation , sharing of setup and shutdown code for tests, aggregation of tests  into collections and independance of tests from reporting frame work . The unittest module provide classes that make esay to support the above qualities for a set of tests.

Important concepts in unittest 

• test fixure – represent preparation needed to perform one or more tests.
• test case – smallest unit of testing ,it check for a specific response for a particular set of input. unittest provide a base class ,TestCase which  used to create test cases.
• test suite – collection of test cases
• test runner – component provide an environment in which tests can execute and provide outcome to user . runner use a graphical interface or textual interface or return a special value to indicate result of executing tests.

Creating test cases

Test cases are most fundamental part of unit testing. A test case answer a simple questions about the code it is testing also a test case run completely itself no need of any human input. It determine by itself whether the function it is testing has passed or failed, without any human interpreting the results. In PyUnit, test cases are represented by the TestCase class in the unittest module. To make your own test cases you must write subclasses of TestCase.

Here  a simple example which i do from the official python tutorial creating test cases , we  can test three functions from python random module.

import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

     def setUp(self):
         self.seq = range(10)

     def test_shuffle(self):
    # make sure the shuffled sequence does not lose any elements
        random.shuffle(self.seq)
        self.seq.sort()
        self.assertEqual(self.seq, range(10))

    # should raise an exception for an immutable sequence
        self.assertRaises(TypeError, random.shuffle, (1,2,3))

     def test_choice(self):
        element = random.choice(self.seq)
        self.assertTrue(element in self.seq)

     def test_sample(self):

        self.assertRaises(ValueError,random.sample,self.seq,20)
        for element in random.sample(self.seq, 5):
          self.assertTrue(element in self.seq)

if __name__ == '__main__':
  unittest.main()

In python a test case is created by subclassing unittest base class TestCase as follows,

    class   TestSequenceFunctions(unittest.TestCase)

Testcase defining test methods whose names are starting with ‘test’.Test methods performing actions and assert methods are to verify the expected behaviour and results. In the above program three functions in random module are tested, and we use three test methods.Each test is call to assert methods to check results. When a setUp() method is defined, the test runner will run that method prior to each test and when a tearDown() method is defined then test runner will run that method after all methods.

Testing for equality

The most common type of assertion is the assertion of equality between two values. If the assertion fails we got the incorrect result and our test will fail. In above example we check whether the sorted list from 0-9 numbers  will equal to range(10). The result is true and test must pass.

assertEqual – This method test equality of two values, if the values are not equal test must fail and raise exception.

  self.assertEqual(self.seq, range(10))

asserNotEqual – This method check two values are not equal .

    self.assertNotEqual(1 + 2, 4)

Testing for exceptions

There is also testing for exceptions also known as testing for failure. It is not enough that function succed when given a good input , we must also test they fail when we given a bad input or they must fail as our expectations. TestCase class of unittest provide assertRaises method to check the expected exceptions ,which take following arguments,

• Exception you are expecting
• Function you are testing
• Arguments you are passing to function

In the above program test method test_sample test the exception

self.assertRaises(ValueError,random.sample,self.seq,20)

Here we select 20 random sample numbers from the range(10).We know that result will raise an exceptions and we use the assertRaises to get our expected exception as ValueError.

Testing for the truth

Some tests are assert the truth of some condition.In the above program test_choice method test the truthness. We select a random element from the sequence and test the presence of truth of statement.

   element = random.choice(self.seq)
        self.assertTrue(element in self.seq)

Test Fixtures

Fixtures are the resources needed by  a test , For example  you are writing several tests for the same class ,those tests all need an instance of those class to use for testing. TestCase include functions to configure and clean up fixtures needed to our test. To configure  fixtures  override setUp() and clean up override tearDown().

Following a simple example to a test fixture ,

import unittest
class FixturesTest(unittest.TestCase):

     def setUp(self):
         print 'In setUp()'
         self.fixture = range(1, 10)

     def tearDown(self):
         print 'In tearDown()'
         del self.fixture

     def test(self):
         print 'in test()'
         self.failUnlessEqual(self.fixture, range(1, 10))

if __name__ == '__main__':
    unittest.main()</pre>

When we run this sample code we can see the order of execution of fixture and test method :

Running testes from the command line

unittest module contain a function called main ,which used to turn a test module into script that it will run the test conatins. unittest.main() automatically load test cases in the current module. When we excute our program in the command line as pyhton test_pgm.py then all tests will run and we get result as follows.

Time complexity of various operations in python list

Time complexity is the amount of time required to execute the program . Time complexity  is calculated using “Big O”

Here i give a description of amount of time take each opeartions in python data types. Let us take input size as “n” and “m’ is additional parameter

List

A list in python is like array contain a sequence . Below a description about time complexity of varous list operations. Individual actions (append etc.) in any operation take  time complexity is O(1) else depend on size of input because after some operation elements must move their positions and time will vary (insert , extend etc.)

append       –  O(1)       insert element at end of list , no moving

insert             – O(n)    inserting begining or at a position , so elements move to right n times

delete item     – O(n)    elements move after deleted positon depend upon size of input

iteration        – O(n)    iterating n times

get slice        –  O(k)    k a value of parameter

extend          –  O(k)    k is new list to extend

get item        –  O(1)

set item        –  O(1)

get length     –  O(1)

i in list         –  O(n)

sort              –  O(n logn)

Optimization strategy of fibonacci series

Fibnoacci series is a basic example which teach recursion . Recursion a powerful programming  paradigm in which a function call itself .

Here a simple program of fibonacci series using recursion

def f(n):
 if n < 2:
   return n
 return  f(n-1) + f(n-2)

But this program just getting f(40) take atleast 8 minutes and for f(50) take really a long time , so it  is less efficient and too slow

why it slow?

The short answer is ‘recursion’ . Let us see how many times fibnocci function  calls itself recursively to compute f(50)

• f(50) is called once
• f(49) is called once
• f(48) is called 2 times
• f(47) is called 3 times
• f(46) is called 5 times
• f(45) is called 8 times
• …..
• f(1) is calculated 12,586,269,025 times

Number of recursively call grows  high rate , this simple program take O(2^n) times.

The main problem is that there is no way of remembering already calulated values . So it recalculate the same again, this recalculation take a larger time and then efficiency decreses and if number of input larger then duplicate calculation increses., and result take more time, therfore  we need a optimization strategy

Memoization

Memoization is a optimization strategy  which store the results for  later use and no recalculation

Below program explain how memoizing in fibonacci series

cache = {}
def fib(n):
  if n in cache:
     return cache[n]
  else:
     if n <2:
       cache[n]=n
     else:
      cache[n]= fib(n-1) + fib(n-2)
     return cache[n]

Here the dictionary named cache store the numbers which already calculated. When fibnocci function is called it check the cache if it contain result, if contain then function return immediately and no more recursive calls. If not ,  it has to compute new value and this value added to cache.

so using memoization there is no recalculation and we got result faster for large numbers. This test take less than one second please try it ….and see how faster it..

Basics of python

Python a scripting language with simple syntax and does not need compilation.  Its codes are too short and flexible, i interested to learn python because it easy to learn and simple.  Due to its open source nature python is portable also we run the program directly from the source code.

In my laptop python version 2.6.4 already installed and i started to programming in python ,this my first step to  world of python

just type python on terminal and then enter,

$ python
Python 2.6.4 (r264:75706, Oct 29 2009, 15:38:25)
[GCC 4.4.1] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> print ‘Hello world’
Hello world
>>>

Python gives you the output of the line immediately! What you just entered is a single Python statement. We use print to  print any value that you supply to it. Here, we are supplying the text Hello world and this is promptly printed to the screen.

python  as a simple calculator

We can type any expression in interpreter and it will write value, operators +, -, * and / work just like in most other languages

>>> 2+2
4
>>> 2-2
0
>>> 2//2
1
>>> 2/2
1b
>>> (50-5*6)/4
5
>>> 7/-3
-3
>>>

Different python datatypes

    Strings

  Strings in python is a series of  characters . Strings surrounded by   ”  ”  or   ‘  ‘

  Strings are immutable

  >>> flower=”Rose”
  >>> print flower
  Rose
 >>> len(flower)
 4
 

String slices   –  Substring of a string called slice

>>> fruit=”banana”
>>> fruit[:3]
‘ban’
>>> fruit[3:]
‘ana’
>>> fruit[2:]
‘nana’

The operator [n:m] returns the part of the string from the n-th character to the m-eth character, including the first but excluding the last.

Lists

Lists are ordered set of values defined with index

>>> list=[1,2,3,4,5]
>>> list
[1, 2, 3, 4, 5]
>>>
Common list operations

• list.append(elem) — adds a single element to the end of the list.
• list.insert(index, elem) — inserts the element at the given index, shifting elements to the right.
• list.extend(list2) -adds the elements in list2 to the end of the list. Using + or += on a list is similar to using extend().
• list.index(elem) — searches for the given element from the start of the list and return index               
• list.remove(elem) — searches for the first instance of the given element and removes it
• list.sort() — sorts the list in place (does not return it). (The sorted() function shown below is preferred.)
• list.reverse() — reverses the list in place (does not return it)
• list.pop(index) — removes and returns the element at the given index. Returns the rightmost element if index is omitted (roughly the opposite of append()).

Dictionary

Python built-in  mapping type. Map keys to values

>>> dict={}
>>> dict[‘a’]=1
>>> dict[‘b’]=2
>>> dict[‘c’]=3
>>> dict
{‘a’: 1, ‘c’: 3, ‘b’: 2}
>>>

File

• To open a file  –   file=open(filename, mode)
•  To read a file  –    file.read()
•  To write to file  – file.write()
• To close a file  –  file.close()                   
  

Regular expressions  –     

Regular expressino is for pattern matching in a text file. ‘re ‘ module provide all functions for regular expressions .

• re.search(pat, text)  – Search a pattern in given text
• match.group() – Return the pattern
• re.findall() – All instances match this expression
• re.sub(paten,  replace, str)  – Search  all instances of given pattern in string and replace

Python utilities      Python provide many utility modules

os module - os module include functions to interact with filesystem

• filenames = os.listdir(dir)  – list of filenames in directory  ‘dir’
• os.path.join(dir,  filename)  – path of filename in directory
• os.path.abspath(path)  – given a path ,return absolute form
• os.path.dirname(path)  – give as dir/foo/bar.html
• os.path.basename(path) – give basename as bar.html
• os.path.exists()   – true if exists
• os.mkdir(dir) – make a directory
• shutil.copy(sourcepath, destpath)  –  copy files from source to destination

 commands module - Run external commands and capture output

• (status, output) = commands.getstatusoutput(cmd) – run command and return status and output.  if status is nonzero command failed
• commands.getstatus()  –  give status
• commands.getoutput()  – return value is a string containing output

urllib module - provide url fetch

• ufile = urllib.urlopen(url) -return file for given url
• text=ufile.read() – read like a file
• info = ufile.info() — the meta info for that request
• baseurl = ufile.geturl() — gets the “base” url for the request
• urllib.urlretrieve(url, filename) — downloads the url data to the given file path
• urlparse.urljoin(baseurl, url) — given a url that may or may not be full, and the baseurl of the page come and return full url