• Creator: Guido van Rossum
• Introduced: 1991
• Open source
• Comes standard on many UNIX/Linux systems, macOS
  • Note: The system Python on macOS and Linux is currently version 2.x and should not be used for this course.
• Windows and macOS installers obtainable from python.org and activestate.com
  • Do not install Python from the Windows Store.
  • On Windows, be sure to select the “Add python.exe to PATH” option when installing.
• Used for system programming/administration, web programming/development, network programming (it was the original bittorrent client), GUI development, games, data science

• CGI/web application programming
  • Web application frameworks
  • Youtube
  • Exploit development/testing
• Application scripting
• System maintenance/installation
  • RedHat/Fedora Linux's Anaconda, DNF package management system
• GUIs
  • TkInter (Tk for Python)
  • PyGTK
• Games
  • Pygame
• Data science
  • Anaconda

• Official site
• The Python Tutorial
• The Python Standard Library (Useful)
• The Python Language Reference
• Dive Into Python (online book)

• Programs typically given .py extension
• Executed with python prog.py or python3 prog.py
• Or use shell script type line at top of Perl script (UNIX systems only)
  • #!/path/to/python
  • and make executable with chmod +x prog.py
• Interactive shell (read-eval-print loop) by running python or python3

• A web-based interactive computational environment that supports Python
• Popular for data science programming in Python
• Notebooks are saved with the .ipynb file extension.
• Link: Installation and getting started
  • Can also use on Google Colab environment for data science/machine learning

• Do this in the python interactive shell (python3)

>>> import calendar
>>> cal = calendar
>>> cal.prmonth(2017,2)
  
   February 2017
Mo Tu We Th Fr Sa Su
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28
  
>>> cal.weekday(2017,2,17)
4
  
>>> cal.weekday(1973,11,14)
2
    
>>> cal.prmonth(1973,11)
   November 1973
Mo Tu We Th Fr Sa Su
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30

>>> # Some object introspection
>>> # Type cal. and hit tab a couple of times
>>> cal.
>>> # Also, try
>>> dir(cal)

>>> x = 'hello, world'
>>> y = x.upper()
>>> y
'HELLO, WORLD!'
  
>>> def swapper(mystr):
.  .  .      return mystr.swapcase() # indent mandatory
.  .  .
>>> swapper(x)
'HELLO, WORLD!'
  
>>> x
'hello, world!'
  
>>> def parts(mystr,sep=','):
.  .  .     return mystr.partition(sep)
.  .  .

>>> parts(x)
('hello', ',', ' world!')

>>> def personalize(greeting, name='Joni'):
.  .  .     # Replaces 'world' with a given name
.  .  .     return greeting.replace('world', name)
.  .  .

>>> x
'hello, world!'
  
>>> personalize(x, 'Joanne')
'hello, Joanne!'

>>> personalize(x) # Use the default name='Joni' parameter
'hello, Joni!'


>>> # Python functions are "first class" (http://tiny.cc/ggh4vz)
  
>>> funclist = [swapper, personalize, parts]
>>> for func in funclist:
.  .  .     func(x)
.  .  .
'HELLO, WORLD!'
'hello, Joni!'
('hello', ',', ' world!')

• Python motto: “There should be one–and preferably only one–obvious way to do it.”
  • Note the contrast with Perl.

• Indentation
  • Python uses indentation to indicate the run of a block.
    • That makes indentation mandatory.
  • Blocks in some other language:

void foo(int x) {
        if (x == 0) {
		bar();
		baz();
	} else {
		quo(x);
		foo(x - 1);
	}
}

• Blocks in Python

def foo(x):
	if x == 0:
	        bar()
		baz()
	else:
		quo(x)
		foo(x - 1)

• Another example

x = 1                       # block 0
if x == 1:                  # header line:
    y = 2                   # block 1
    if y == 2:              # header line:
	print('in block2')  # block 2
    print('in block1')      # block 1
print('in block0')          # block 0

• Exceptions to the indentation-as-blocks rule or the “whitespace thing”

# open list bracket [] pairs may span lines
L = ["Good",
     "Bad",
     "Ugly"]

# Backslashes allow line continuation
if a == b and c == d and \
   d == e and f == g:
   print('old')

# Parentheses allow line continuation, usually
if (a == b and c == d and
    d == e and e == f):
    print('new')

• stdin/out in Perl:

while ($myline = <STDIN>) {
	print $myline;
}

• Equivalent in Python: (Exercise 1)

import sys
for line in sys.stdin:
	sys.stdout.write( line )

• if/elif/else, while/else, for/else

# Assume these assignments:
x = 10
y = 10
b = 1


# if then else
if (b == 1):
   y = 1
elif (x == 10):
   y = 2
else:
   y = 3


# while (else) loop
while (x != 0):
   x = x - 1
   if (b == 1): continue # continue with next loop repetition
   break                 # break out of loop; skip else:
else:                    # run if we didn't exit loop with break
   x = x + 1


# for (else) loop
for x in range(4):      # repeats 4 times x=0..3
   y = y + 1
   if (b == 1): continue
   break                # break out of loop; skip else:
else:                   # run if we didn't exit loop with break
   y = y + 1

• “Everything is an object”
  • Object types
    • Numbers - 3.1415, 1234
    • Strings - 'spam', “guido's”
    • Lists - [1, 2, 3, 4], ['one', 'two', 'three']
      • As in Perl, arrays are named lists
    • Dictionaries - {'food': 'spam', 'taste': 'yum'}
      • These are Python's associative arrays (hashes)
    • Tuples - (1, 'spam', 4, 'U')
      • immutable lists
    • Files - text = open('eggs.txt', 'r').read()

• Strings, lists, and tuples are categorized as built-in “Sequence Types” in Python.
  • See https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range
    • Strings and Tuples are “immutable” sequence types.
      • Once they are created, they cannot be modified.
    • Lists are “mutable” sequence types.
• Dictionaries are categorized as a built-in “Mapping Type”
  • See https://docs.python.org/3/library/stdtypes.html#mapping-types-dict

• concatenating: str1 + str2
  • cannot normally mix types around “+” when concatenating
• repeating: str2 * 3
• indexing: str2[ i ]
• slicing: str2[ i:j ]
• length: len( str2 )
• methods:
  • str2.find( 'pa' )
  • str2.replace( 'pa', 'xx' )
  • str1.split()
• convert to string with str function: str( len( str2 ) )

print("one is " + str(1))

• See https://docs.python.org/3/library/stdtypes.html#string-methods for complete list of Python string methods
• Do Exercise 2.

• This is old.
• similar to printf function in other languages
• Syntax:

'formatting codes corresponding to list of objects, and other characters' % ( comma-separated list of objects )

• Example:

print("Number of %i character words: %i" % ( x, char_count[ x ] ))      # %i denotes integer

• See https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting for string formatting operations.

• It is recommended that you use the newer f-strings instead.
  • available since Python 3.6
  • https://docs.python.org/3/tutorial/inputoutput.html

• Lists can be “anonymous”

for x in [1, 2, 3]:   # list [1, 2, 3] is anonymous
   print(x)

• The range() function generates a list-like iterable sequence type object of values, most frequently numbers.

for x in range( 0, 10 ): # range( 0, 10 ) or just range( 10 ) yields the range 0..9
   print(x)

• Named lists are like arrays
  • But in Python, you call them “lists”, not “arrays”.

List1 = [0, 1, 2, 3]
List2 = range( 1, 5 )     # Not a list, but a range object
List2 = list(range(1, 5)) # Convert range to list containing [1, 2, 3, 4]

• Lists have to be “declared” if starting as an empty list

List3 = []   # an empty list

• Lists are a “Mutable Sequence Type” in Python.
  • See http://docs.python.org/library/stdtypes.html#mutable-sequence-types for operations and methods that apply to mutable sequnce types.

• Size of a list

len( List3 )

• Concatenate lists

list1 = list(range( 1, 5 ))
list2 = list(range( 6, 10 ))
list3 = list1 + list2
print(list3)   # output: [1, 2, 3, 4, 6, 7, 8, 9]

• Grow list by one object

list1.append( 4 )

• Grow list by a list of objects

list1.extend( [5, 6, 7] )

• Sort, reverse
  • Beware: in-place alteration of list contents
    • Make a list copy first, if needed

list1.reverse()
print(list1)   # output: [4, 3, 2, 1]
list1.sort()
print(list1)   # output: [1, 2, 3, 4]

• Shrink list by one object

del list3[ len(list3) - 1 ]   # delete last element at last index of list3

x = list3.pop()   # delete last element of list3 and assign val to x

• Using Python's for structure

# Method 1:
for x in list3:
   print(x)   # print values in list3, one per line

# Method 2:
for x in range( len( list3 ) ):
   print(list[ x ])   # print index-accessed values in list3, one per line

• See http://docs.python.org/library/stdtypes.html#mapping-types-dict

• Python's version of Perl's hashes:

D2 = { 'spam': 2, 'eggs': 3 }   # 2 string keys and 2 int values
D3 = { 1: 10, 2: 14 }           # 2 int keys and 2 int values

• Index by key

D2[ 'eggs' ] += 1   # increment number of eggs by 1
print(D2['eggs'])

D3[ 1 ] += 1        # increment value at key=1 by 1
print(D3[ 1 ])

D3[ 5 ] = 4         # new key:value pair added to D3

• get keys

print(D2.keys())    # outputs an keys object containing the list: ['eggs', 'spam']

• get values

print(D2.values())  # outputs a values object containing a list: [2, 4]

• get key:value pairs

print(D2.items())

• get number of key:value pairs

print(len(D2))

• get value or a default value if nonexistent

print(D2.get('bacon'))       # outputs 'None' since nonexistent key
print(D2.get('bacon', -1))   # outputs -1 since nonexistent key

• See if a key exists

D3.has_key( 3 )   # (Python 2 only) returns False since the key 3 does not exist in D3
3 in D3           # Check if key 3 is in dictionary D3

• “merge” two dictionaries

D3 = { 'toast':4, 'muffin':5, 'spam':1000 }
D2.update( D3 )   # 'spam' value from D3 overwrites 'spam' value in D2

• Using Python's for structure

table = {'Python': 'Guido van Rossum',
	'Perl':   'Larry Wall',
	'Tcl':    'John Ousterhout' }

for lang in table.keys():
   print (lang, '\t', table[ lang ])

• Note: Python dictionaries have been insertion ordered since version 3.6.
  • Prior to that, dictionaries were unordered like Perl hashes.

• See https://docs.python.org/3/library/io.html#module-io

• Reading

x = open("input.txt", "r")             # open file for input and assign to object x

while 1:
   y = x.readline()                    # read next line of x
   if (not y):
	  break

for eachline in x.readlines():         # x.readlines() creates [list] of lines in x
   y = eachline

y = x.read()                           # read entire file into a string
y = x.readline()                       # read next line
y = x.readlines()                      # read file into list of strings

x.close()

• The read(), readline() and readlines() methods also apply to Python stdin.

• Writing

file = open("output.txt", "w") 

file.write("Hello World") 
file.write("This is our new text file.")

file.close() 

• The with statement for resource management (including files)
  • ensures that files are closed properly even if exceptions occur
    • All file operations must occur within the with block.

with open('input.txt', 'r') as input_file, open('output.txt', 'w') as output_file:
    input = input_file.read()
    # ...
    # process input into output
    # ...
    output_file.write(output)

• sys.argv returns the List of command line arguments, including the script name

import sys

x = sys.argv   # List of command line arguments
print(x)

print(x[ 1 ])


output:
$ python cmdargs.py 1 2 3
['cmdargs.py', '1', '2', '3']   
1

# Note: All elements in sys.argv are strings, even args that contain only digits

def factorial(n):                      # define a function
   if (n == 1):
	  return (1)
   else:
	  return (n * factorial(n-1))      # recursion

x = factorial(5)                       # call a function

• scope issues
  • to access global scope vars, use global varname

def accessglobal():
   global glib                         # access a global scope var
   glib = 100

glib = 0
accessglobal()
print("glib is %i after call to accessglobal()" % glib)

• See http://docs.python.org/library/re.html

# python_re.py

import re

str = 'I am a string'

# store RE in a RE pattern object:
regex = re.compile(r'string$')

#
# matching:
#
# finds first instance of regex using module's (re) search function:
if re.search(r'string$', str):
	print("str ends with 'string'")
	
# finds first instance of regex using RE pattern object:
if regex.search(str):
	print("str ends with 'string'")

# finds all instances of regex (returns list of matching substrings)
if regex.findall(str): # findall being used as an if condition
	print("str ends with 'string'")

print(re.findall(r'[AEIOUaeiou]', str)) # findall being used normally


#
# search/replace:
#
# replaces all instances of ' a ' with ' another ' in str;
# so the default is global search and replace:
str = re.sub(r' a ', ' another ', str)
print("str is now: " + str)

# Added "1" to replace only the first instance of ' a ' with ' another ' in str:
str = re.sub(r' a ', ' another ', str, 1)


#
# split:
#
pattern = re.compile(r'\W+')  # \W matches any non-alphanumeric character;

print(pattern.split('This is a test, short and sweet, of split().'))
# output: ['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', '']

print(pattern.split('This is a test, short and sweet, of split().', 3))
# At most 3 splits are performed, and the rest of the string is left unsplit.
# output: ['This', 'is', 'a', 'test, short and sweet, of split().']

# Can use re module split method without using a pattern object:
print(re.split(r'\W+', 'This is a test, short and sweet, of split().'))


#
# regexp groups (back references in Perl):
#
mystring = "abcdefg"
mygroups = re.search( '(a.)(c.)(e.)(g)', mystring )
# mygroups is a "match object" (See http://docs.python.org/library/re.html#match-objects)

print(mygroups.group( 0 ))   # group( 0 ) is the whole of mystring, i.e., abcdefg
print(mygroups.group( 1 ))   # group( 1 ) is ab
print(mygroups.group( 2 ))
print(mygroups.group( 3 ))
print(mygroups.group( 4 ))   # group( 4 ) is g

# The match object .group() method returns regular strings:
mynewstring = mygroups.group(4) + mygroups.group(3) + mygroups.group(2)

print(mynewstring)

• The Python interpreter imports modules from the Python library search path, defined in sys.path.

import sys
print(sys.path)

• Python looks in several places when you try to import a module
• sys.path is a standard Python list, which can be modified with standard list methods.
• Python modules generally end in .py or .pyc (if compiled)
  • Some are built into Python, such as the sys module
  • Not all modules are written in Python; some are written in C for greater speed.
  • One can add a new path, e.g. /export/home/hawkdom2/jchung/lib/python/test, to Python's search path at runtime by appending the path name to sys.path
    • Python will then also look in that path for modules, whenever you try to import a module.
    • The effect lasts as long as Python is running.

• Altering sys.path in a Python program

import sys

# Add my own module path to sys.path;
# possible because sys.path is a standard Python list
sys.path.append( '/export/home/hawkdom2/jchung/lib/python/test' )

print(sys.path)

• function defs in user-defined modules become methods of those modules

# Import /export/home/hawkdom2/jchung/lib/python/test/mymod.py
# where the mymod.py contains a simple function def:
#
#    def greeting():
#       print "This is being printed in jchung's mymod module."
#
import mymod

# Calls the greeting() method in the imported mymod module
mymod.greeting()

• When the Python interpreter executes import mymod, the module file mymod.py is automatically byte compiled.
  • A file named mymod.pyc or a similar name may be found in a directory called __pycache__.
  • Python modules can also be manually byte-compiled using the python3 -m py_compile modname.py.
  • Byte-compiled modules load faster than non-compiled modules; they don't execute any faster.
    • They may also be useful for code obfuscation (information hiding).
• sys.modules is a run-time dictionary that contains all the modules that are loaded by a Python program.

• The import statement
  • import identifies an external file to be loaded.
    • The name that is imported also becomes a variable in the program, a reference to the module object:

import module1                   # Get module as a whole
module1.printer('Hello world!')  # Access module member names using module name.member_name

• The from..import statement
  • from..import imports member names from a module, so there's no need to qualify these member names:

from module1 import printer      # Import member name from module1
printer('Hello world!')          # Access module member names directly.

• The from..import * statement
  • Special form of from that imports all module member names:

from module1 import *          # Import 'printer()' and any other names from module1
printer('Hello world!')

• Packages are collections of modules, typically in a file system directory hierarchy.
  • Example: /usr/lib/python3.9/email
    • The package here is email
    • email is a subdirectory within /usr/lib/python3.9, which is inside Python3.9's default sys.path
    • Inside email is a collection of modules (.py files and their .pyc byte-compiled counterparts) that provide Python email handling functions.
    • Also inside email is a special module __init__.py that identifies email as a package
      • Without __init__.py, attempts to import email.something will fail.
    • Package modules and submodules are accessed using “.” notation
      • import email.mime.audio imports the audio module from /usr/lib/python3.9/email/mime/audio.py
      • from email import message imports the message module from email
      • from email import * imports all module member names from email
        • from … import * requires the __all__ list to be defined in __init__.py to work

• See http://docs.python.org/tutorial/datastructures.html#list-comprehensions
• List comprehension: Syntactic construct available in some programming languages for creating a list based on existing lists
• Each python list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses.
  • The result is a list resulting from evaluating the expression in the context of the for and if clauses which follow it.

• Examples:

S = [2*x for x in range(101) if x**2 > 3]
print S

Output: [4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50,
		 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100,
		 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140,
		 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180,
		 182, 184, 186, 188, 190, 192, 194, 196, 198, 200]




freshfruit = ['  banana', '  loganberry ', 'passion fruit  ']
print [weapon.strip() for weapon in freshfruit]

Output: ['banana', 'loganberry', 'passion fruit']




# multiple lists in a list comprehension:
vec1 = [2, 4, 6]
vec2 = [4, 3, -9]
print [x*y for x in vec1 for y in vec2]

Output: [8, 6, -18, 16, 12, -36, 24, 18, -54]




# If the expression would evaluate to a tuple, it must be parenthesized:
vec = [2, 4, 6]
print [(x, x**2) for x in vec]

Output: [(2, 4), (4, 16), (6, 36)]




# The dict() constructor builds dictionaries directly from lists of 
# key-value pairs stored as tuples.
# When the pairs form a pattern, list comprehensions can compactly
# specify the key-value list.
print dict([(x, x**2) for x in (2, 4, 6)]) # dict applied to a list comprehension

Output: {2: 4, 4: 16, 6: 36}