The With Statement in Python

We use the with statement in Python to create a context. Whatever statements we execute inside the context created using the with statement does not affect the outer environment. The with statement simplifies exception handling by encapsulating common preparation and cleanup tasks. Hence, you can get better syntax and exception handling by using the “With” statement in Python.

The open() Function in Python

We use the open() function to open a file in Python. It takes the filename as its first input argument and the python literals “r”, “w”, “r+”, etc as its second input argument to specify the mode in which the file is opened. After execution, it returns a file pointer.

We can use the file pointer to read from the file or write data to the file. For example, consider that we have the following text file.

In the above file, we have five lines of text, let us open the file using the open() function and read the file contents. For this, we will write the following code in open_example.py file and execute it.

file = open("romeo.txt","r")
data = file.read()
print("The file contents are:")
print (data)
file.close()  # It's important to close the file when you're done with it

Once we execute the above code, it prints the output of the romeo.txt file as shown below.

When we use the open() statement, we also need to close the file using the close() method. Otherwise, the file may get corrupted. Also, if we write any data to the file and don’t close it, the contents written to the file will not be saved. Hence, it is really important to close the file.

However, we might forget to use the close() statement to close the file. In such a case, we can lose data or even corrupt the file. To avoid this, we can create a context using the with statement and open a file inside it using the open() function. Let us discuss how to use the with statement and the open() function together in Python.

With Statement Usage with Open() Function in Python

As discussed earlier, whatever we do inside the context created using the with statement stays inside the context. Hence, we can also use the with statement with the open() function in Python to open a file, perform operations on it, and close it. The syntax for using the with statement is as follows.

with open(filename, mode) as myFile:
    //read from the file
    // write to the file.

In the above syntax, the file is opened, and the file pointer is assigned to the myFile variable. Then, we can perform read and write operations on the file using the myFile object. When all the statements inside the with block are executed, the file is automatically closed.

For example, we can rewrite the code used in the previous example using the with statement and the open() function as shown below.

with open("romeo.txt","r") as file:
    data=file.read()
    print("The file contents are:")
    print(data)

The above code works in the same manner as the previous code without the need to use the close() method. Hence, combining the with statement and the open() function in Python helps us read and write to the file without being worried about closing it.

In the above code, you can observe that we have opened the romeo.txt file using the with open statement. The statement returns a file pointer that is assigned to the variable “file”. Now, we can perform any operation on the file object inside the context of the with statement. Once all the statements are executed and the execution reaches the end of the with context block, the file is automatically closed by the Python interpreter.

Also, if the program runs into any exception inside the with block, the with open context in Python closes the file before terminating the program. In this way, the data inside the file remains safe even if the program is terminated abruptly. Notice, that we didn’t have to write “file.close()”. That will automatically be called.

Conclusion

In this article, we have discussed how you can use the with open statement instead of the open()
method to open a file in Python. To learn more about Python programming, you can read this article on string manipulation in Python. You might also like this article on Python if else shorthand.

The post With Open Statement in Python appeared first on PythonForBeginners.com.

When you’re working with Python, you don’t need to import a library in order to read and write to a file. It’s handled natively in the language, albeit in a unique manner. Below, we outline the simple steps to read and write to a file in Python.

The first thing you’ll need to do is use the built-in python open file function to get a file object.

The open function opens a file. It’s simple. This is the first step in reading and writing files in python.

When you use the open function, it returns something called a file object. File objects contain methods and attributes that can be used to collect information about the file you opened. They can also be used to manipulate said file.

For example, the mode attribute of a file object tells you which mode a file was opened in. And the name attribute tells you the name of the file.

You must understand that a file and file object are two wholly separate – yet related – things.

File Types

What you may know as a file is slightly different in Python.

In Windows, for example, a file can be any item manipulated, edited or created by the user/OS. That means files can be images, text documents, executables, and excel file and much more. Most files are organized by keeping them in individual folders.

A file In Python is categorized as either text or binary, and the difference between the two file types is important.

Text files are structured as a sequence of lines, where each line includes a sequence of characters. This is what you know as code or syntax.

Each line is terminated with a special character, called the EOL or End of Line character. There are several types, but the most common is the comma {,} or newline character. It ends the current line and tells the interpreter a new one has begun.

A backslash character can also be used, and it tells the interpreter that the next character – following the slash – should be treated as a new line. This character is useful when you don’t want to start a new line in the text itself but in the code.

A binary file is any type of file that is not a text file. Because of their nature, binary files can only be processed by an application that know or understand the file’s structure. In other words, they must be applications that can read and interpret binary.

Reading Files in Python

In Python, files are read using the open() method. This is one of Python’s built-in methods, made for opening files. 

The open() function takes two arguments: a filename and a file opening mode. The filename points to the path of the file on your computer, while the file opening mode is used to tell the open() function how we plan to interact with the file.

By default, the file opening mode is set to read-only, meaning we’ll only have permission to open and examine the contents of the file.

On my computer is a folder called PythonForBeginners. In that folder are three files. One is a text file named emily_dickinson.txt, and the other two are python files: read.py and write.py.

The text file contains the following poem, written by poet Emily Dickinson. Perhaps we are working on a poetry program and have our poems stored as files on the computer.

Success is counted sweetest
By those who ne’er succeed.
To comprehend a nectar
Requires sorest need.

Not one of all the Purple Host
Who took the Flag today
Can tell the definition
So clear of Victory

As he defeated, dying
On whose forbidden ear
The distant strains of triumph
Burst agonized and clear.

Before we can do anything with the contents of the poem file, we’ll need to tell Python to open it. The file read.py, contains all the python code necessary to read the poem.

Any text editor can be used to write the code. I’m using the Atom code editor, which is my editor of choice for working in python.

# read.py
# loading a file with open()
myfile = open(“emily_dickinson.txt”)

# reading each line of the file and printing to the console
for line in myfile:
	print(line)

I’ve used Python comments to explain each step in the code. Follow this link to learn more about what a Python comment is.

The example above illustrates how using a simple loop in Python can read the contents of a file.

When it comes to reading files, Python takes care of the heaving lifting behind the scenes. Run the script by navigating to the file using the Command Prompt — or Terminal  — and typing ‘python’ followed by the name of the file.

Windows Users: Before you can use the python keyword in your Command Prompt, you’ll need to set up the environment variables. This should have happened automatically when you installed Python, but in case it didn’t, you may need to do it manually. 

>python read.py

Data provided by the open() method is usually stored in a new variable. In this example, the contents of the poem are stored in the variable “myfile.”

Once the file is created, we can use a for loop to read every line in the file and print its contents to the command line.

This is a very simple example of how to open a file in Python, but student’s should be aware that the open() method is quite powerful. For some projects it will be the only thing needed to read and write files with Python.

Writing Files in Python

Before we can write to a file in Python, it must first be opened in a different file opening mode. We can do this by supplying the open() method with a special argument.

In Python, write to file using the open() method. You’ll need to pass both a filename and a special character that tells Python we intend to write to the file.

Add the following code to write.py. We’ll tell Python to look for a file named “sample.txt” and overwrite its contents with a new message.

# open the file in write mode
myfile = open(“sample.txt”,’w’)

myfile.write(“Hello from Python!”)

Passing ‘w’ to the open() method tells Python to open the file in write mode. In this mode, any data already in the file is lost when the new data is written.

If the file doesn’t exist, Python will create a new file. In this case, a new file named “sample.txt” will be created when the program runs.

Run the program using the Command Prompt:

>python write.py

Python can also write multiple lines to a file. The easiest way to do this is with the writelines() method.

# open the file in write mode
myfile = open(“sample.txt”,’w’)

myfile.writelines(“Hello World!”,”We’re learning Python!”)

# close the file
myfile.close()

We can also write multiple lines to a file using special characters:

# open the file in write mode
myfile = open("poem.txt", 'w')

line1 = "Roses are red.\n"
line2 = "Violets are blue.\n"
line3 = "Python is great.\n"
line4 = "And so are you.\n"

myfile.write(line1 + line2 + line3 + line4)

Using string concatenation makes it possible for Python to save text data in a variety of ways.

However, if we wanted to avoid overwriting the data in a file, and instead append it or change it, we’d have to open the file using another file opening mode.

File Opening Modes

By default, Python will open the file in read-only mode. If we want to do anything other than just read a file, we’ll need to manually tell Python what we intend to do with it.

• ‘r’ – Read Mode: This is the default mode for open(). The file is opened and a pointer is positioned at the beginning of the file’s content.
• ‘w’ – Write Mode: Using this mode will overwrite any existing content in a file. If the given file does not exist, a new one will be created. 
• ‘r+’ – Read/Write Mode: Use this mode if you need to simultaneously read and write to a file.
• ‘a’ – Append Mode: With this mode the user can append the data without overwriting any already existing data in the file. 
• ‘a+’ – Append and Read Mode: In this mode you can read and append the data without overwriting the original file.
• ‘x’ – Exclusive Creating Mode: This mode is for the sole purpose of creating new files. Use this mode if you know the file to be written doesn’t exist beforehand.

Note: These examples assume the user is working with text file types. If the intention is to read or write to a binary file type, an additional argument must be passed to the open() method: the ‘b’ character. 

# binary files need a special argument: ‘b’
binary_file = open(“song_data.mp3”,’rb’)
song_data = binary_file.read()

# close the file
binary_file.close()

Closing Files with Python

After opening a file in Python, it’s important to close it after you’re done with it. Closing a file ensures that the program can no longer access its contents.

Close a file with the close() method.

# open a file
myfile = open(“poem.txt”)
# an array to store the contents of the file
lines = []
For line in myfile:
	lines.append(line)
	
# close the file
myfile.close()

For line in liens:
	print(line)

Opening Other File Types

The open() method can read and write many different file types. We’ve seen how to open binary files and text files. Python can also open images, allowing you to view and edit their pixel data.

Before Python can open an image file, the Pillow library (Python Imaging Library) must be installed. It’s easiest to install this module using pip. 

pip install Pillow

With Pillow installed, Python can open image files and read their contents.

From PIL import Image

# tell Pillow to open the image file
img = Image.open(“your_image_file.jpg”)
img.show()
img.close()

The Pillow library includes powerful tools for editing images. This has made it one of the most popular Python libraries.

With Statement

You can also work with file objects using the with statement. It is designed to provide much cleaner syntax and exceptions handling when you are working with code. That explains why it’s good practice to use the with statement where applicable.

One bonus of using this method is that any files opened will be closed automatically after you are done. This leaves less to worry about during cleanup.

To use the with statement to open a file:

with open(“filename”) as file: 

Now that you understand how to call this statement, let’s take a look at a few examples.

with open(“poem.txt”) as file:  
data = file.read() 
do something with data 

You can also call upon other methods while using this statement. For instance, you can do something like loop over a file object:

with open(“poem.txt”) as f: 
for line in f: 
print line, 

You’ll also notice that in the above example we didn’t use the “file.close()” method because the with statement will automatically call that for us upon execution. It really makes things a lot easier, doesn’t it?

Splitting Lines in a Text File

As a final example, let’s explore a unique function that allows you to split the lines taken from a text file. What this is designed to do, is split the string contained in variable data whenever the interpreter encounters a space character.

But just because we are going to use it to split lines after a space character, doesn’t mean that’s the only way. You can actually split your text using any character you wish – such as a colon, for instance.

The code to do this (also using a with statement) is:

with open(“hello.text”, “r”) as f:
data = f.readlines()
 
for line in data:
words = line.split()
print words

If you wanted to use a colon instead of a space to split your text, you would simply change line.split() to line.split(“:”).

The output for this will be:

[“hello”, “world”, “how”, “are”, “you”, “today?”]
[“today”, “is”, “Saturday”]

The reason the words are presented in this manner is because they are stored – and returned – as an array. Be sure to remember this when working with the split function.

Conclusion

Reading and writing files in Python involves an understanding of the open() method. By taking advantage of this method’s versatility, it’s possible to read, write, and create files in Python.

Files Python can either be text files or binary files. It’s also possible to open and edit image data using the Pillow module.

Once a file’s data is loaded in Python, there’s virtually no end to what can be done with it. Programmers often work with a large number of files, using programs to generate them automatically.

As with any lesson, there is only so much that can be covered in the space provided. Hopefully you’ve learned enough to get started reading and writing files in Python.

More Reading

Official Python Documentation – Reading and Writing Files

Python File Handling Cheat Sheet

The post Reading and Writing Files in Python appeared first on PythonForBeginners.com.

For a long time I have been using os.system() when dealing with system
administration tasks in Python.

The main reason for that, was that I thought that was the simplest way of
running Linux commands.

In the official python documentation we can read that subprocess should be used
for accessing system commands.

The subprocess module allows us to spawn processes, connect to their
input/output/error pipes, and obtain their return codes.

Subprocess intends to replace several other, older modules and functions,
like: os.system, os.spawn*, os.popen*, popen2.* commands.

Let’s start looking into the different functions of subprocess.

subprocess.call()

Run the command described by “args”.

We can run the command line with the arguments passed as a list of strings
(example 1) or by setting the shell argument to a True value (example 2)

Note, the default value of the shell argument is False.

Let’s look at two examples where we show the summary of disk usage using
subprocess.call()

subprocess.call(['df', '-h'])

This time we set the shell argument to True

subprocess.call('du -hs $HOME', shell=True)

Note, the official Python documentation states a warning about using the
shell=True argument.

“Invoking the system shell with shell=True can be a security hazard if combined
with untrusted input” [source]

Now, let’s move on and look at the Input / Output.

Input and Output

With subprocess you can suppress the output, which is very handy when you
want to run a system call but are not interested about the standard output.

It also gives you a way to cleanly integrate shell commands into your scripts
while managing input/output in a standard way.

Return Codes

You can use subprocess.call return codes to determine the success of the command.

Every process will return an exit code and you can do something with your script
based on that code.

If the return code is anything else than zero, it means that an error occurred.

If you want to do system administration in Python, I recommend reading
Python for Unix and Linux System Administration

stdin, stdout and stderr

One of the trickiest part I had with subprocess was how to work with pipes
and to pipe commands together.

PIPE indicates that a new pipe to the child should be created.

The default setting is “None”, which means that no redirection will occur.

The standard error (or stderr) can be STDOUT, which indicates that the stderr
data from the child process should be captured into the same file handle as
for stdout.

subprocess.Popen()

The underlying process creation and management in the subprocess module is
handled by the Popen class. subprocess.popen is replacing os.popen.

Let’s get started with some real examples.

subprocess.Popen takes a list of arguments

import subprocess

p = subprocess.Popen(["echo", "hello world"], stdout=subprocess.PIPE)

print p.communicate()

>>>('hello world', None)

Note, even though you could have used “shell=True”, it is not the recommended
way of doing it.

If you know that you will only work with specific subprocess functions,
such as Popen and PIPE, then it is enough to only import those.

from subprocess import Popen, PIPE

p1 = Popen(["dmesg"], stdout=PIPE)

print p1.communicate()

Popen.communicate()

The communicate() method returns a tuple (stdoutdata, stderrdata).

Popen.communicate() interacts with process: Send data to stdin.

Read data from stdout and stderr, until end-of-file is reached.

Wait for process to terminate.

The optional input argument should be a string to be sent to the
child process, or None, if no data should be sent to the child.

Basically, when you use communicate() it means that you want to
execute the command

Ping program using subprocess

In the “More Reading” section below, you can find links to read more
about the subprocess module, but also examples.

Let’s write our own ping program where we first ask the user for input,
and then perform the ping request to that host.

# Import the module
import subprocess

# Ask the user for input
host = raw_input("Enter a host to ping: ")	

# Set up the echo command and direct the output to a pipe
p1 = subprocess.Popen(['ping', '-c 2', host], stdout=subprocess.PIPE)

# Run the command
output = p1.communicate()[0]

print output

Let’s show one more example. This time we use the host command.

target = raw_input("Enter an IP or Host to ping:
")

host = subprocess.Popen(['host', target], stdout = subprocess.PIPE).communicate()[0]

print host

I recommend that you read the links below to gain more knowledge about the
subprocess module in Python.

If you have any questions or comments, please use the comment field below.

More Reading

How to sh in Python
http://docs.python.org/2/library/subprocess.html
The ever useful and neat subprocess module
http://pymotw.com/2/subprocess/
http://www.bogotobogo.com/python/python_subprocess_module.php

The post Subprocess and Shell Commands in Python appeared first on PythonForBeginners.com.

The OS module in Python provides a way of using operating system dependent
functionality.

The functions that the OS module provides allows you to interface with the
underlying operating system that Python is running on. (Windows, Mac or
Linux.

You can find important information about your location or about the process.

Before we start, make sure that you have imported the OS module “import os”

OS Functions explained

os.system()	# Executing a shell command

os.stat()	# Get the status of a file

os.environ()    # Get the users environment

os.chdir()   	# Move focus to a different directory

os.getcwd()    	# Returns the current working directory

os.getgid()    	# Return the real group id of the current process

os.getuid()    	# Return the current process’s user id

os.getpid()     # Returns the real process ID of the current process

os.getlogin()   # Return the name of the user logged

os.access()   	# Check read permissions

os.chmod()    	# Change the mode of path to the numeric mode

os.chown()   	# Change the owner and group id

os.umask(mask)  # Set the current numeric umask

os.getsize()   	# Get the size of a file

os.environ()    # Get the users environment

os.uname()   	# Return information about the current operating system

os.chroot(path) # Change the root directory of the current process to path

os.listdir(path)# List of the entries in the directory given by path

os.getloadavg() # Show queue averaged over the last 1, 5, and 15 minutes

os.path.exists()# Check if a path exists

os.walk()   	# Print out all directories, sub-directories and files

os.mkdir(path)	# Create a directory named path with numeric mode mode

os.remove(path)	# Remove (delete) the file path

os.rmdir(path)  # Remove (delete) the directory path

os.makedirs(path)# Recursive directory creation function

os.removedirs(path) # Remove directories recursively

os.rename(src, dst) # Rename the file or directory src to dst

OS Functions Examples

Let’s get started to see how we can use these OS functions.

#Get current working directory with os.getcwd()
print os.getcwd()

#Get the status of a file with os.stat()
print "Getting the status of: ", os.stat('/usr/bin/python')

#Execute a shell command with os.system()
os.system('ls -l')

#Return the current process id with os.getpid()
print os.getpid()
os.chmod(path, mode)

#Change the owner and group id of path to the numeric uid and gid with os.chown()
os.chown(path, uid, gid)

#Processes in the system run queue averaged over the last 1, 5, and 15 minutes
print os.getloadavg()

#Check if a path exists with os.path.exists()
if os.path.exists("file.txt"):

#Create a new directory named 'new_directory' if it doesn't exist already"
os.path.exists("new_directory") or os.mkdir("new_directory")

#Check if the path is a directory or a file with os.path.isdir() & os.path.isfile()
path = "/tmp"
if os.path.isdir(path): print "That's a directory"
if os.path.isfile(path): print "That's a file"

#Create a directory with os.makedir()
print os.mkdir('new_directory', 0666)

#Recursive create directories with os.makedirs()
os.makedirs('dir_a/dir_b/dir_c')

#Remove a directory with os.rmdir()
print os.rmdir('directory')

#Recursively remove empty directories with os.rmdirs()
os.removedirs('dir_a/dir_b/dir_c')

#Rename a file with os.rename()
print os.rename('/path/to/old/file', '/path/to/new/file')

#Rename a file with shutil.move()
print shutil.move('/path/to/old/file', '/path/to/new/file')

#Rename a file with shutil.copy()
print shutil.copy('/path/to/old/file', '/path/to/new/file')

#Get the users home directory
print os.path.expanduser('~')

#Check read permissions with os.access()
path = '/tmp/file.txt'
print os.access(path, os.R_OK)

#Get the users environment with os.environmen()
home =  os.environ['HOME']
print home

#Move focus to a different directory with os.chdir()
print os.chdir('/tmp')

#Print out all directories, sub-directories and files with os.walk()
for root, dirs, files in os.walk("/tmp"):
    print root
    print dirs
    print files

#Get the last time a directory was accessed with os.path.getatime()
os.path.getatime('/tmp')

#Get the last time a directory was modified with os.path.getmtime()
os.path.getmtime('/tmp')

#Get the user ID with os.getuid()
if os.getuid() != 0: print "you are not root"

#Get the group ID with os.getgid()
print os.getgid()

#Return the name of the user logged in with os.getlogin()
print os.getlogin()

#Returns a list of all files in a directory with os.listdir()
for filename in os.listdir("/tmp"):
    print "This is inside /tmp", filename

#Get the size of a file with os.path.getsize()
path.getsize("/tmp/file.txt")

Using Python in your daily work is a good way to automate system administration tasks, when you feel that your shell scripts are to limited.

The post Python System Administration appeared first on PythonForBeginners.com.

I earlier wrote a post on "How to use Fabric in Python", which can be found here. 

I received a lot of responses from that article, so I decided to write another
post about Fabric. This time I would like to focus on how to use it in a
development environment. 

What is Fabric?

Just to recap what Fabric is

Fabric is a Python (2.5 or higher) library and command-line tool for streamlining
the use of SSH for application deployment or systems administration tasks.

It provides a basic suite of operations for executing local or remote shell
commands (normally or via sudo) and uploading/downloading files, as well as
auxiliary functionality such as prompting the running user for input, or aborting
execution.

Why use Fabric in the development environment?

So, we can use Fabric for streamlining the use of SSH for application deployment
or systems administration tasks.

In a development environment, where you have multiple servers with multiple people
pushing the code multiple times per day, this can be very useful.

If you would be the only person in the project who worked on a single server,
a git pull (A git pull is what you would do to bring your repository up to date
with a remote repository) and start working with it.

Often, you are not the only developer in a projet and there is where Fabric comes
into the picture. To avoid logging into multiple servers and running remote
commands, we can use Fabric to automate this whole process.

It can configure the system, execute commands on local/remote server, deploy your
application, do rollbacks etc. A common practice when developing is to use Git to
deploy and Fabric to automate it.

Fabric requirements

An SSH server like OpenSSH needs to be installed on your server and an SSH client
needs to be installed on your client.

Fabric relies on the SSH Model, you can use SSH Keys but you can also control
access to root via sudoers. 

The user on the server does not need to be added to "~/.ssh/authorized_keys",
but if it is you don't have to type the password every time you want to execute
a command.

If you ever have to disable the access to a user, just turn off their SSH account.

There is no transfer of code, so you only need to have ssh running on the remote
machine and have some sort of shell (bash is assumed by default).

Fabric Installation

First off, we will have to install it, that can be done through pip

pip install fabric

Fabric and Fabfile.py

The installation added a Python script called "fab" to a directory in your path. 

This is the command Fabric will use when it logs into one or more servers.

The fabile.py (see below) will be executed by the fab command.

All fabric does at its core is execute commands locally and remotely.

Fabric just provides a nice pythonic way of doing it.

The fabile is what Fabric uses to to exectue task and we need to specify which
shell commands we want Fabric to execute. In short, a fabfile is what controls
what Fabric executes. 

The commands that we put into the fabfile will be  executed on one or more hosts. 

Every fabric scripts need to be in a file called fabfile.py. These hosts can be
defined either in the fabfile or on the command line.

All the functions defined in this file will show up as fab subcommands. 

Fabric functions

Fabric provides a set of commands in fabric.api that are simple but powerful,
below are some of them

local	# execute a local command)
run	# execute a remote command on all specific hosts, user-level permissions)
sudo	# sudo a command on the remote server)
cd	# changes the directory on the serverside, use with the "with" statement)
put  	# uploads a local file to a remote server)
get 	# download a file from the remote server)
prompt	# prompt user with text and return the input (like raw_input))
reboot	# reboot the remote system, disconnect, and wait for wait seconds)

One of the most common Fabric functions is run()

It executes whatever command you put in as a parameter on the server. 

Below you can see an example.

run("uptime")

This will run the uptime command on any server that we specify in the fabfile.py.

The server(s) are set using env.hosts:

env.hosts = ['superuser@host1.example.com']

This tells the script to use the username 'superuser' with the server address
'host1.example.com'

Basically, it's just like ssh'ing into a box and running the commands you've put
into run() and sudo(), but Fabric allows you to use several more calls. 

Setting up the development environment

Before we start with creating Fabric functions, I want to show an example of how
a directory structure may look like:

$ mkdir /var/www/yourapplication
$ cd /var/www/yourapplication
$ virtualenv --distribute env

Before we can do anything, we first need to create a fabile.py.

If you are located in the same directory as "fabfile.py" you can go "fab --list"
to see a list of available commands and then "fab [COMMAND_NAME]" to execute a
command.

Now when everything is setup and we have an fabfile.py, let's put something in it.

Let's first start defining roles and then specify what action to do.

from fabric.api import *
 
# Define sets of servers as roles
env.roledefs = {
    'www': ['www1', 'www2', 'www3', 'www4', 'www5'],
    'databases': ['db1', 'db2']
}
 
# Set the user to use for ssh
env.user = 'fabuser'
 
# Restrict the function to the 'www' role (that we created above)
@roles('www')
def uptime():
    run('uptime')

If we now want to see the uptime on all our web-servers, all we have to do is run: 
$ fab -R www

# To run uptime on both roles (www and databases);
$ fab uptime

Any function you write in your fab script can be assigned to one or more roles. 

You can also include a single server in multiple roles.

So this is nice, Fabric makes it really easy to run commands across sets of
machines.

Deployment using Fabric

Let's see now how we can deploy something with Fabric. Often, we want to create
more than one function, depending on how your environment looks like (live,
staging, dev)

env.roledefs = {
    'hostsDev': ['dev1', 'dev2'],
    'hostsStaging':[ 'stg1', 'stg2'],
}
 
@roles('hostsDev')

def deploy_dev():
    
  # Specify the path to where your codebase is located
  path = "/home/fabuser/codebase"
 
  # Get the code from Github
  run ("git clone https://www.github.com/user/repo.git")

@roles('hostsStaging')

def deploy_staging():
    # do something

If you now want to do a git clone to your development servers, all you need to do
is use the fab command:
$ fab deploy_dev

That's how you can create a deploy function using Fabric. 

Virtualenv and Requirements

Now we have cloned our code from the Github repository. That is nice, but we can
do more, we can also install the requirements into a virtual environment. 

For those of you that don't know what "requirements" is; 
It's basically a file called requirements.txt which contain a list of packages
to install.

Instead of running something like pip install MyApp and getting whatever libraries
come along, you can create a requirements file that includes all dependencies.

Example of a requirements.txt file

BeautifulSoup==3.2.0
python-dateutil==1.4.1
django==1.3.0
django-debug-toolbar==0.8.5
django-tagging==0.3
Markdown==2.0.1
...
...

To install all packages that are in that file, simple type

pip install -r requirements.txt

A virtual environment can be created with /bin/virtualenv 'app_name' and
activated by typing source /bin/activate

Use Fabric to install packages from requirements.txt

Having that information, let's create a Fabric script to install it.

# Import the module
from fabric.api import *
 
# Specify the user and host
env.hosts = ['fabuser@host1.example.com']
 
def deploy():
     with cd("/home/fabuser/codebase/"):

          run("git clone https://github.com/user/repo.git")
 
     run("source /home/fabuser/codebase/venv/bin/activate && pip install -r /path/to/requirements.txt")

That small script will activate the virtual environment and install all packages
specified in the requirements.txt file. 

Sample output

Downloading/unpacking BeautifulSoup==3.2.0 ....
  Downloading BeautifulSoup-3.2.0.tar.gz
  Running setup.py egg_info for package BeautifulSoup
...
...
...

Storing complete log in /home/fabuser/.pip/pip.log

Use Fabric to restart Apache servers

We can use Fabric for many other things, we can also create a Fabric script for
restarting our Apache servers. 

That would look something like this

from fabric.api import *

env.hosts = ['superuser@host1.example.com:22']

def restart_apache2():
    sudo('service apache2 restart')

To restart apache2 on host 'host1.example.com', the only thing we have to do is

$fab restart_apache2

Use Fabric to send emails

We aren't limited to hosts in "roledefs", we could also for example put in email
addresses.

from fabric.api import env, run

# Define sets of email addresses as role

env.roledefs = {
    'test': ['localhost'],
    'dev': ['user@dev.example.com'],
    'staging': ['user@staging.example.com'],
    'production': ['user@production.example.com']
} 

def deploy():
    run('echo test')

To run it a specific role use the -R switch

$ fab -R test deploy

[localhost] Executing task 'deploy'

Further reading

http://www.virtualenv.org/en/latest/
http://www.pip-installer.org/en/latest/requirements.html
http://docs.fabfile.org/en/1.6/
Stackoverflow

The post How to use Fabric in a development environment appeared first on PythonForBeginners.com.

Fabric is a Python library and command-line tool for streamlining the use of SSH
for application deployment or systems administration tasks. 

Typical usage involves creating a Python module containing one or more functions,
then executing them via the fab command-line tool.

You can execute shell commands over SSH, so you only need to have SSH running on
the remote machine. It interact with the remote machines that you specify as if
they were local. 

Installation

Fabric requires Python version 2.5 or 2.6 (Fabric has not yet been tested on
Python 3.x)

The most common ways of installing Fabric is via, pip, easy_install or via
the operating system's package manager:

pip install fabric

sudo easy_install fabric

sudo apt-get install fabric	# (the package is typically called fabric or python-fabric.)

Please read the official documentation for more information (dependancies etc..) 

Fabric Usage

On their website they write:

"it provides a basic suite of operations for executing local or remote shell
commands (normally or via sudo) and uploading/downloading files, as well as
auxiliary functionality such as prompting the running user for input, or
aborting execution"

Having that information, let's move on.

The installation process added a Python script called fab to a directory in
your path. 

This is the script which will be used to make everything happen with Fabric. 

Just running fab from the command-line won't do much at all.

In order to do anything interesting, we'll need to create our first fabfile. 

Before we do that I would like to show some of the functions in Fabric. 

Fabric functions

Fabric provides a set of commands in fabric.api that are simple but powerful.

With Fabric, you can use simple Fabric calls like

local	# execute a local command)
run	# execute a remote command on all specific hosts, user-level permissions)
sudo	# sudo a command on the remote server)
put	# copy over a local file to a remote destination)
get	# download a file from the remote server)
prompt	# prompt user with text and return the input (like raw_input))
reboot	# reboot the remote system, disconnect, and wait for wait seconds)

Authentication

Fabric relies on the SSH Model, you can use SSH Keys but you can also control
access to root via sudoers. 

The user on the server does not need to be added to "~/.ssh/authorized_keys",
but if it is you don't have to type the password every time you want to execute
a command.

If you ever have to disable the access to a user, just turn off their SSH account.

The Fabfile

You can load Python modules with Fabric. 

By default, it looks for something named either fabfile or fabfile.py. 

This is the file that Fabric uses to execute tasks. 

Each task is a simple function.

The fabfile should be in the same directory where you run the Fabric tool. 

The fabfile is where all of your functions, roles, configurations, etc. will
be defined. 

It's just a little bit of Python which tells Fabric exactly what it needs to do. 

The "fabfile.py" file only has to be stored on your client. 

An SSH server like OpenSSH needs to be installed on your server and an
SSH client needs to be installed on your client.

Creating a Fabfile

To start just create a blank file called fabfile.py in the directory you’d like
to use the fabric commands from.

You basically write rules that do something and then you (can) specify on which
servers the rules will run on. 

Fabric then logs into one or more servers in turn and executes the shell commands 
defined in "fabfile.py". 

If you are located in the same dir as "fabfile.py" you can go "fab --list"
to see a list of available commands and then "fab [COMMAND_NAME]" to execute a
command.

From https://github.com/fabric/fabric

Below is a small but complete "fabfile" containing a single task:

from fabric.api import run
def host_type():
    run('uname -s')

Once a task is defined, it may be run on one or more servers, like so

$ fab -H localhost,linuxbox host_type

[localhost] run: uname -s
[localhost] out: Darwin
[linuxbox] run: uname -s
[linuxbox] out: Linux

Done.
Disconnecting from localhost... done.
Disconnecting from linuxbox... done.

You can run fab -h for a full list of command line options

In addition to use via the fab tool, Fabric's components may be imported into
other Python code, providing a Pythonic interface to the SSH protocol suite at
a higher level than that provided by e.g. the ssh library, 
(which Fabric itself uses.)

Connecting to remote servers

As you can see above, Fabric will look for the function host_type in the
fabfile.py of the current working directory.

In the next example we can see how the Fabric Api uses an internal env variable
to manage information for connecting to remote servers.

The user is the user name used to login remotely to the servers and the hosts is
a list of hosts to connect to. 

Fabric will use these values to connect remotely for use with the run
and sudo commands. 

It will prompt you for any passwords needed to execute commands or connect to
machines as this user. 

# First we import the Fabric api
from fabric.api import *

# We can then specify host(s) and run the same commands across those systems
env.user = 'username'

env.hosts = ['serverX']

def uptime():
    run("uptime")

This will first load the file ~/fabfile.py

Compiling the file into ~/fabfile.pyc # a standard pythonism

Connect via SSH to the host 'serverX'

Using the username 'username'

Show the output of running the command "uptime" upon that remote host.

Now, when we run the fabfile, we can see that we can connect to the remote server.
$ fab uptime

If you have different usernames on different hosts, you can use:

env.user = 'username'
env.hosts = ['userX@192.168.1.1', 'serverX']

Now userX username would be used on 192.168.1.1 and 'username' would be used
on 'serverX'

Roles

You can define roles in Fabric, and only run actions on servers tied to a
specific role. 

This script will run get_version on hosts members of the role "webservers",
by running first on www1, then www2 etc.

fab -R webservers

from fabric.api import *

# Define sets of servers as roles

env.roledefs = {
    'webservers': ['www1', 'www2', 'www3', 'www4', 'www5'],
    'databases': ['db1', 'db2']
}

# Set the user to use for ssh
env.user = 'fabuser'

# Restrict the function to the 'webservers' role

@roles('webservers')

def get_version():
    run('cat /etc/issue')

# To run get_version on both roles (webservers and databases);

$ fab get_version

@roles ('webservers', 'databases')

    def get_version():

 run('cat /etc/issue')

Any function you write in your fab script can be assigned to one or more roles. 

You can also include a single server in multiple roles.

Copy a directory to a remote machine

I would like to end this introduction of Fabric, by showing an example of
how to copy a directory to a remote machine.

from fabric.api import *

env.hosts = ['userX@serverX']

def copy():
    # make sure the directory is there!
    run('mkdir -p /home/userX/mynewfolder')

    # our local 'localdirectory' (it may contain files or subdirectories)
    put('localdirectory', '/home/userX/mynewfolder')

Conclusion

Fabric can be used for many things, including deploying, restarting servers,
stopping and restarting processes. 

You write the description of what is to be done in Python and Fabric takes
care of executing it on all the machines you specify. 

Once you've used it a bunch of times you'll accumulate many "fab files" that
you can re-use.

Basically, any time we would need to log in to multiple servers to do something,
you can use Fabric. 

It’s simple and powerful and the documentation is really good.

Further reading

http://docs.fabfile.org/en/1.5/index.html
https://gist.github.com/DavidWittman/1886632
http://mattsnider.com/using-fabric-for-painless-scripting/
http://www.clemesha.org/blog/modern-python-hacker-tools-virtualenv-fabric-pip/
http://code.mixpanel.com/2010/09/09/easy-python-deployment-with-fabric-and-git/
http://stackoverflow.com/questions/6308686/understanding-fabric/9894988#9894988

The post How to use Fabric in Python appeared first on PythonForBeginners.com.

It’s a string pattern written in a compact syntax, that allows us to quickly check
whether a given string matches or contains a given pattern.

The power of regular expressions is that they can specify patterns, not just
fixed characters. Many examples in this articles can be found on:
Googles Python Course

Basic patterns

a, X, 9
ordinary characters just match themselves exactly.

. ^ $ * + ? { [ ] | ( )
meta-characters with special meanings (see below)

. (a period)
matches any single character except newline ‘n’

w
matches a “word” character: a letter or digit or underbar [a-zA-Z0-9_].
It only matches a single character not a whole word.

W
matches any non-word character.

w+
matches one or more words / characters

b
boundary between word and non-word

s
matches a single whitespace character, space, newline, return, tab, form

S
matches any non-whitespace character.

t, n, r
tab, newline, return

D
matches anything but a digit

d
matches a decimal digit [0-9]

d{1,5}
matches a digit between 1 and 5 in lengths.

{n} d{5}
matches for 5 digits in a row

^
match the start of the string

$
match the of the string end

*
matches 0 or more repetitions

?
matches 0 or 1 characters of whatever precedes it

use . to match a period or to match a slash.

If you are unsure if a character has special meaning, such as ‘@’, you can
put a slash in front of it, @, to make sure it is treated just as a character.

re.findall

The findall() is probably the single most powerful function in the re module
and we will use that function in this script.

In the example below we create a string that have a text with many email
addresses.

We then create a variable (emails) that will contain a list of all the found
email strings.

Lastly, we use a for loop that we can do something with for each email string
that is found.

str = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher'

## Here re.findall() returns a list of all the found email strings
emails = re.findall(r'[w.-]+@[w.-]+', str) ## ['alice@google.com', 'bob@abc.com']
  
for email in emails:
    # do something with each found email string
    print email

We can also apply this for files. If you have a file and want to iterate over
the lines of the file, just feed it into findall() and let it return a list of
all the matches in a single step

read() returns the whole text of a file in a single string.

(If you want to read more about file handling in Python, we have written a
‘Cheat Sheet’ that you can find here)

# Open file
f = open('test.txt', 'r')

# Feed the file text into findall(); it returns a list of all the found strings
strings = re.findall(r'some pattern', f.read())

re.search

The re.search() method takes a regular expression pattern and a string and
searches for that pattern within the string.

The syntax is re.search(pattern, string).

where:
pattern
regular expression to be matched.

string
the string which would be searched to match the pattern anywhere in the string.

It searches for first occurrence of RE pattern within string with optional flags.

If the search is successful, search() returns a match object or None otherwise.

Therefore, the search is usually immediately followed by an if-statement to test
if the search succeeded.

It is common to use the ‘r’ at the start of the pattern string, that designates
a python “raw” string which passes through backslashes without change which is
very handy for regular expressions.

This example searches for the pattern ‘word:’ followed by a 3 letter word.

The code match = re.search(pat, str) stores the search result in a variable
named “match”.

Then the if-statement tests the match, if true the search succeeded and
match.group() is the matching text (e.g. ‘word:cat’).

If the match is false, the search did not succeed, and there is no matching text.

str = 'an example word:cat!!'

match = re.search(r'word:www', str)

# If-statement after search() tests if it succeeded
  if match:                      
    print 'found', match.group() ## 'found word:cat'

  else:
    print 'did not find'

As you can see in the example below, I have used the | operator, which search for either pattern I specify.

import re
programming = ["Python", "Perl", "PHP", "C++"]

pat = "^B|^P|i$|H$"

for lang in programming:
    
    if re.search(pat,lang,re.IGNORECASE):
        print lang , "FOUND"
    
    else:
        print lang, "NOT FOUND"

The output of above script will be:

Python FOUND
Perl FOUND
PHP FOUND
C++ NOT FOUND

re.sub

The re.sub() function in the re module can be used to replace substrings.

The syntax for re.sub() is re.sub(pattern,repl,string).

That will replace the matches in string with repl.

In this example, I will replace all occurrences of the re pattern (“cool”)
in string (text) with repl (“good”).

import re
text = "Python for beginner is a very cool website"
pattern = re.sub("cool", "good", text)
print text2

Here is another example (taken from Googles Python class ) which searches for all
the email addresses, and changes them to keep the user (1) but have
yo-yo-dyne.com as the host.

str = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher' 

## re.sub(pat, replacement, str) -- returns new string with all replacements,

## 1 is group(1), 2 group(2) in the replacement

print re.sub(r'([w.-]+)@([w.-]+)', r'1@yo-yo-dyne.com', str)

## purple alice@yo-yo-dyne.com, blah monkey bob@yo-yo-dyne.com blah dishwasher

re.compile

With the re.compile() function we can compile pattern into pattern objects,
which have methods for various operations such as searching for pattern matches
or performing string substitutions.

Let’s see two examples, using the re.compile() function.

The first example checks if the input from the user contains only letters,
spaces or . (no digits)

Any other character is not allowed.

import re

name_check = re.compile(r"[^A-Za-zs.]")

name = raw_input ("Please, enter your name: ")

while name_check.search(name):
    print "Please enter your name correctly!"
    name = raw_input ("Please, enter your name: ")

The second example checks if the input from the user contains only numbers,
parentheses, spaces or hyphen (no letters)

Any other character is not allowed

import re

phone_check = re.compile(r"[^0-9s-()]")

phone = raw_input ("Please, enter your phone: ")

while phone_check.search(phone):
    print "Please enter your phone correctly!"
    phone = raw_input ("Please, enter your phone: ")

The output of above script will be:

Please, enter your phone: s

Please enter your phone correctly!

It will continue to ask until you put in numbers only.

Find Email Domain in Address

Let’s end this article about regular expressions in Python with a neat script I
found on stackoverflow.

@
scan till you see this character

[w.]
a set of characters to potentially match, so w is all alphanumeric characters,
and the trailing period . adds to that set of characters.

+
one or more of the previous set.

Because this regex is matching the period character and every alphanumeric
after an @, it’ll match email domains even in the middle of sentences.

import re

s = 'My name is Conrad, and blahblah@gmail.com is my email.'

domain = re.search("@[w.]+", s)

print domain.group()

outputs:
@gmail.com

More Reading

https://developers.google.com/edu/python/regular-expressions
http://www.doughellmann.com/PyMOTW/re/
http://www.daniweb.com/

The post Regular Expressions in Python appeared first on PythonForBeginners.com.

The OS module in Python provides a way of using operating system dependent functionality.

The functions that the OS module provides allows you to interface with the underlying operating system that Python is running on – be that Windows, Mac or Linux.

You can find important information about your location or about the process. In this post I will show some of these functions.

OS functions

import os

Executing a shell command
os.system()    

Get the users environment 
os.environ()   

#Returns the current working directory.
os.getcwd()   

Return the real group id of the current process.
os.getgid()       

Return the current process’s user id.
os.getuid()    

Returns the real process ID of the current process.
os.getpid()     

Set the current numeric umask and return the previous umask.
os.umask(mask)   

Return information identifying the current operating system.
os.uname()     

Change the root directory of the current process to path.
os.chroot(path)   

Return a list of the entries in the directory given by path.
os.listdir(path) 

Create a directory named path with numeric mode mode.
os.mkdir(path)    

Recursive directory creation function.
os.makedirs(path)  

Remove (delete) the file path.
os.remove(path)    

Remove directories recursively.
os.removedirs(path) 

Rename the file or directory src to dst.
os.rename(src, dst)  

Remove (delete) the directory path.
os.rmdir(path)    

The post Python’s OS Module appeared first on PythonForBeginners.com.

The reason I started to use Pexepect was because I was looking for a module that can take care of some of the automation needs I have (mostly with ssh and ftp).

You can use other modules such as subprocess, but I find this module easier to use.

Note, this post is not for a Python beginner, but hey it’s always fun to learn new things

What is Pexpect?

Pexpect is a pure Python module that makes Python a better tool for controlling
and automating other programs.

Pexpect is basically a pattern matching system. It runs program and watches
output.

When output matches a given pattern Pexpect can respond as if a human were
typing responses.

What can Pexpect be used for?

Pexpect can be used for automation, testing, and screen scraping.

Pexpect can be used for automating interactive console applications such as
ssh, ftp, passwd, telnet, etc.

It can also be used to control web applications via `lynx`, `w3m`, or some
other text-based web browser.

Installing Pexpect

The latest version of Pexpect can be found here

wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install
 
# If your systems support yum or apt-get, you might be able to use the
# commands below to install the pexpect package. 

sudo yum install pexpect.noarch

# or

sudo apt-get install python-pexpect

Pexpect Methods

There are two important methods in Pexpect: expect() and send() (or sendline()
which is like send() with a linefeed).

The expect() method

Waits for the child application to return a given strong.

The string you specify is a regular expression, so you can match complicated
patterns.

emember that any time you try to match a pattern that needs look-ahead that
you will always get a minimal match.

The following will always return just one character:
child.expect (‘.+’)

Specify correctly the text you expect back, you can add ‘.*’ to the beginning
or to the end of the text you’re expecting to make sure you’re catching
unexpected characters

This example will match successfully, but will always return no characters:
child.expect (‘.*’)

Generally any star * expression will match as little as possible.

The pattern given to expect() may also be a list of regular expressions,
this allows you to match multiple optional responses.
(example if you get various responses from the server)

The send() method

Writes a string to the child application.

From the child’s point of view it looks just like someone typed the text from
a terminal.

The before and after properties

After each call to expect() the before and after properties will be set to the
text printed by child application.

The before property will contain all text up to the expected string pattern.

You can use child.before to print the output from the other side of the connection

The after string will contain the text that was matched by the expected pattern.

The match property is set to the re MatchObject.

Connect and download a file from a remote FTP Server

This connects to the openbsd ftp site and downloads the recursive directory
listing.

You can use this technique with any application.

This is especially handy if you are writing automated test tools.

Again, this example is copied from here

import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')

In the second example, we can see how to get back the control from Pexpect

Connect to a remote FTP Server and get control

This example uses ftp to login to the OpenBSD site (just as above),
list files in a directory and then pass interactive control of the ftp session
to the human user.

import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
print child.before    # Print the result of the ls command.
child.interact()       # Give control of the child to the user.

EOF, Timeout and End Of Line

There are special patters to match the End Of File or a Timeout condition.

I will not write about it in this article, but refer to the official documentation
because it is good to know how it works.

The post How to use the Pexpect Module appeared first on PythonForBeginners.com.

What is a CSV File?

CSV files are used to store a large number of variables – or data. They are incredibly simplified spreadsheets – think Excel – only the content is stored in plaintext.

And the CSV module is a built-in function that allows Python to parse these types of files.

It’s worth noting that when you work with a CSV file, you are dabbling in JSON development.

JSON – which stands for JavaScript Object Notation – is a format that is used to store information as JavaScript code in plaintext files. You don’t need to know JavaScript to work with these files, nor is the practice confined to that language. Obviously, since we’re working with Python here.

The text inside a CSV file is laid out in rows, and each of those has columns, all separated by commas. Every line in the file is a row in the spreadsheet, while the commas are used to define and separate cells.

Working with the CSV Module

To pull information from CSV files you use loop and split methods to get the data from individual columns.

The CSV module explicitly exists to handle this task, making it much easier to deal with CSV formatted files. This becomes especially important when you are working with data that’s been exported from actual spreadsheets and databases to text files. This information can be tough to read on its own.

Unfortunately, there is no standard so the CSV module uses “dialects” to support parsing using different parameters. Along with a generic reader and writer, the module includes a dialect for working with Microsoft Excel and related files.

CSV Functions

The CSV module includes all the necessary functions built in. They are:

• csv.reader
• csv.writer
• csv.register_dialect
• csv.unregister_dialect
• csv.get_dialect
• csv.list_dialects
• csv.field_size_limit

In this guide we are only going to focus on the reader and writer functions which allow you to edit, modify, and manipulate the data stored in a CSV file.

Reading CSV Files

To pull data from a CSV file, you must use the reader function to generate a reader object.

The reader function is designed to take each line of the file and make a list of all columns. Then, you just choose the column you want the variable data for.

It sounds a lot more complicated than it is. To prove it, let’s take a look at an example.

import CSV
With open(‘some.csv’, ‘rb’) as f:
reader = csv.reader(f)
for row in reader:
print row

Notice how the first command is used to import the CSV module?

Let’s look at another example.

import csv 
import sys
 
f = open(sys.argv[1], ‘rb’)
reader = csv.reader(f)
for row in reader
print row
 
f.close()

In the first two lines, we are importing the CSV and sys modules. Then, we open the CSV file we want to pull information from.

Next, we create the reader object, iterate the rows of the file, and then print them. Finally, we close out the operation.

CSV Sample File

We’re going to take a look at an example CSV file. Pay attention to how the information is stored and presented.

Title,Release Date,Director
And Now For Something Completely Different,1971,Ian MacNaughton
Monty Python And The Holy Grail,1975,Terry Gilliam and Terry Jones
Monty Python's Life Of Brian,1979,Terry Jones
Monty Python Live At The Hollywood Bowl,1982,Terry Hughes
Monty Python's The Meaning Of Life,1983,Terry Jones

Reading CSV Files Example

We’re going to start with a basic CSV file that has 3 columns, containing the variables “A”, “B”, “C”, and “D”.

$ cat test.csv
A,B,”C D”
1,2,”3 4”
5,6,7

Then, we’ll use the following Python program to read and display the contents of the above CSV file.

import csv
 
ifile = open(‘test.csv’, “rb”)
reader = csv.reader(ifile)
 
rownum = 0
for row in reader:
# Save header row.
if rownum ==0:
header = row
else:
colnum = 0
for col in row:
print ‘%-8s: %s’ % (header[colnum], col)
colnum + = 1
 
rownum + = 1
 
ifile.close()

When we execute this program in Python, the output will look like this:

$ python csv1.py
A      : 1 
B      : 2
C D      : 3 4
A      : 5 
B      : 6
C D      : 7

Writing to CSV Files

When you have a set of data that you would like to store inside a CSV file, it’s time to do the opposite and use the write function. Believe it or not, this is just as easy to accomplish as reading them.

The writer() function will create an object suitable for writing. To iterate the data over the rows, you will need to use the writerow() function.

Here’s an example.

The following Python program converts a file called “test.csv” to a CSV file that uses tabs as a value separator with all values quoted. The delimiter character and the quote character, as well as how/when to quote, are specified when the writer is created. These same options are available when creating reader objects.

import csv
 
ifile  = open('test.csv', "rb")
reader = csv.reader(ifile)
ofile  = open('ttest.csv', "wb")
writer = csv.writer(ofile, delimiter='', quotechar='"', quoting=csv.QUOTE_ALL)
 
for row in reader:
    writer.writerow(row)
 
ifile.close()
ofile.close()

When you execute this program, the output will be:

$ python csv2.py
$ cat ttest.csv
"A"     "B"     "C D"
"1"     "2"     "3 4"
"5"     "6"     "7"

Quoting CSV Files

With the CSV module, you can also perform a variety of quoting functions.

They are:

• csv.QUOTE_ALL – Quote everything, regardless of type.
• csv.QUOTE_MINIMAL – Quote fields with special characters
• csv.QUOTE_NONNUMERIC – Quote all fields that are not integers or floats
• csv.QUOTE_NONE – Do not quote anything on output

More Python Reading and Resources

http://docs.python.org/2/library/csv.html http://www.doughellmann.com/PyMOTW/csv/ http://effbot.org/librarybook/csv.htm http://www.linuxjournal.com/content/handling-csv-files-python http://programming-crash-course.codepoint.net/there_are_columns

The post Using the CSV module in Python appeared first on PythonForBeginners.com.