Simple HTTP handling with urllib2


I am continuing to poke around python this week looking the “standard way” to make HTTP requests and process HTTP responses. In Perl, there is the very complete LWP set of modules. In Java, there is the Apache HTTP client library. Python has several modules with related functionality, including httplib, urllib and urllib2 (which will change with Python 3.0).

What I need to do is create HTTP headers in the request and later read headers from the response. Oh, and I will want to look at the response’s body too.

The code below makes an HTTP to a small PHP script I have on this server (use it as needed). It sets a header, which is echoed by the PHP script. Later all the headers from the response are also echoed.

One of the things I like about urllib2 is the Request class. In the things I want to do (web services, OAuth), it is very helpful to build the request and be able to inspect it apart from the mechanism that performs the HTTP fetch.

import urllib2
import pprint
import sys

url = ""

R = urllib2.Request(url)
R.add_header("User-agent", "jjohn/1.0")

try :
    f = urllib2.urlopen(R)
    headers =
    c = # content                                                      
except :
    e = sys.exc_info()[0]
    print "Oops: ", e

print "Content: " + c + "\n"
print "HEADERS: ";
for k in headers :
    print "'" + k + "'"