Latest News

It occurred to me today that there is an important philosophical difference between the older web service protocols like XML-RPC and SOAP think of themselves and the way REST does.

SOAP and XML-RPC spend a lot of their spec on format of their messages. In particular, how application data is serialized into XML. How that message is moved from one host to another is less important. Both protocols assume that developers will use HTTP, but certainly other protocols can be used. In fact, Leostream at one point used XML-RPC messages across a pipe rather than TPC socket.

REST on the other hand focuses almost entirely on how messages are passed. That is, REST is tightly bound to HTTP. Application data can be serialized in a number of ways, none of which are particularly important to REST.

For a long time, I've ignore the Representational State Transfer (REST) architecture. For one thing, I don't particularly agree with its premise that remote procedure calls (RPC) that use HTTP as a transport mechanism should obey the same semantics as regular web traffic. Things like XML-RPC and SOAP are, to my thinking, happening on an entirely different layer of the application stack than HTTP. Indeed, there are implementations of XML-RPC that do no use HTTP at all.

I remember pretty heated arguments I witnessed at tech conferences in the early 2000s about this seemingly unimportant technical point. For REST adherents, web services are another form of web traffic and should be treated as such. Given that Twitter, Facebook and Bit.ly all use REST for their APIs and older apps like liveJournal use XML-RPC/SOAP, I guess REST is the new hotness.

I've recently had reason to interact with the Twitter and Bit.ly APIs. This has made me come to terms with REST RPC mechnanisms. I admit, the sad, sick part of me that enjoys playing around with low-level HTTP stuff finds satisfaction in the way these API leverage existing HTTP features like basic authentication, extra path info, and GET and POST semantics. In this post, I thought I would show a bit of Perl code I wrote post status updates to Twitter, an activity more commonly referred to as "tweeting."

Twitter's API documentation is relatively straight forward, if you already have a solid grounding in HTTP. The API call to tweet is called "statuses/update". The basics of the RPC mechanism are easy enough:

• The caller makes a HTTP GET or POST request
• The sender replies with content in the form of JSON or XML

Let's start with the request. There are serveral bits of information required by the API: user credentials, the URL and additional query parameters. The user credentials are passed as part of the HTTP request header as a basic authentication field, which is merely a base64 string that is the concatenation of the username and password of your Twitter account. Fortunately, Perl's HTTP::Request::Common class makes it easy to add basic auth credentials to the request without knowing how this information is encoded in the HTTP request.

The next bit is the URL to the function. This is a core idea of REST -- function calls should have URIs and look like ordinary web resources. In this case, the URL is http://twitter.com/statuses/update.xml. Interestingly, the response from twitter can be encoded in a number of formats. These formats are determined by the extension you give to the URL. For instance, I could have request the metainformation about myself in JSON with the following URL: http://twitter.com/users/show/taskboy3000.json.

The text of the tweet must be passed to the URL as if it were POSTed from a form. The parameter name is status. The status must be encoded as if the data were submitted from an HTML form. Again, Perl makes this very easy, as will be shown below.

use LWP::UserAgent;
use HTTP::Request::Common ('POST')

my $api_url = q[http://twitter.com/statuses/update.xml];
my $status = "Tweeting from the API!";
my $twitter_username = "taskboy3000";
my $twitter_password = "s3cr3t";

my $ua = LWP::UserAgent->new;
my $req = POST($api_url => [status => $status]);
$req->authorization_basic($twitter_username 
			  => $twitter_password);

# Make the request
my $res = $ua->request($req);

The code above is sets up and makes the status RPC call to twitter. The first thing needed is an LWP::UserAgent object, which is kind of like a web browser. It makes HTTP requests of web servers. To construct the POST request, I use HTTP::Request::Common::POST. Because I can pass in form parameters as plain perl data structures, it frees me from worrying about urlencoding values and fooling around with HTTP headers that are germain to the task at hand. POST() returns an HTTP::Request object.

Adding my twitter account credentials to the request is a simple one line call to authorization_basic(). Very handy and very clean. That's all the setup I need to make the request. I pass in the HTTP::Request object to the User Agent object. That makes the actual network connection to the URL. The response comes back in the form of an HTTP::Response object, which I'll discuss next.

If all has gone well with the request, I'll get back an XML document that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<status>
<created_at>Tue Apr 07 22:52:51 +0000 2009</created_at>
<id>1472669360</id>
<text>At least I can get your humor through tweets. 
RT @abdur: I don't mean this in a bad way, but 
genetically speaking your a cul-de-sac.</text>
<truncated>false</truncated>
<in_reply_to_status_id>1472669230</in_reply_to_status_id>
<in_reply_to_user_id>10759032</in_reply_to_user_id>
<favorited>false</favorited>
<in_reply_to_screen_name></in_reply_to_screen_name>
<user>
<id>1401881</id>
 <name>Doug Williams</name>
 <screen_name>dougw</screen_name>
 <location>San Francisco, CA</location>
 <description>Twitter API Support. Internet, greed, 
users, dougw and opportunities are 
my passions.</description>
 <url>http://www.igudo.com</url>
 <protected>false</protected>
 <followers_count>1027</followers_count>
 <profile_text_color>000000</profile_text_color>
 <profile_link_color>0000ff</profile_link_color>
 <friends_count>293</friends_count>
 <created_at>Sun Mar 18 06:42:26 +0000 2007</created_at>
 <favourites_count>0</favourites_count>
 <utc_offset>-18000</utc_offset>
 <time_zone>Eastern Time (US & Canada)</time_zone>
 <profile_background_tile>false</profile_background_tile>
 <statuses_count>3390</statuses_count>
 <notifications>false</notifications>
 <following>false</following>
 <verified>true</verified>
</user>
</status>

Most of this, I don't care about. However, I do want to see if there's an tag. If so, there was a problem with the post. The way I handle this error checking can be see in the following code.

 
unless ($res->is_success) {
    my $c = $res->content;
    my ($errstr) = ($c =~ m!<error>([^<]+)</error>!);
    warn(sprintf("Post failed (%d): $errstr\n", $res->code));
    exit 1;
}

print "OK\n";
exit 0; 

Without the services of a full XML parser, it's relatively easy to look for an error tag and extract the contents for display. The error message I've encountered most is essentially "you used the API too much". Twitter does restrict the usage of some of their API calls, but not the status one.

If you collapse all the Perl code, you're looking at less than 20 lines of code. If you wanted to, you could even make posts using the very handy command line tool curl: curl -u taskboy:s3cr3t -d "status=hello curl" \
http://twitter.com/statuses/update.xml

I will leave the checking of error messages from curl output as an excerise for the reader.

As I said, REST RPC mechanisms are fun and interesting if you already understand HTTP. However, not everyone does. I think XML-RPC and SOAP libraries to a better job of insulating the programmer from the HTTP protocol, allowing him to focus on the API task at hand.

Just an update for those keeping track: my girlfriend Sally is washing the hand soap in the bathroom.

No word yet on what sort of soap soap she's using to get the soap clean.

Will post an update when I learn what she uses to clean the soap that cleans the soap she's cleaning now.

E_RECURSION_TOO_DEEP

Since use.perl.org has become my de facto backup solution, I now post the scripts I use to blog from winders. These are modified versions of the scripts I mentioned in a use.perl.org article published a while ago.

The emacs file:

(defvar prog
   "C:/perl/bin/perl.exe F:/blog/use_perl_blog.pl"
   "use_perl_journal: A SOAP client for use.perl journaling"
)

(defun edit-entry ()
   "Add an entry or edit an existing one"
   (interactive)
   (setq cmd (concat prog " edit"))
   (widen)
   (shell-command-on-region (point-min) (point-max) cmd)
)

(defun get-entry (n)
  "Get journal entry from use.perl.org"
  (interactive "sJournal ID: ")
  (setq buffer (generate-new-buffer "*use_perl_journal*"))
  (switch-to-buffer buffer)
  (setq cmd (concat prog (concat " -i " (concat n " get"))))
  (shell-command-on-region (point-min) (point-max) 
        cmd 1 nil nil) 
)

(defun list-entries (uid limit)
   "Get journal entries"
   (interactive "sUser ID: nsLimit: ")
   (setq buffer (generate-new-buffer 
        "*use_perl:list_entries*"))
   (switch-to-buffer buffer)
   (setq cmd (concat prog (concat " -l " (concat 
        limit " -i " (concat uid " list")))))
   (shell-command-on-region (point-min) (point-max) 
        cmd 1 nil nil)
)


(defun delete-entry (jid)
  "Delete journal entry"
  (interactive "nEntry ID: ")
  (setq cmd (concat prog (concat " -i " (concat jid 
        (concat " delete")))))
  (shell-command-on-region (point-min) (point-max) 
        cmd 1 nil nil)
)

;; don't use tabs
(setq-default indent-tabs-mode nil)

(global-set-key "C-xtl" `list-entries)
(global-set-key "C-xtg" `get-entry)
(global-set-key "C-xts" `edit-entry)
(global-set-key "C-xtm" `edit-entry)
(global-set-key "C-xtd" `delete-entry)

The perl script:

# -*-cperl-*-
# A SOAP client to post USE.PERL.ORG journal entries

use strict;
use HTTP::Cookies;
use SOAP::Lite;
use File::Basename;
use Digest::MD5 'md5_hex';
use Data::Dumper;
use Getopt::Std;

use constant DEBUG => 0;
use constant UID   => -1; # your UID here
use constant PW    => 's3cr3t'; # your pw here
use constant URI   => 'http://use.perl.org/Slash/Journal/SOAP';
use constant PROXY => 'http://use.perl.org/journal.pl';

my $Dispatch = {
                 'get'  => &get_entry,
                 'list' => &list_entries,
                 'add'  => &add_entry,
                 'edit' => &edit_entry,
                 'delete' => &delete_entry,
               };

my $opts = {};
getopts('h?vi:u:l:', $opts);

my $action = pop @ARGV;

unless ($action) {
  print usage(), "n";
  exit;
}

my $soap_client = make_soap();

my $exit_value = 0;
if (defined $Dispatch->{$action}) {
  $exit_value = !$Dispatch->{$action}->($opts, $soap_client);
} else {
  warn("Unknown action '$action'");
  print usage();
  $exit_value = 1;
}

exit $exit_value;

#------
# subs
#------

sub usage {
  my $base = basename($0);
  return qq[
$base - manage use.perl.org blog

 USAGE: 
   $base [options] [actions]

 OPTIONS:
   ?       print this screen
   h       print this screen
   v       verbose mode
   i   entry ID 
   l  limit the number of listed entries to this number
   u   use.perl.org user ID

 ACTIONS:  
  add
  delete 
  edit
  get
  list
 Input files take the following form:
	  id:
	  subject:
          body:
];
}

sub make_soap {
  my $cookie = HTTP::Cookies->new;
  $cookie->set_cookie( 0,
		       user => bakeUserCookie(&UID, &PW),
		       "/", 
		       "use.perl.org",
		     );

  return SOAP::Lite->uri(URI)->proxy(PROXY, 
                                  cookie_jar => $cookie);
}

sub add_entry {
  my ($opts, $c, $in) = @_;
  
  $in ||= parse_input();

  my $ret;
  if ($in->{subject} && $in->{body}) {
    if ($in->{id}) {
      return edit_entry(@_, $in);
    } else {
      $ret = $c->add_entry($in->{subject}, $in->{body}); 
    }
  } else {
    $ret = $c->add_entry("Random thought #$$", $in->{all});
  }

  return if had_transport_error($ret);
  print "add_entry got articleID: ", $ret->result, "n";
  return 1;
}

sub delete_entry {
  my ($opts, $c) = @_;

  my ($id) = $opts->{i} || 
        die "delete requires a journal IDn";
  my $ret = $c->delete_entry($id);
  return if had_transport_error($ret);
  print "Deleted article ID '$id'n";
  return 1;
}

sub edit_entry {
  my ($opts, $c, $in) = @_;

  # add_entry may have already read STDIN
  $in ||= parse_input(); 
 
  unless ($in->{id}) {
    # warn("No article IDn");
    return add_entry($opts, $c, $in);
  }

  my $ret = $c->modify_entry($in->{id},
			     subject => $in->{subject},
			     body => $in->{body},
			    );

  return if had_transport_error($ret);

  print "Updated article $in->{id}n";

  return 1;
}

sub get_entry {
  my ($opts, $c) = @_;

  my $id = $opts->{i} 
        || die "get_entry requires a journal IDn";
  my $ret = $c->get_entry($id);
  return if had_transport_error($ret);

  if (my $hr = $ret->result) {
    while (my ($k,$v) = each %{$hr}) {
      print "$k: $vn";
    }

  } else {
    warn ("Couldn't fetch journal entry '$id'n");
    return;
  }
  return 1;
}

sub list_entries {
  my ($opts, $c) = @_;
  my ($uid, $limit) = (($opts->{u} || &UID), $opts->{l});

  my $ret = $c->get_entries($uid, $limit);
  return if had_transport_error($ret);

  my $ar = $ret->result;
  for my $row (@{$ar}) {
    while (my ($k,$v) = each %{$row}) {
      print "$k: $vn";
    }
    print "n";
  }

  return 1;
}


sub parse_input {
  my %rec;

  my $last_field = 'all';
  while (defined ($_ = )) {
    chomp($_);
    if (/^(w+):s*(.*)/) {
      $last_field = $1;
      $rec{$last_field} = $2;
    } else {
      $rec{$last_field} .= "n$_";
    }
  }

  return %rec;
}

sub bakeUserCookie {
  my ($uid, $pw) = @_;
  my $c = $uid . "::" . md5_hex($pw);
  $c =~ s/(.)/sprintf("%%%02x", ord($1))/ge;
  $c =~ s/%/%25/g;
  return $c;
}

sub had_transport_error {
  my ($ret) = @_;

  if ($ret->fault) {
    warn ("Oops: ", $ret->faultString, "n");
    return 1;
  }

  return;
}

To post:

• M-x load-file
• new buffer with "id:nsubject:nbody:";
• add blog content to buffer
• M-x t s to publish blog to use.perl

[Original use.perl.org post and comments.]

SOAP::Lite is a suite of Perl modules for doing SOAP RPC. SOAP::Lite was originally written by mad Russians. Its incredible flexibility is also it's main drawback. Debugging isn't as obvious as it could be with this library.

Most of the time you the would-be SOAP scripter want to know what the request and response XML messages look like. The SOAP::Trace doc doesn't make this clear (since the sample code is filled with mistakes). Here is one way to get SOAP to spew the XML messages to STDOUT they way you'd expect from a more humble module like Frontier::Client.

use SOAP::Lite "trace" => ["transport" => \&log_it];
sub log_it {
  my ($in) = @_;

  if (ref $in && $in->can("content")) {
    printf "**GOT: %sn", (ref $in);
    print "-"x60, "n";
    print $in->content, "n";
    print "-"x60, "n";
  }

}

There's a lot of fun things going on in the log_it() subroutine. Notice the use of the oft-forgotten can() method, which all Perl objects have. Yes, this is a very special code review.

Should SOAP::Lite have a simple flag like "trace_xml" which does all this for you? Sure it should. But until then, you've got this humble blog and your old Uncle jjohn to help you.

Now get the hell off my lawn, you kids!