Latest News

It occurred to me today that there is an important philosophical difference between the older web service protocols like XML-RPC and SOAP think of themselves and the way REST does.

SOAP and XML-RPC spend a lot of their spec on format of their messages. In particular, how application data is serialized into XML. How that message is moved from one host to another is less important. Both protocols assume that developers will use HTTP, but certainly other protocols can be used. In fact, Leostream at one point used XML-RPC messages across a pipe rather than TPC socket.

REST on the other hand focuses almost entirely on how messages are passed. That is, REST is tightly bound to HTTP. Application data can be serialized in a number of ways, none of which are particularly important to REST.

For a long time, I've ignore the Representational State Transfer (REST) architecture. For one thing, I don't particularly agree with its premise that remote procedure calls (RPC) that use HTTP as a transport mechanism should obey the same semantics as regular web traffic. Things like XML-RPC and SOAP are, to my thinking, happening on an entirely different layer of the application stack than HTTP. Indeed, there are implementations of XML-RPC that do no use HTTP at all.

I remember pretty heated arguments I witnessed at tech conferences in the early 2000s about this seemingly unimportant technical point. For REST adherents, web services are another form of web traffic and should be treated as such. Given that Twitter, Facebook and Bit.ly all use REST for their APIs and older apps like liveJournal use XML-RPC/SOAP, I guess REST is the new hotness.

I've recently had reason to interact with the Twitter and Bit.ly APIs. This has made me come to terms with REST RPC mechnanisms. I admit, the sad, sick part of me that enjoys playing around with low-level HTTP stuff finds satisfaction in the way these API leverage existing HTTP features like basic authentication, extra path info, and GET and POST semantics. In this post, I thought I would show a bit of Perl code I wrote post status updates to Twitter, an activity more commonly referred to as "tweeting."

Twitter's API documentation is relatively straight forward, if you already have a solid grounding in HTTP. The API call to tweet is called "statuses/update". The basics of the RPC mechanism are easy enough:

• The caller makes a HTTP GET or POST request
• The sender replies with content in the form of JSON or XML

Let's start with the request. There are serveral bits of information required by the API: user credentials, the URL and additional query parameters. The user credentials are passed as part of the HTTP request header as a basic authentication field, which is merely a base64 string that is the concatenation of the username and password of your Twitter account. Fortunately, Perl's HTTP::Request::Common class makes it easy to add basic auth credentials to the request without knowing how this information is encoded in the HTTP request.

The next bit is the URL to the function. This is a core idea of REST -- function calls should have URIs and look like ordinary web resources. In this case, the URL is http://twitter.com/statuses/update.xml. Interestingly, the response from twitter can be encoded in a number of formats. These formats are determined by the extension you give to the URL. For instance, I could have request the metainformation about myself in JSON with the following URL: http://twitter.com/users/show/taskboy3000.json.

The text of the tweet must be passed to the URL as if it were POSTed from a form. The parameter name is status. The status must be encoded as if the data were submitted from an HTML form. Again, Perl makes this very easy, as will be shown below.

use LWP::UserAgent;
use HTTP::Request::Common ('POST')

my $api_url = q[http://twitter.com/statuses/update.xml];
my $status = "Tweeting from the API!";
my $twitter_username = "taskboy3000";
my $twitter_password = "s3cr3t";

my $ua = LWP::UserAgent->new;
my $req = POST($api_url => [status => $status]);
$req->authorization_basic($twitter_username 
			  => $twitter_password);

# Make the request
my $res = $ua->request($req);

The code above is sets up and makes the status RPC call to twitter. The first thing needed is an LWP::UserAgent object, which is kind of like a web browser. It makes HTTP requests of web servers. To construct the POST request, I use HTTP::Request::Common::POST. Because I can pass in form parameters as plain perl data structures, it frees me from worrying about urlencoding values and fooling around with HTTP headers that are germain to the task at hand. POST() returns an HTTP::Request object.

Adding my twitter account credentials to the request is a simple one line call to authorization_basic(). Very handy and very clean. That's all the setup I need to make the request. I pass in the HTTP::Request object to the User Agent object. That makes the actual network connection to the URL. The response comes back in the form of an HTTP::Response object, which I'll discuss next.

If all has gone well with the request, I'll get back an XML document that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<status>
<created_at>Tue Apr 07 22:52:51 +0000 2009</created_at>
<id>1472669360</id>
<text>At least I can get your humor through tweets. 
RT @abdur: I don't mean this in a bad way, but 
genetically speaking your a cul-de-sac.</text>
<truncated>false</truncated>
<in_reply_to_status_id>1472669230</in_reply_to_status_id>
<in_reply_to_user_id>10759032</in_reply_to_user_id>
<favorited>false</favorited>
<in_reply_to_screen_name></in_reply_to_screen_name>
<user>
<id>1401881</id>
 <name>Doug Williams</name>
 <screen_name>dougw</screen_name>
 <location>San Francisco, CA</location>
 <description>Twitter API Support. Internet, greed, 
users, dougw and opportunities are 
my passions.</description>
 <url>http://www.igudo.com</url>
 <protected>false</protected>
 <followers_count>1027</followers_count>
 <profile_text_color>000000</profile_text_color>
 <profile_link_color>0000ff</profile_link_color>
 <friends_count>293</friends_count>
 <created_at>Sun Mar 18 06:42:26 +0000 2007</created_at>
 <favourites_count>0</favourites_count>
 <utc_offset>-18000</utc_offset>
 <time_zone>Eastern Time (US & Canada)</time_zone>
 <profile_background_tile>false</profile_background_tile>
 <statuses_count>3390</statuses_count>
 <notifications>false</notifications>
 <following>false</following>
 <verified>true</verified>
</user>
</status>

Most of this, I don't care about. However, I do want to see if there's an tag. If so, there was a problem with the post. The way I handle this error checking can be see in the following code.

 
unless ($res->is_success) {
    my $c = $res->content;
    my ($errstr) = ($c =~ m!<error>([^<]+)</error>!);
    warn(sprintf("Post failed (%d): $errstr\n", $res->code));
    exit 1;
}

print "OK\n";
exit 0; 

Without the services of a full XML parser, it's relatively easy to look for an error tag and extract the contents for display. The error message I've encountered most is essentially "you used the API too much". Twitter does restrict the usage of some of their API calls, but not the status one.

If you collapse all the Perl code, you're looking at less than 20 lines of code. If you wanted to, you could even make posts using the very handy command line tool curl: curl -u taskboy:s3cr3t -d "status=hello curl" \
http://twitter.com/statuses/update.xml

I will leave the checking of error messages from curl output as an excerise for the reader.

As I said, REST RPC mechanisms are fun and interesting if you already understand HTTP. However, not everyone does. I think XML-RPC and SOAP libraries to a better job of insulating the programmer from the HTTP protocol, allowing him to focus on the API task at hand.

Perl now has three implementions of XML::RPC, Frontier::RPC2, SOAP::Lite and RPC::XML. Each is interesting and somewhat broken in its own way.

Frontier::RPC2 0.06 has issues with Boolean, iso8601 and Base64 because these classes have a bug in their constructors.

Given:

sub new {
    my $type = shift; 
    my $value = shift;

    return bless \$value, $type;
}

Instead of checking to see if $type is a reference, this code blesses. If a reference is passed in, the blessed class is sometime like "Frontier::RPC2::Base64=SCALAR(0x65432)". This breaks code later than encodes/decodes these values into XML. Perltoot has the solution to this problem

RPC::XML is an interesting module in that it brings type checking to Perl, in a sense. When creating an XML-RPC server, each remote procedure has to the arguments it accepts along with its return type. This is call a signature. Although type checking is unPerlish, it is very helpful when dealing with other languages that are subPerl. More docs on signatures would be great.

Finally, SOAP::Lite brings its own bad self to the XML-RPC party. Although in my testing it works with all the XML-RPC datatypes, the programming interface is similar to SOAP::Lite (not surprisingly). It is a style which takes a bit of getting usef to.

The good news is that, at least from a client level, all these modules seemed to be compatible. That is, a Frontier::Client can talk to an XMLRPC::Lite server.

More to come.

[Original use.perl.org post and comments.]

Those interested in XML-RPC will enjoy my latest article on Using XML-RPC with C. It should remind you why Perl is so much more fun to program in.

Also, please note that my hair isn't in an afro. The picture is just oddly cropped. Perhaps, I need a better headshot or a better afro.

[Original use.perl.org post and comments.]

[Original use.perl.org post and comments.]

I just noticed that the London Perl Mongers review the XML-RPC book. Thanks to Dean Wilson for a pretty balanced review.

As the author of the ASP/COM section, I'd like to take a minute to answer a valid criticisim of that chapter that Wilson raises:

My only disappointment with this chapter is that it relegates the mention of the COM based XML-RPC library to two paragraphs tucked away at the end of the chapter and does no more than acknowledge its existence and provide a link. Adding another small section covering its use with something like Visual Basic would have been a welcome addition.

At the time of that book's writing, the COM XML-RPC library looked moldy and forgotten. I was none too keen to drive people to use it. I don't have Visual Studio, so I couldn't really experiment with that library anyway (which would have required a C++ compiler, I believe). When the next edition of the book comes out, I'll see if we can't flesh out that chapter a bit more with more Windows Tech (eek!).

Wilson also pointed out that I mixed a fair amount of Perl into that chapter. I had two reasons for doing so. The first reason was didactic: I wanted to stress the language independence of the protocol. I would have like to have seen more of that in the book, actually. The second reason is that I wanted to reach out to Unix bigots like myself who might see XML-RPC as a way to control Winders boxen without haven't to do much programming on them. XML-RPC has the potential to create easy remote admin tools -- very useful for controlling point'n'click environments from a remote unix shell. ;-)

Thanks again to Dean Wilson for taking the time to review the book.

[Original use.perl.org post and comments.]