Latest News
«If I could save time in a bottle
The first thing that I'd like to do
Is to save every day
Till Eternity passes away
Just to spend them with you»
--Jim Croce
If you're anything like me, you live and die by your calendar. Over the years, I've moved from simple paper calendars to the Palm Pilot back to paper and finally ending up online with Yahoo and Google. This brief post talks about creating hyperlinks that add an event to the clicker's yahoo or google calendar.
Of course, not everyone uses these online calendars. Some use Outlook or iCal
or some other desktop solution. The key to publishing events for these systems
is to use the ICalendar format
which most desktop apps can import. The MIME type for such an ICalendar file is
text/calendar. You might also consider using the XHTML microformat
hCalendar, which I mentioned briefly in an
earlier post.
However, this microformat isn't likely to be understood by most desktop apps.
First, let's suppose that we have an event that we want to publish. I turn 40 next year, so let's create an birthday party event for that. In the iCalendar format, such an event might look like this:
BEGIN:VCALENDAR VERSION:2.0 PRODID:-//taskboy calendar app BEGIN:VEVENT SUMMARY:Joe turns 40 just this once ORGANIZER;CN=Joe Johnston:MAILTO:jjohn@taskboy.com DTSTAMP:20100421T105300 DTSTART:20111212T190000 DTEND:20111212T200000 END:VEVENT END:VCALENDAR
A great deal of this format is boilerplate stuff. The overall container is VCALENDAR, which has a BEGIN and END. Immediately after that is meta-information (like HTML's HEAD section) containing the format version and the app's product ID (which is arbitrary). After the header, the actual VEVENT section begins. There are many types of objects VCalendar can contain including VREPLY, VJOURNAL and VTODO, but this isn't about those (read the RFC for more info). Let's look at the VEVENT attributes in tabular form:
| Attribute | Meaning | |
|---|---|---|
| SUMMARY | A brief description of the event | |
| ORGANIZER | Who organized this event, in LDAP format | |
| DTSTAMP | When this ICalendar file was created | |
| DTSTART | When this event starts | |
| DTEND | When this event ends |
The purpose these elements is relatively self-evident. The datetime format used throughout is the ever-popular ISO8601. The ORGANIZER is given in LDAP format, but the spec does not require it to be so. However, most applications, like Outlook, will attempt to locate ORGANIZER in an LDAP system, so this makes a bit of sense.
Above is a basic, serviceable ICalendar file. You can even check this using the handy ICalendar validator. Enough about the desktop. Let's move on to web-based calendars.
Yahoo Calendar does not, as far as I can see, publish an API. However, people have ferreted out enough information to be useful. By creating a simple HTTP GET request (in the form of a hyperlink), events can be entered into Yahoo Calendar.
The base URL for Yahoo Calendar is: http://calendar.yahoo.com/. The following
parameters are required: v=60 and TITLE=event. Next come the
metadata for the event itself:
| Parameter | Meaning |
|---|---|
| DESC | A brief description of the event |
| ST | ISO8601 datetime of when the event begins |
| DUR | How long the event lasts in HHMM format |
| URL | A URL to a page describing the event |
| in_loc | A brief label for the event location |
| in_st | Street address of the event |
| in_csz | City/State/Zip of the event |
Remember: all parameters must be URLencoded. There is a TYPE parameter that can specify the kind of event. The default is "Appointment" (type 10). See the link to Chris's notes for more options there. The following link will add my Birthday event to your Calendar:
Google Calendar has a similar system, but it is
documented.
The base URL for adding calendar events is http://www.google.com/calendar/event.
There is one required parameter: action=TEMPLATE. Here is a rundown of their
parameters:
| Parameter | Meaning |
|---|---|
| text | A label for the event |
| dates | Of the form: START/END where START and END are in ISO8601 format |
| name | Description of the event |
| details | A description of the event |
| location | Description of the event |
These parameters need to be URLencoded too. Google differs slightly from the above formats in that the event can have both a label and a description. The start and end time of the event are given in one parameter, which is unnerving. Here's my example event for Google:
Because these links will not return the user to the site they came from, consider using a popup window for these links.
That's the gist of adding events to calendars. Of course, there are a lot more parameters that can be added to these publications and I haven't touched recurring events at all. Still, this is a good jumping off point for your own exploration of the magic of event scheduling.
«iSCSI has become the storage connectivity of choice for small business, with applications such as VMware VI3, Exchange, and MS SQL certified on it. It runs on standard Ethernet networks.»
iSCSI is a network protocol that allows remote disks to appear as local devices to computer. At a very high level, this schema falls under the rubric of Storage Area Network. The difference between iSCSI and NAS protocols that deliver file systems like NFS or Samba, is that that the shared iSCSI resources look like local devices to the system using them. This has some interesting consequences for diskless terminals and the like.
iSCSI is a client-server system. The server presents iSCSI targets (hard drives or files that pretend to be hard drives) clients that can use these resources. The clients are called initiators since they start the protocol. That is, servers with targets do not broadcast their services. iSCSI targets must be discovered by initiators.
When a client finds a desired target, it asks for authorization to use the resource via a login command. If that succeeds, a new SCSI devices appears on the client system that's read for partitioning and provisioning. If the disk as already been initialized, then the resouce can be mounted like any other filesystem.
It turns out, configuring iSCSI on Ubuntu (and other linux distros) is pretty straight-forward. Let's first set up a system that offers iSCSI targets. In this example, the target will be a file rather than a physical device.
Log into your Ubuntu machine. Become root (sudo su -) of you
aren't.
Create a disk image (dd if=/dev/zero of=/iscsi.disk count=0 obs=1
seek=2G). In this case, a empty 2GB file is created at the root of the
filesystem. Install the iscsitarget package (apt-get install
iscsitarget). Define the new target in /etc/ietd.conf. Edit the file
so that it looks something like:
Target iqn.2010-filer.network.local:iscsi.lun1 Lun 0 Path=/iscsi.disk,Type=fileio
The name of the target is pretty arbitrary, but you must have a unique naming convention for resources across your network of iSCSI targets. For more on this, see the Wikipedia article cited above. Since this target is the first one list, it gets Lun 0. This is analogous to how the first disk in a SCSI chain is listed by hardware.
Now, we need to twiddle some permissions. By default, iSCSI targets
are not published and allow no initiators. To enable targets to be discovered,
edit /etc/default/iscsitarget and set the ENABLE flag to "true". Now allow
initiators to discover your targets by editing /etc/initiators.allow. Add
a the following line to the bottom of the file ALL ALL.
Finally, we're ready to start the target daemon. This is the process
that initiators talk to. To start (or restart it), type
/etc/init.d/iscsitarget restart.
Now we're ready to configure the initiator. Simple install the following
utilities: apt-get install open-iscsi open-iscsi-utils. Now,
discover the targets. In this example, assume that the target machine is
called filer.network.local.
iscsiadm -m discovery -t sendtargets -p filer.network.local
You should see a list of available targets offered by the filer. Select one (e.g. "iqn.2010-filer.network.local:iscsi.lun1") and get authorized to use it:
iscsiadm -m node -p filer.network.local \ --targetname "iqn.2010-filer.network.local:iscsi.lun1" --login
Please note that I merely split the line for display purposes. If all has
gone well, you should be able to see the new device through fdisk:
fdisk -l. You may see a new and empty device called /dev/sdb (if
you already have one SCSI disk). This disk is ready for partitioning with
fdisk, provisioning with mkfs and mounting on your root filesystem.
When you reboot, your iSCSI disks should be available without using the iscsiadm tool. The daemon processed that do this are controlled by the /etc/init.d/open-iscsi script.
What I presented here is fine for local, trusted networks. There is zero security in this configuration. There are all sorts of authentication schemes for targets and encryption options for the iSCSI traffic. Also, using a fake disk (i.e. a file) as a target will not give you very good performance on the host. I leave all of these enhancements as an exercise for the reader.
UPDATE: Here a link to the Microsoft iSCSI initiator for Windows XP and above.
«I think there is a world market for maybe five computers»
--Fictional Quote by Thomas Watson
Last weekend, Apple's iPad went on sale in their retail stores. Coincidentally, my iPod classic (a 6 year old hand-me-down) had recently become nearly unusable due to hard drive errors (I told you this was a bad design years ago). Desiring a new MP3 device, I waddled into the Apple store in Nashua, NH. There were two armed security guards there and a gaggle of gawkers pawing at a table full of iPads. I moved deeper into the store, found an employee with a lot of gray hair (I'm not hip enough for the tattoo'ed set) and obtained a 32GB iPod Touch. Why on Earth would I choose an iPod over the fancy iPad? Let me illuminate the ways.
I want to promulgate an essentialist argument about purpose of three devices: iPad, iPod Touch and the Amazon Kindle. The Kindle (which I received as a Christmas present) is designed as an ebook reader, library and store front. That is, not only can you read books on it, but you can purchase new ones from Amazon and have them immediately downloaded to your device. Additionally, the Kindle comes with free 3G service so that you can access a low-resolution version of the Internet anywhere. This feature alone should scare Apple into a price/feature war with Amazon. Add a decent battery life and a daylight-readable display, you will quickly see that the Kindle's offering is excellent for those seeking an excellent ebook experience.
Enter the iPad. The iPad has a large color LCD screen and the Apple mobile OS. It does not come with any 3G service, but it does include wifi support. This makes it a large iPod Touch, but without the microphone headphones that come standard with the new Touch models. As a laptop, the iPad needs a keyboard, which is both proprietary and sold separately. As a gaming device, the iPad, with the built-in gyroscope and networking, will offer hours of casual gaming fun, but it won't kill the traditional gaming consoles. As an ebook reader, it does not compete well against the kindle. The iPad is significantly more expensive and worse on battery life. Also, you may not want to spend more of your time looking at yet another LCD screen for leisure.
The iPad looks a lot like a new kind of netbook. It is a general purpose computer ideally suited for causal computing tasks like web browsing, tweeting, and a game or two. In this capacity, I expect the iPad to meet with moderate success. It is prettier than the other netbooks, if a bit pricier. However, it is extremely portable (although not pocket-portable) and I expect to see iPad proliferate in cafes very soon.
And what of the now lowly iPod Touch? Why would anyone buy this little brother iPad now? There are two compelling features of the newer Touch models: size and the microphone headset. The Touch is essentially a very small general purpose device. As an MP3 player, I find it much harder to use than the older iPod classic, the iPod Nano and especially the Shuffle. Those devices are single-purposed: they play music and their interface is bent to this single task. Those devices are excellent. Don't believe me? Take an iPod Touch and try to get it to play a random selection of music. First, you have to hit the only button on the device. Then you use the touch pad to "unlock" the UI. Then you search for the button labelled "Music". Try not to get confused with the iTunes icon, which is what you'd click on a Mac OS X machine. Then you hit the Music button at the bottom only to get (perhaps) a list of songs. If you don't see a list of songs, you must press the "Songs" icon at the button of the app. Scroll to the top of this list and you'll see an area called Shuffle. It might occur to you to then press in the "Shuffle" area. Why no button? Buttons afford pressing. I though the Apple UI guys where supposed to be good at this kind of thing. If you get through all this, you'll get some music. And what if you want to skip ahead to a new song? If the Touch as gone to sleep, you might find it hard to get back to the one screen that has the song forward control (hint, your Touch needs to be upright, not on its side).
Try this same set of tasks on the other iPods and you'll see that those devices are optimized for playing and managing music.
This is all to say that I didn't buy the Touch as a music player and that I do not believe that it is a music player at all. It's the tiniest netbook on the market. I can and do actually read web sites with this thing. I can check my yahoo mail with it. I even use Skype on it. That's right, Skype. At least in my house within range of my wifi, I have a cheap iPhone (itself a netbook/phone hybrid).
Where does this leave our three devices? For my money, the Kindle is without peer for reading ebooks. As a netbook, it's not very good (but that is by design). The iPad is great for those looking for a netbook with great screen real estate. The Touch is a netbook for those who put a premium on weight and space. The other iPods are still the kings of portable music devices.
UDPATE: I just learned that I can shake the Touch to change to the next song. Fun UI, I admit, it's not exactly a visible UI element either.
«Someone's sitting in the shade today because someone planted a tree a long time ago.»
In the first part of this series, I illuminated what I believe to be the important first steps of electronic social media (that is, social groups created and maintained primarily through computers). In this conclusion, I trace how social media moved beyond the technically savvy and privileged to the rest of us.
The term World Wide Web is rarely used anymore and that's a shame. The Web, a system of interconnected HTML documents retrieved through the HTTP protocol, is mass transit system of nearly all modern electronic communication that travels over the Internet. I suppose its a kind of compliment that when most folks refer to the Web, they call it the "Internet" (or "netternet" or "interwebs"). Whatever it's called, the system that allowed for the creation and proliferation of web sites allowed for the next generation.
For many, the Web was a kind of Graphic User Interface to the Internet. In a sense, everything that had been invented for UNIX needed to be brought into the world of personal computers (including UNIX itself in the form of Linux and *BSD). Is it a surprise that the twenty year old unix utility came back in the form of AOL's Instant Message in the late nineties? Not to me. If an earlier generation had lived socially through email, there was now a younger generation that lived on IM or ICQ. Mobile phones, which already had SMS messaging, began supporting IM directly so that you could always text chat with your friends when when you were away from your computer. IM is mostly a 1-1 connection. That is, one person chating with only one other person. Another limitation of IM is that the conversation were private and generally impermanent. For private conversations, that's probably OK, but clearly there was a market for more.
While some folks were making dinner plans over IM, others were journaling their rage through weblogs. Weblogs, for those recently revived coma victims, are a form of website built to be updated with new content frequently. The writing staff of blogs can be as small as one person or as many as dozens (or thousands in the case of kuro5hin, slashdot, digg, metafilter, etc). What makes blogs social is the commenting system that allows readers to respond to particular blog posts. Moreover, bloggers will reference the posts of other blogs. Being web pages, blogs are long-term publications that can be referred to in the future through search engines. Blogs exploded in the first part of the first decade of the twenty first century and now (2010) look to seriously challenge and even replace a large number of traditional print media organizations. Of couse, someone needs to figure out how to incorporated an editorial staff into blogs at some point, but that's a different article.
2002 saw the quiet beginnings of websites that were something both less and more than blogs. Sites like Friendster and MySpace were designed for what I'll call "small updates." A user might upload a picture or a song along with a brief updated on what they were doing to a "profile page." User content was limited to one page. Rarely would these posts be longer than a few sentences.
The real value of these sites was that they allowed users to identify connections with other users on the system. This is the beginning of web-based social networks, which is a bit of a dehumanizing term. A network is a system of nodes connected by routes. In social networking, you are a node. Your value, at least to marketers, was the number of your connections rather than the content of your profile.
However, the value of social networks to users was to build a broadcast system to a community of your friends (i.e. trusted contacts). This notion is quiet powerful and would continue its evolution in a web site designed to bring together academics. That site is Facebook. Facebook took all of the elements of social media that had been developed in other site and made user status updates private (by default). This element of exclusivity at first seems paradoxical, but many user want to limit the scope of their broadcasting. Facebook provides an excellent vehicle for this.
I would like to note that the blogging site LiveJournal also has an interesting feature to limit the visibility of posts using "locks". LJ posts are generally publically visible, but locked posts are visible only to friends.
There seems to be a lot of mileage to be had by limiting the functionality of blogs. If Facebook limits the scope of broadcasting, Twitter limits the length of posts. Twitter is a bit like a UNIX system log. It receives short updates from multiple programs. Any UNIX admin will tell you the utility of reading the system log. It's a snapshot of the current state of the system. In the same way, Twitter is not a media of deep intellectually pondering. It's a system that captures the zeitgeist of its users.
Twitter and Facebook operate as very different social media engines. Facebook (and linkedIn for that matter) are tools that identify existing relationships. Twitter encourages browsing for new friends.
And that's where social media is today. The web expanding the reach of social networks by removing a lot of the technical requirements of its users. It's hard to see what the next step for social media will be. I suspect it will be in the realm of mobile devices. Perhaps some kind of location-based ad-hoc social network. Perhaps there is value is creating networks of networks. The future is now.
«Adhocracy is a type of organization being antonymous to bureaucracy. The term was first popularized in 1970 by Alvin Toffler, and has since become often used in the theory of management of organizations (particularly online organizations), further developed by academics such as Henry Mintzberg.»
Electronic social media is not new, but the scale of it, perhaps is.
A recent OnPoint radio program talked about William Sims Bainbridge's, a social scientist, excersions into the World of Warcraft. Aside from a lot of old-manisms from the host, the show gave perhaps the most positive spin on the effect of this wildly popular MMORPG I've yet heard. After literally thousands of hours of play, Bainbridge reports what millions of people have already figured out: that humans in virtual worlds behave pretty much the same way they do in real life, but more so.
In meatspace, groups form around common goals or interests out of people in the local population. That is, the physical locality of people is a filter that will determine the size and composition of an interest group. Virtual worlds, of course, do not eliminate physicality entirely, but they do expand the definition of a locality. Even though you can have one million users from all across the planet together in one virtual space at once, the abstraction is a little leaky. Humans still need to sleep and the sun still needs twenty four hours to travel around the globe. Timezones and sleep schedules are the new locality filter for "real-time" group communication.
For grumpy old men like myself, electronic social media has been around for decades. Twitter and Facebook may be grabbing headlines today for creating a space in which millions interact daily, but that activity has been going on almost since the Internet went online. Below is a thumbnail sketch of the evolution of electronic communication that acted as social media.
Table: A Chronological List of Social Media
- chat/wall
- uucp/usernet/email
- BBS
- IRC
- IM
- Weblogs
- Friendster/Myspace
- Facebook/Twitter
In the pleistocene era of computing (the 1970s), users typically shared one computer system that often hadden upwards of 8MB of RAM. This multiuser systems (UNIX, VMS, MVS) had ways of sending messages to other logged in users on the system. These programs, like the UNIX utilities chat and wall, went a long way towards users seeing computers as places of social activity. Of course, the total worldwide population of computer users in the 70s was probably in low thousands. Social activity was limited to users on the same system.
Not long after (let's call it the 1980s), the ability to send messages to users on different systems came with the advent of unix-to-unix-copy (uucp) and email. These utilities helped make 1-1 connections among users of different computer installations. The system the collected emails into a taxonomic forum made its participants feel like there was one global electronic world. That system was Internet News or Usenet. The way people interacted on Usenet continues to be an excellent model for how people behave on all subsequent electronic media.
The growth of the Internet in the 80s and early 90s was happening in universities, government agencies and large corporations. However, the general public was dimly aware of this growth at all there was no easy (or interesting) way to bring this connectivity to the consumer. However, the sales of personal computers was booming and many unwashed savages were snapping up these devices to create spreadsheets and terrible dot-matrix printed flyers. If (generally speaking) the Internet was unavailable to these early computer users, modems were not. Small operators were setting Bulletin Board Systems (BBS) on their PCs and letting anyone dial into them. That is, you the PC owner had to find the phone number of a BBS, get some modem/terminal software, dial up the BBS and pay standard phone rates for the call as you were connected to the system. It all sounds barbaric now, but it was, at the time, cutting edging and exciting as a cyberpunk movie. I know this because that's when I began to get interested into becoming a programmer.
BBS created a very strong sense of community and ownership in its users. Remember that users committed real money to their BBS experience, so bad behavior was not long tolerated by the SysOp. As fun as BBS were, they had some fundamental limitations. The size of the online population was limited to the number of modems and phone lines the sysop had. Users might be allowed to upload static content, but not dynamic. These issues would be addressed by a different system that emerged in the mid-nineties.
As much as I'd like to present a neat timeline of descrete evolutionary steps for social media, that's not really how the real world works. While PC users were exchanging "documents" about UFOs and Roswell on BBS, university Internet users were chatting away in Internet Relay Chat (IRC) channels. Channels were ad-hoc virtual areas where nominally specific topics were discussed. The beauty of IRC is that it was a distributed system that allowed users from any Internet node to connect and chat. For me, IRC is the first modern social media that has all of the characterics that are often ascribed to the web based tools of 2008-2010. IRC was realtime, distributed chatting. If you wanted to know what was going on in another part of the world, IRC was often a good place to hang out. Don't believe me? IRC was used to report on the 1991 Soviet coup when all other media was blacked out. I'm waiting for Twitter to top that.
Unfortunately, IRC is mostly a command line application and its popularity was often limited to somewhat tech-savvy people. It is my hope that history will remember it more kindly that it currently does.
In the conclusion of this article, I'll discuss how electronic social media swelled its ranks from thousands of users to millions.
Doc Searls points out our growing dependency on Google. In brief, he equates Google with a kind of free public utility that provides the following functionality:
- Maps/Satellite data
- Search
- Mobile phones (via Android)
To that list, I would add the following features on which many people and companies have come to depend:
- Advertising (via AdWords)
- Ad-based venue (via AdSense)
- Email and Instant Messaging (via Gmail)
- Voice Mail/Phone routing (via Talk)
- New aggregation (via News)
Searls, with a "me too" from Dave Winer, declaim that Google has become "too big to fail." They worry that Google, fattened and dependent on its advertising engine, is vulnerable to economic bubbles in advertising. They worry that Google is in a bubble right now.
I do not think this is the case. Google no doubt enjoys quite a bit of revenue from advertising right now. However, no one outside of Google has a complete accounting of the company's revenue. If Google survived the recessions of 2001-2003 and 2008-2010, there is good reason to believe that they will weather future economic storms.
However, Doc Searls points to a more immediate danger that Google presents to consumers: that of the digital monoculture. Google rarely extracts money from users directly. There are only a handful of subscription services offered by them. However, the user base for Gmail is enormous. The same can be said for their advertising services. What would our online life be like if Google went offline tomorrow?
Just looking at one service, Gmail, is illustrative of the scope of the problem. Many companies have outsourced their mail handling to entirely to Google. Recall that in the 1990's, IT staffs spent a considerable amount of time and money setting up corporate email systems. Although the largest companies still do this, many simply outsource this tasks and reap significant savings and reliability over in-house mail systems. Without Google, a very large number of companies and people would not be able to conduct business. Sure, there would be work-arounds: alternate email accounts, telephones, etc. However, this distruption would cost real and measurable dollars.
Perhaps the most immediate effect would be the loss of Google's wonderful, if easily forgotten, search function. To remind those readers recently recovered from a coma, the current neologism for searching online for something is "googling." Imagine the sort of a day you would have if your browser returned a 404 missing page error when accessing http://www.google.com/. That would not be a salad day at all.
The open source community, of which both Searls and Winer are associated, has longed battled against digital monocultures (e.g. IBM, Microsoft, Apple, etc.). Consumers usually benefit from choice (although not always [remember the mess of the home computer market in the 1980s]). Healthy competition promotes innovation and cost-savings. It also creates a healthier ecosystem in which the failure of one entity does not threaten the survival of everyone.
To this end, consumers of free digital products ought to consider how much they depend on these services. I practice what I preach. I pay for Yahoo Mail Plus, which is $20 a year. It's a fair deal: I see no ads and I can use POP mail. That's pretty short money for a service that has yet to have a outage more than 5 minutes in three years. The same goes for my blog hosted on bluehost.com. For $7 a month, I get shell access to very reasonable Linux environment. Of course, there are plenty of free choices for blog hosting these days, but I need to control my content and the context in which it appears. Free services can close shop without notice and there is little consumers can do to retrieve their content.
The dangers of monoculture become readily apparent after a failure. In groups, humans are not noted for their ability to successfully anticipate future disasters. It seems that now, we don't even recall past calamities all that well. One would think that the near fatal collapse of traditional lending institutions who participated in rank speculation would produce a rapid and perhaps onerous regulatory response. However, that has not yet proven to be the case nearly two years after the shock.
From a strictly selfish perspective, I would love to see Google fail completely tomorrow. Business opportunities abound in chaos and fortune favors the bold.
UPDATE: It looks like no one at Yale reads my blog.
I write this blog from a Stabucks in Westford on my new Kindle. I do not pay for the net connection, Amazon does. No, the connection is not fast nor is he Kindle the mot excellent wordprocessing platform, but the price is right and as is the coverage. Apple will announce its tabulet device soon. They should follow Amazon's ubiquituos and free net lead.
Note to self: Example interface files for ubuntu/debian systems.
DHCP:
iface eth0 inet dhcp
Static:
iface eth0 inet static address 1.2.3.4 netmask 255.0.0.0 gateway 1.0.0.1
Now, don't forget it!
Enterprise IT: The Final Frontier
When people use the term "Enterprise Information Technology", what kind of environment are people really talking about? For me, this term refers to the facilitation and management of integrated corporate information by software and hardware for medium to large organizations. Enterprise IT differs from IT at smaller organizations in both scale and the requirement of a lot more management and automation tools. Tools that work quite well in Small Office/Home Office (SOHO) environments often do not scale for larger organizations, do not address the correct problem or cannot integrate with the kinds of systems common to enterprise IT structures.
This is not to imply that SOHO applications are inferior or less robust. Even if application costs were not an issue, there are many enterprise IT applications that are simply not appropriate for SOHO environments due to hardware requirements, installation complexity, administrative overhead or missing network dependencies (for instance, some applications may require SNMP-aware switches that are lacking in most SOHOs).
The key attributes of an enterprise IT solution are scalability and integration. Scalability is the process by which an application may be altered to handle increasing amounts of usage. Notice that I said "process". Too many vendors tout scalability as a feature, which it is not. Application scale through many techniques, but all require the intervention of system administrators to implement. Google search engine, for instance, scales to millions of users searching billions of pages of content, but it requires thousands of commodity servers deployed in standard shipping containers distributed through the world. Without scalability, an application can meet the growing demands of the end-user.
Integration is the other distinguishing feature of IT software. Large IT systems must be able to exchange information with other IT systems in an automated way. For instance, a network management tool needs to be able to retrieve network statistics from smart switches and routers. SSLVPN routers need to be able to talk to a center Authentication and Authorization system, like Active Directory. Without integration, information gets repeated in individual applications. This creates the undesirable "siloization" of information that significately reduces the efficiency of IT departments and can even compromize security.
There are several components of enterprise IT that can be broadly categorized into the areas discussed in the next part of this essay, coming soon.
For complicated reasons, I started playing with Flat Assembler today. Ripping apart someone else's code, I was able to come up with this gem that creates a dialog box with a dead button. When compiled, the executable is a mere 2,500 bytes. I'm not sure I'd want to do an entire app in assembler, but it does seem to cut to the chase of the Windows API very well.
format PE GUI 4.0
entry codestart
include 'win32a.inc'
IDD_MAIN = 100
ID_START = 201
section '.data' data readable writeable
hInstance dd ?
section '.code' code readable executable
codestart:
invoke GetModuleHandle, 0
mov [hInstance], eax
invoke DialogBoxParam, eax, IDD_MAIN, HWND_DESKTOP, \
MainDlg, 0
invoke ExitProcess, 0
proc MainDlg hdlg, msg, wparam, lparam
push ebx esi edi
cmp [msg], WM_INITDIALOG
je .wminitdlg
cmp [msg], WM_COMMAND
je .wmcommand
cmp [msg], WM_CLOSE
je .wmclose
xor eax, eax
jmp .finish
.wminitdlg:
jmp .finish
.wmcommand:
cmp [wparam], BN_CLICKED shl 16 + ID_START
je .startbutton
.wmclose:
invoke EndDialog, [hdlg], 0
.startbutton:
jmp .finish
.finish:
pop edi esi ebx
ret
endp
section '.idata' import data readable writeable
library kernel, 'KERNEL32.DLL',\
user, 'USER32.DLL'
import kernel,\
GetModuleHandle,'GetModuleHandleA',\
ExitProcess, 'ExitProcess'
import user,\
DialogBoxParam, 'DialogBoxParamA',\
EndDialog, 'EndDialog'
section '.rsrc' resource data readable
directory RT_DIALOG, dialogs
resource dialogs,\
IDD_MAIN, LANG_ENGLISH + SUBLANG_DEFAULT, \
main_dialog
dialog main_dialog, 'Dialog test', 0, 0, 150, 50, \
WS_CAPTION + WS_POPUP + WS_SYSMENU +\
DS_MODALFRAME + DS_CENTER
dialogitem 'BUTTON', 'Hello', ID_START, \
50, 15, 50, 20, WS_VISIBLE + WS_TABSTOP
enddialog
If you want to spider a site from a path far down a branch of the document tree, try the following wget invocation:
wget --mirror --relative --no-parent [URL]
This prevents wget from traversing back up the parent and fetching the whole site.
For the last seven years, I have been developing applications for Virtual Machines(VM) as a part of the Leostream Corporation. This market has developed from essentially an academic research project to a multi-billion dollar a year ecosystem with many players. The technologies involved in virtualization are well covered in Wikipedia and elsewhere, but they are summarized here and links to more detailed information is provided. Much of this has been discussed in the standard industry media outlets for some time, but I'd like to offer my perspective on the industry, its players and where things are likely to go.
The Technology
Below, three broad categories of virtualization are discussed that create isolated environments on a single, physical system. The computer system that runs this kind of technology has physical hardware and is typically called a host system. The isolated environments hosted on such systems are called guests and run entirely as software. Although the term virtualization has recently come to mean just paravirtualization, the technologies discussed all can be used to varying degrees for the applications listed later in this article.
Any hardware can be emulated in software. Emulators have been critically important to firmware developers who need to design against drivers for hardware that isn't available to them. Developers of console games and mobile phone applications frequently use emulators to reduce development cycles.
Although emulation allows any machine architecture to be hosted on the user's workstation, the emulated guest typically lacks performance. There is a lot of overhead in the application pretending to be a different machine. For many data center applications of virtualization, emulated x86 machines do not perform well enough. However, for development purposes, emulation is often adequate.
Another common approach to creating an isolated operating system (OS) environment is the use of partitioning. Partitioning takes many forms. The designers of Unix were trying to create a multiuser version of a single time-sharing system called Multics. In a standard Unix system, many users may be logged into the system, but each has his own directory and process space. However, users can see the files and processes of other users on the systems, which can lead to some security concerns.
To improve on the initial security model of Unix, chroot was developed to more fully isolate users more from each. An extreme extension of this idea can be found in User-mode Linux, in which each user appears to have a dedicated copy of the underlying OS. In UML, an unique instance of the filesystem appears to each user. With Virtuozzo Containers or Solaris Containers, Windows or Solaris OSes can be partitioned too.
From the hardware/OS point of view, partitioning is very cheap to implement. Many "guest" containers can be hosted on even commodity hardware with decent performance. The drawback is that the guests are typically copies of the underlying OS. For instance, a Virtuozzo server running on Windows 2003 can only have containers running the same version of Windows 2003 as the host.
For performance and flexibility, paravirtualization is the current state of the art in hosting technology. This technique uses as much of the bare metal components of the host machine do as possible to do work for the guests. Typically, guests see a standard set of virtual hardware, virtual storage and BIOS irrespective of the actual hardware of the host. Virtualized guests can run nearly any OS that can run on the host system's architecture.
Paravirtualization is a flexible compromise between emulation, which allows any OS for any architecture in the guest to run, and containers, that allow a very high number of guests that run a version of the host OS. Early paravirtualization software runs mostly as a usermode application hosted in a well-known OS like Windows or Linux. However, hypervisors have begun to replace these older applications. Hypervisors are tiny OS kernels optimized to run as many VMs as possible. Hypervisors are controlled either through a special console VM or a client application that runs on the administrator's workstation.
Business Drivers of Virtualization
To understand the business-case for virtualization technology, a brief digression into recent Information Technology (IT) history is productive.
At the end of the twentieth century, many businesses had accumulated a not-so-small army of workstations, dedicated servers, minicomputers and mainframes. While there were many business-critical applications that demanded the entire computation and I/O horsepower of the machines on which they were hosted, there also existed a growing class of under-utilized machines. These machines often served lightly-used, but important applications that were not easily be migrated to a new hosts or capable of sharing a single host with other applications.
The proliferation of these one-off machines burdened IT budgets. There are several costs associated with maintaining a machine in a corporate datacenter including the following: power, air conditioning, monitoring, hardware replacement and rackspace. These are recurring costs that quickly exceed the original price of the hardware. Despite this, server applications (particularly web servers and databases) continued to be in demand throughout most organizations through the late 1990's and 2000's.
The utility of ubiquitous computing in an organization had very measurable results in productivity and profit, but the maintenance and security of physical machines introduced really concerns for IT and management. Fortunately, three events happened that would ameliorate these issues: fast processors, fast and ubiquitous networking, and VMware.
VMware has its roots in a research project at Standford. The idea was to bring paravirtualization from the IBM mainframe world into the more limited x86 platform. The i386 could easily emulate 8086 machines in hardware, but could not easily handle its own more sophisticated architecture. VMware created a product that could create pentium-class Virtual Machines (VM) on pentium hosts, with better performance than emulation.
Moore's law fits with virtualization very well. By the mid-2000's, 32-bit Pentium 4s and similar AMD chips could host, with enough RAM, 4-10 VMs. 64-bit CPUs with multiple cores and hardware support for virtualization improved the host's ability to run dozens of guests.
The Applications of Virtualization
There are three general uses of virtualization technology that achieve real business goals today: server consolidation, development and hosted desktops. While any of the virtualization methods previously outlined will work for these applications, some technologies marry better to some purposes than others. Let's first discuss these applications and the business drivers behind them.
Server consolidation is the process of replacing dedicated physical machines with guests running on a few well-provisioned hosts. This is often what brings virtualization into IT departments. By consolidating hardware, maintenance and power savings become immediately clear. Sometimes, end-users see the benefits of consolidation when their application moves from a older host to a VM running on much newer and faster hardware.
Consolidation is most often done with paravirtualization, which often requires very little adjustment of the target legacy application. The process of migrating a physical host to a virtual machine can be done either manually through backups or automated with P-to-V software, like the kind Leostream used to sell. These days, there are a number of commodity tools available to handle P2V conversions.
Application development and quality assurance befits incredibly from any kind of virtualization technology. Developers and QA engineers often need access to a number of machines with specific OS versions and patch levels. Because this use-case generates a lot of sparsely used machines quickly, the need for managing libraries of guests across across dozens of hosts lead Leostream to develop the Virtual Machine Controller in 2003.
This group of users tends to be inside the corporate LAN using high-speed network connections to get console access to their VMs. This profile is very different from the folks in the next group.
Hosted desktops are an attempt by organizations to replace standard workstations with thin client terminals or terminal emulators. By keeping desktops in the data centers, sensitive information stays within the corporate firewall and support costs for workstations reduces significantly. Users benefit also by not being tied to one workstation and can get to their virtual desktops using some remote protocol like RDP, VNC or ICA.
While remote access to machines isn't that revolutionary for most IT workers, there are many challenges to moving the general population to remote desktops. The most pressing is in matching up users with target desktops. While a 1-1 mapping of users to machines isn't too hard to manage, VMs offer much richer schemes with complex policies. As the task of managing users and machine increases, the need for special connection broker software becomes critical.
The basic components of a host desktop deployment are a client, some kind of broker and a target desktops. This simple picture hides a wealth of complexity found in large deployments, in which SSLVPNs, corporate network policies, staffing policies, and departmental hardware and software requirements often create a tangled skein indeed.
Because VMs are often used as the target desktops, VMware created the term Virtual Desktop Initiative (VDI) to describe all the pieces involved in a hosted desktop deployment. However, many companies are using a hybrid approach to hosting desktops that uses physical machines, VMs and Citrix.
Many companies are currently pursuing the hosted desktop market, which some analysts have predicted will be far larger than the server consolidation market. If most large companies decide to replace their employee's workstations with thin clients, the market will be very large.
The Virtualization Layer Players
Below are the four most interesting players offering virtualization layers. This, of course, is an arbitrary list, but a useful one for someone new to the field. There are many, many more players in the virtualization space that offer more specialized offerings.
It would be hard not to mention VMware first in any discussion of virtualization. Although they did not invent the idea (IBM did), they have been the leader in x86 virtualization since 1998. VMware now offers a number of free to use virtualization products including VMware server and ESX 3i, but of which I use at home.
In 2003, Microsoft bought Connectix, makers of Virtual PC and entered the virtualization market. After some early missteps with Virtual Server, Microsoft has bundled Windows 2008 and Windows 7 with a hypervisor that should offer VMware some competition at the commodity end of the market. It is my impression that Microsoft doesn't quite know what they expect this technology to do for them. It's not a simple mass-market product like MS Office. Right now, virtualization is an enterprise IT thing. Perhaps Microsoft is setting the stage for ubiquitous client-side hypervisors, but it's not clear to me how that will benefit them. Virtual Computer, on the other hand, should do quite well in this kind of world.
In 2007, Citrix, who brought remote desktops to Windows in the 1990's, bought XenSource, the commercial arm of the Open Source Xen virtualization layer. Xen has been bundled into XenApp, which used to be called Presentation Server. Citrix appears to be using Xen to capture the server consolidation market, but is hedging its bets in hosted desktops with Xen. Perhaps 2010 will be a crossover year for them.
Parallels is a youngish company that offers a compelling combination of paravirtualization and containers (through their merger with SWSoft). Unfortunately, I don't believe their marketing is cutting through the noise of the bigger elephants in the virtual room. However, they continue to do descent business. Parallel Containers (nee Virtuozzo) is very popular with ISPs who need to cut costs at every opportunity.
There is another group of virtualization vendors that deserve mention. This group includes some well-known names as well as some that you may not have heard of. In your own projects, you may find the wares offered by these companies to be very compelling.
IBM is the original inventor and patent holder of virtualization on their mainframe equipment. The offer a broad spectrum of products and services around virtualization, including the HC12 teradici-enabled blades that offer the very high performance PC-over-IP remote access protocol. If I could afford it, I would buy four of these blades and client pucks for my home.
Sun (or is it Oracle now?) offers the entire VDI stack, just not as a boxed product. Desktops can be hosted on Sun SANs served through Solaris Containers or VirtualBox, brokered through their connection broker and accessed using Sun Ray thin clients. Since I worked a lot with the Sun Ray thin clients, I'm a little biased towards them. Their APL protocol optimizes the desktop session over high-latency networks. If your users are primarily outside the corporate LAN, this is pretty much the only thin client to use.
VirtualIron offers an optimized Xen hypervisor with a proprietary Java management interface. They have succeeded in targeting the server consolidation market. Their product is a snap to install and scales well horizontally.
Virtualization has been a core part of Linux for a while now. Red Hat and Suse both ship with the Xen/KVM/QEMU virtualization suite and tools to easily create Windows VMs on host machines with VT-enabled, 64-bit processors. I have been disappointed that neither company has gone after the hosted desktop market with the zeal I'd expect. But it is still early in that market.
VirtualComputer is leading the charge in client-side hypervisors. This idea is a bit like VMware's ACE in that the user has a VM on his laptop that can be management by corporate IT. The critical difference is that the laptop is running a hypervisor and the user is experiencing the VM as if it were the console. Potentially, you could swap between several VM images. The advantage of this abstraction is that the corporate image can be locked down tightly while allowing a more open OS image to be used at the discretion of the end user.
The Future
The immediate future of virtualization is clear: it will become completely ubiquitous. End users will become comfortable connect to remote machines and the methods to make those connections will become faster and more transparent. Service providers like Comcast and Verizon will rent VMs to their Internet or cable TV customers, who will use their digital receiver as a thin client. The technology to do this exists today.
Another fallout of pervasive VMs will be the rise of virtual appliances. Already, there is a small cadre of networking applications out there. I see virtual appliances becoming the dominant paradigm for shipping certain kinds of applications. Think of this is a kind of Software as a Service to-go. I have first hand experience developing for virtual appliances and the benefits to the publisher are many. What's lacking are installation tools that making virtual appliances install like traditional ones. But I'm sure someone is tackling that problem now.
Virtualization provokes a disturbing question for OS vendors: who controls the hardware? Traditionally, the OS made the hardware accessible to applications. Hypervisors force the OS to a higher level in the application stack. Will hypervisors soon appear only in BIOS or hardware? If so, what does that mean for OS vendors? Nothing good.
Makers of peripheral hardware may also lament the rise of VMs. You cannot install a new video card in a virtual machine, for instance. I suppose there may arise an open standard that allows VMs to more fully access physical hardware, but none now exists.
Software vendors are the clear benefactors of virtualization. There's a whole new class of problems to solve.
It seems my spam filter has been very aggressive lately. If I haven't responded to your email, I probably only just fished it out of the trash.
Just a reminder from Tech Recipes:
- T-Mobile: phonenumber@tmomail.net
- Virgin Mobile: [phonenumber]@vmobl.com
- Cingular: [phonenumber]@cingularme.com
- Sprint: [phonenumber]@messaging.sprintpcs.com
- Verizon: [phonenumber]@vtext.com
- Nextel: [phonenumber]@messaging.nextel.com
Where, oddly enough, [phonenumber] is the 10-digit phone number.
Behold: Ubuntu Linux on the XO laptop!
Since I already had an 8GB SD disk for my XO, I had the storage capacity to handle the new distro. I did need to apply for a developer key, which is needed to unlock the openfirmware "BIOS" system build into the XO. For reasons that aren't clear, it took 24 hours to generate the key.
While I appreciate the accomplishment of the default Sugar interface (I've got the latest 707 build running), I find that it gets in my way more than not. And it seems a little too pokey.
Ubuntu has a build just for the XO that uses the lightweight XFCE shell and comes with Firefox. Aside from that, there's not much in the way of apps for this system. For instance, no Flash support for Firefox. Also, I'm sad that I lost the camera functionality. And pygame isn't installed. All of these things are correctable, of course. The XO makes a pretty servicable netbook.
Now you can see more unsettling pictures of me than ever before!
In the software/hardware paradigm that bifurcates humanity, I'm squarely in the software camp. However, I have successful created a CAT 5e run of ethernet cable that goes from the basement to my second floor office terminating in a real, honest to goodness RJ-45 jack in my wall. I couldn't be more proud. If you attempt to install wired ethernet in your home, be aware of the following:
1. Running the cable is the hardest part of the job
If you have a new home, running cable is as easy as getting into the walls before the drywall/plaster is installed. Otherwise, you'll need snake the cable through those unfinished walls in your house, like those found in basements and attics. You could also rip open your finished walls, install the wiring and patch it later, but you're clearly more butch than I. In my case, there was existing speaker wire that went from the basement to the attic. I simply attached one end of a 100' spool of CAT 5e cable to top of the speaker wires with copious tape. I then pulled (carefully) on the wire from the basement to get whatever gravity-assist I could. Your mileage will vary.
2. Get the right tools
You will need an ethernet jack punch down tool. You can get this at Lowes, Microcenter or many other places online. Do not confuse this with a crimping tool, which is designed to fit a male RJ-45 coupling onto the end of an ethernet cable. Punch down tools force the individual wires of ethernet into specific recepticals on the RJ-45 jack. You should not strip these wires before punch down. The act of punch down removes the casing around the wire. Some tools will cut the extra length of wire for you, but any wire cutting tool will work adequately for the job.
You might also need fish tape to help you run the ethernet cable into the hard to reach areas of your wall.
3. Read the mating diagram on the RJ-45 jack
Ethernet cabling consists of 4 twisted pairs of wires, which are typically colored brown, blue, orange and yellow. Each color has a mate that striped with white for a total of eight wires. To attach to an RJ-45 jack, each of these eight wires needs to be punched into the right saddle. Most RJ-45 jacks are sold with a diagram that explicitly shows where each wire goes. Follow the directions carefully. You cannot fudge this part.
4. Unless you have a house older than 50 years old, use new construction low voltage brackets
Low voltage brackets are the bits that are nailed or screwed into the wood of a joist so that the face plate can be attached to the wall. These seemingly simply devices come in a broad variety of forms. One of the dimensions of variation is in how the bracket attaches to the wall. Brackets that have screws or nails go parallel to the face plate are called "new construction" mounted. Brackets that have nails or screws that are perpendicular to the face plate are said to have "old construction" mounting. I can't really think of a situation where old construction mounting works, but as I said, I'm not a hardware guy. That's also why I bought an old construction mounting first. Live and learn.
It took me three days to do this admittedly simply installation, because I didn't have all the right tools and parts initially.
This is a great project for all you home-owning nerds out there. There are many more detailed web resources out there that explain the process in more detail than I've provided. Also, check out You Tube for helpful videos. There was at least one that shows how to punch down an RJ-45 jack correctly.
It seems that a few of you are using or trying to use the XML::RSS::Podcast module I published on this blog. As was mentioned there, modern version of XML::RSS do not support the encode function required for generating well-formed XML documents.
That's all changed thanks to Rohan Carly. [Add your thanks to Rohan in the comments.]
I present the complete version of XML::RSS::Podcast below. If you find this module
useful, I'll attempt to create a CPAN module for it.
package XML::RSS::Podcast;
use XML::RSS;
use HTML::Entities qw[encode_entities_numeric encode_entities];
@XML::RSS::Podcast::ISA = qw[XML::RSS];
our $VERSION = q[1.1];
# encode by Rohan Carly
# Stolen from XML::RSS::Private::Output::Base;
sub encode {
my ($self,$text) = @_;
return unless defined($text);
my $encoded_text = '';
while ($text =~ s,(.*?)(<!\[CDATA\[.*?\]\]>),,s) {
# we use &named; entities here because it's HTML
$encoded_text .= encode_entities($1) . $2;
}
# we use numeric entities here because it's XML
$encoded_text .= encode_entities_numeric($text);
return $encoded_text;
};
sub as_string {
my $self = shift;
return $self->as_podcast_rss;
}
sub as_podcast_rss {
my $self = shift;
my $enc = $self->{encoding};
my $output = qq[<?xml version="1.0" encoding="$enc"?>
<rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
version="2.0">];
$output .= $self->podcast_start_channel;
for my $i (@{$self->{items}}) {
$output .= $self->podcast_item($i);
}
$output .= $self->podcast_end_channel;
return $output .= "\n</rss>\n";
}
sub podcast_start_channel {
my $self = shift;
my @fields = qw[ttl title description link language
pubDate lastBuildDate creator
webMaster copyright
];
my @image_fields = qw[title url description link
width height];
my @itunes_fields = qw[subtitle author summary
image explicit]; # Thanks Rohan
my $output = "<channel>\n";
for my $f (@fields) {
if (defined($self->{channel}->{$f})) {
my $s = $self->encode($self->{channel}->{$f});
$output .= "\t<$f>$s</$f>\n";
}
}
my $seen_image = 0;
for my $f (@image_fields) {
if (defined($self->{image}->{$f})) {
unless ($seen_image) {
$output .= "\t<image>\n";
$seen_image = 1;
}
my $s = $self->encode($self->{image}->{$f});
$output .= "\t\t<$f>$s</$f>\n";
}
}
if ($seen_image) {
$output .= "\t</image>\n";
}
# Owner name/email not handled
for my $f (@itunes_fields) {
if (defined($self->{channel}->{itunes}->{$f})) {
my $s=$self->encode($self->{channel}->{itunes}->{$f});
$output .= "\t<itunes:$f>$s</itunes:$f>\n";
}
}
# Rohan's sub-cat handling code
# Expects an array: [category, sub-category]
if (ref $self->{channel}->{itunes}->{category} eq 'ARRAY') {
my $major = $self->encode(${$self->{channel}->{itunes}->{category}}[0]);
my $minor = $self->encode(${$self->{channel}->{itunes}->{category}}[1]);
$output .= qq[\t<itunes:category text="$major">\n];
$output .= qq[\t\t<itunes:category text="$minor">\n];
$output .= qq[\t\t</itunes:category>\n];
$output .= qq[\t</itunes:category>\n];
}
return $output . "\n";
}
sub podcast_end_channel {
return "</channel>\n";
}
sub podcast_item {
my $self = shift;
my $item = shift;
my @fields = qw[title guid pubDate description link];
my @itunes_fields = qw[author subtitle summary
duration keywords explicit];
my $output = "\t<item>\n";
for my $f (@fields) {
if (defined($item->{$f})) {
$s = $self->encode($item->{$f});
my $perma = "";
if ($f eq "guid") {
$perma = qq[isPermaLink="false"];
}
$output .= "\t\t<$f$perma>$s</$f>\n";
}
}
if (ref $item->{enclosure}) {
$output .= "<enclosure";
for my $f (qw[url length type]) {
if (defined $item->{enclosure}->{$f}) {
$output .= sprintf("$f=%s", $self->encode($item->{enclosure}->{$f}));
}
}
$output .= "/>";
}
for my $f (@itunes_fields) {
if (defined $item->{itunes}->{$f}) {
$s = $self->encode($item->{itunes}->{$f});
$output .= "\t\t<itunes:$f>$s</itunes:$f>\n";
}
}
return $output .= "\t</item>\n";
}
1;
For an example of usage, please see the previous post cited above. Report bugs in the comments or through email. Thanks.
There are two utilities I've found that let Windows users manage files on EXT-2 (and so EXT-3) filesystems.
The first is a filesystem driver that let's you mount EXT-2 drives just like any other VFAT, NTFS drive. Appears to work like a champ.
The other is a less invasive tool that allows you explore EXT-2 filesystems without mounting them.
I have a network filer running linux with RAID-1, which is great. I have a SATA drive in a USB enclosure for backups. Now, I can directly access the backup drive from Windows, should the filer fail.
EXT-2/3 is nice because it handles large files and large capacities. VFAT just can't handle the new drive capacities without magic (even though the linux vfat drive can handle 300GB drives without complaint).
Although the XO laptop is constrained on memory (256M), it has a lot of hardware features.
I shoved a 2GB USB drive into the XO, along with an apple keyboard and an MS optical mouse. I wanted to record a video with the Record activity, but I was low on disk space. Most of the pre-installed "activities" save files in /home/olpc/.sugar/default/org.laptop.[ACTIVITY]. I simply repointed org.laptop.RecordActivity/instance to /media/KINGSTON. That gave me room for videos. Unfortunately, the RecordActivity seems to allow only 45 seconds of video to be recorded. That's not really true. It only *offers* you 45 seconds. Since most of the sugar UI is python, and the RecordActivity is python, I was able to add the following to /usr/share/activities/Record.activity/constants.py:
DURATIONS.append(600)
That changed the UI to allow 10 minute recordings.
I'd like to hack the UI to tell me the filename of the recording. That's not too much to ask, is it?
Also, I'm using my emacs blogging tools from the XO to write and publish this. That means I had to shove Perl and GPG on here. It's getting pretty crowded on the 1GB drive.
Another note to jjohn: since you can't figure out how to import keys into gpg, just move the *gpg files in the blog project to where gpg expects them. Not you. No one cares where you think these files should go. This is the second blog post I've had to make to you about this. Perhaps you need to write an article about using GPG so that you don't forget again? Still, GPG bets the hell out of X509 certificates. What a bureaucratic nightmare those are.
This article on using the little known "control" directive (USERPASSWORDS2) to access control panels I've never seen shows that GUIs are as mysterious as command line interfaces.
Also see this article for even more goodies.
Amazing...
UPDATE: Wait, it gets better. Hack the registry to force an automatic login. Seems like someone will create a .REG file to do this...
While it's not entirely perfect, CutePDF does allow easy PDF creation on Windows for free. And you have to like that. It installed in seconds.
I'd love to control the bookmarks and links in the PDF output, but that's why there are professional applications to do that.
Well, here I am at the Javaroom in Chelmsford with my XO. It's a cool, if underpowered little device. It continues to draw more attention than even those fancy mac Air notebooks. I think its the lime green rabbit ears. Or maybe it's my own rabbit ears.
The file upload HTML widget works weirdly on the Browse Activity. Recall that the XO attempts to hide the filesystem from the user and instead has a "journal" that remembers everything you did. So the file upload, instead of presenting a file chooser dialog, presents this weird version of the journal.
Must get firefox on this bad boy someday...
Still, I can blog from my XO.
Hey, Windows Users!
Are you annoyed with the seemingly constant prodding from Apple to upgrade your iTunes/Quicktime installation with newer updates? Ever been typing away happily in an application only to have Apple's nagware grab your keyboard and mouse focus away?
Stop being a victim and fight back!
You can disable this misfeature at the root. Click on your trusty Start button and get to Programs > Accessories > System Tools > Scheduled Tasks. Your screen will look something like the screenshot above. Either delete the AppleSoftwareUpdate item all together or simply disable it by right clicking on the icon and going to Properties.
Push verses Pull technologies: I think we know where I stand on this issue.
I'm writing this from my new XO laptop, from the One Laptop Per Child project, which I'm not linking to, out of sloth.
While the UI takes a while to get used to, this device is a great if your looking for a portable terminal. It reminds me favorably of the Mac Classics.
I'll post more about this remarkable device later. For now, I want to play with it!
python -m pdb [script]That is all.
Dear Lazyweb,
I have a few Dell PowerEdge 2450's and 2650's that I'd like to boot through a Wake on LAN facility. However, I cannot figure out where in the BIOS one enables WOL.
Perhaps one fine readers could point in the right direction?
Note: these are eBay machines, so I don't have the original install disks.
Thanks in advance, emergent intelligence bio-mechanical entity called "World Wide Web."
For no particular reason, I'd like to record my preferred order for tackling the minibosses at the beginning of the NES game Megaman 2. Pay close attention.
- Airman: Use your shooter
- Metalman: Use your shooter
- Woodman: Use metal blades
- Bubbleman: Use metal blades
- Crashman: Use air attack
- Heatman: Use bubble attack
- Flashman: Use heat attack or shooter
- Quickman: Use crash bombs
If you follow this order, you should have all the special weapons you need when you need to get 1UPs and other goodies.
Good hunting.
This is my current hobby. It's giving me a much clearer understanding how to actually use assembler to do common programming tasks. While there is little call for 6502 programmers these days, I feel like I can better understand general purpose CPU programming now.
This code is a moderate hack of the kernel code shown here.
;; Built with: http://www.atari2600.org/DASM/DASM22010b.zip
;; bin\DOS\DASm.exe playfield2.asm \
;; -IDASM\Machines\ATARI2600 \
;; -f3 -v5 -o..\ROMS\a.bin
;;
processor 6502
include "VCS.h"
include "macro.h"
PATTERN = $80 ; first byte of RAM
SHAPE = $55
BGCOLOR_ADDR = $81
BGCOLOR = $0A
BGTIMER_ADDR = $82
BGTIMER = 40
PLAYFIELD_COLOR = $0
TIMETOCHANGE = 20
SEG
ORG $F000
Reset
;; Zero out RAM, TIA
;; RAM - $80-$FF
;; TIA - 0 - $7F
ldx #0
lda #0
RAMLoop
sta 0,x
inx
bne RAMLoop
;; -------
;; Init playfield
;; -------
lda SHAPE
sta PATTERN
lda #PLAYFIELD_COLOR
sta COLUPF
ldy #0 ; animation clock
lda #$10 ; Remember,
; high nibble
; is read
; backwards
sta PF0
lda #$80 ; Entire byte
; read backwards
sta PF2
lda #$01
sta CTRLPF
;; setup the bgcolor stuff
ldx #BGTIMER
stx BGTIMER_ADDR
ldx #BGCOLOR
stx BGCOLOR_ADDR
stx COLUBK
StartOfFrame
; Start of vblank processing
lda #0
sta VBLANK
lda #2
sta VSYNC
; 3 scanlines of VSYNCH
sta WSYNC
sta WSYNC
sta WSYNC
lda #0
sta VSYNC
; 37 lines of vertical blank
ldx #$CA
VertBlank
inx
sta WSYNC
bne VertBlank
;; Playfield animation
iny
cpy #TIMETOCHANGE
bne NOTYET
ldy #0 ; reset ani clock
;; bit shift the pattern
asl PATTERN
bne NOTYET
;; Reset pattern
lda SHAPE
sta PATTERN
NOTYET
lda PATTERN
sta PF1
;; Change the color by adding
;; the aniclock tick to the
;; base color
clc
tya
adc PLAYFIELD_COLOR
sta COLUPF
;; 192 scanlines of picture
ldx #$C0
MainPicture
sta WSYNC ; strobe Wait for Sync
dex
bne MainPicture
;; Is it time to change
;; the BG COLOR?
dec BGTIMER_ADDR
bne LOAD_BGCOLOR
;; Yes - restore timer,
;; change color
ldx #BGTIMER
stx BGTIMER_ADDR
asl BGCOLOR_ADDR
bne LOAD_BGCOLOR
;; Reset BGCOLOR
ldx #BGCOLOR
stx BGCOLOR_ADDR
LOAD_BGCOLOR
ldx BGCOLOR_ADDR
stx COLUBK
;; Vertical Blanking
lda #%01000010
sta VBLANK
; 30 scanlines of overscan...
; #D2 = 226, 256-30
ldx #$D2
OverScanLoop
inx
sta WSYNC
bne OverScanLoop
jmp StartOfFrame
ORG $FFFA
;; Interrupt Vectors
.word Reset ; NMI
.word Reset ; RESET
.word Reset ; IRQ
END
I'd like to make some playable game for the atari 2600 some day. That would bring my childhood dreams to life.
If you are using the Perl module XML::RSS in the following way:
my $R = XML::RSS->new;
foreach my $url (@urls) {
my $content = wget($url);
$R->parse($content);
}
In older versions of the XML::RSS module, this code worked fine. However, if you have upgraded the module recently, you might have noticed the error message:
Modification of non-creatable array value attempted, subscript -1 at /usr/local/lib/perl5/site_perl/5.8.5/XML
/RSS.pm line 792.
It is not the feed that has gone rancid on you, but the library. Try instantiating a RSS object within the loop like this:
foreach my $url (@urls) {
my $content = wget($url);
my $R = XML::RSS->new;
eval{$R->parse($content)};
if ($@) {
warn("Parse error: $@");
next;
}
}
I've added some "exception" handling for free in this example so that parse errors don't blow up your program.
I'm afraid I did not dive into XML::RSS.pm to figure out the problem, but if someone with that knowledge wishes to post below, I'm sure won't be the only one who welcomes enlightenment.
This is an abbreviated primer to remind me of this common, yet difficult to find documented, procedure.
The problem: You have an Active Directory domain with groups of users on it that you want to allow to log into varies local machines through RDP. You have tried to add the domain group to the local "Remote Desktop Users" group on a target terminal server, but you cannot do this through the Manage Computer MMC.
The solution: use the net command on the target machine like this:
Assuming you have RDP enabled on that machine, you should be able to RDP to it and use the credentials of a user in the domain group you just added to log in.
I know what you're thinking: "why not just add the domain builtin group 'Remote Desktop Users' to the local group?" You cannot add domain builtins to the local groups. I think this has to do with the underlying LDAP schema of AD. The "net localgroup" command is looking the CN=Users sub-tree of AD, it seems. Put your OU or groups in there.
Of course, this solution makes you visit each target terminal server/desktop. You could script this with ssh or, more Windows-y, WMI, I think. You might even be able to do this stuff with Perl or python to visit all the machines too. There's also a Group Policy manager from Microsoft that should make this easy to manage central, but I couldn't get that software.
«It's your lucky day! I'm giving away an original Nintendo Entertainment System with three cartridges: Super Mario Bros. 3, Krusty's Funhouse and The Mafat Conspiracy. Includes AC adapter and RF adapter for old sk00l TV fun.The Catch? I don't know that the console works anymore. I didn't spend a lot of time debugging the problem. The carts probably are OK. I'm not interested in shipping this stuff (around 10 lbs.), but if you are serious about it, we'll do some further negotiation. Otherwise, we can meet in a public place for a drop off.
So, if you need spare NES parts, dinner is prepared!»
If you are interested, email me directly.
UPDATE: The NES is taken. Sorry Horny 4 NES. Maybe next time.
This is a reminder to myself.
If you need to reset the IP address of the HP LJ printer named in the subject line, turn on the printer with the JOB CANCEL button pressed for 7-10 seconds. More troubleshooting can be found here.
Perl now has three implementions of XML::RPC, Frontier::RPC2, SOAP::Lite and RPC::XML. Each is interesting and somewhat broken in its own way.
Frontier::RPC2 0.06 has issues with Boolean, iso8601 and Base64 because these classes have a bug in their constructors.
Given:
sub new {
my $type = shift;
my $value = shift;
return bless \$value, $type;
}
Instead of checking to see if $type is a reference, this code blesses. If a reference is passed in, the blessed class is sometime like "Frontier::RPC2::Base64=SCALAR(0x65432)". This breaks code later than encodes/decodes these values into XML. Perltoot has the solution to this problem
RPC::XML is an interesting module in that it brings type checking to Perl, in a sense. When creating an XML-RPC server, each remote procedure has to the arguments it accepts along with its return type. This is call a signature. Although type checking is unPerlish, it is very helpful when dealing with other languages that are subPerl. More docs on signatures would be great.
Finally, SOAP::Lite brings its own bad self to the XML-RPC party. Although in my testing it works with all the XML-RPC datatypes, the programming interface is similar to SOAP::Lite (not surprisingly). It is a style which takes a bit of getting usef to.
The good news is that, at least from a client level, all these modules seemed to be compatible. That is, a Frontier::Client can talk to an XMLRPC::Lite server.
More to come.
Some good friends of mine stongly urged me to stop watching CNN for a spell. Apparently, I'm becoming hysterical. How they isolated the cause of this hysteria to CNN is beyond my feeble powers to discern. Nevertheless, I going to avoid CNN and the Boston Globe for 5 days or so.
The effect of this isolation is that my journal entries will devolve into increasing esoteric topics. Hence, the following.
In this age of increasing complex X desktop environments, I've taking a courageous step back in time by tweaking my .xinitrc and .twmrc files. That's right, my current window manager of choice is the tab window manager. Take a second to try out these settings. (note: hard coded for 1028x768 screen resolution).
.xinitrc
xsetroot -solid gray85 &
xterm -fg blue -bg white -g 80x25+0+0 &
xterm -fg red -bg white -g 80x25+0-0 &
xterm -fg orange -bg white -g 80x25-0+0 &
xclock -digital -update 5 -bg white -fg gray45 -g -0+420 &
exec twm
.twmrc
ShowIconManager
IconManagerGeometry "191x200-0+468" 2
IconifyByUnmapping
RandomPlacement
NoGrabServer
RestartPreviousState
DecorateTransients
TitleFont "-misc-fixed-medium-r-normal--*-90-*-*-*-*-*-*"
ResizeFont "-misc-fixed-medium-r-normal--*-90-*-*-*-*-*-*"
MenuFont "-misc-fixed-medium-r-normal--*-90-*-*-*-*-*-*"
IconFont "-misc-fixed-medium-r-normal--*-90-*-*-*-*-*-*"
IconManagerFont "-misc-fixed-medium-r-normal--*-90-*-*-*"
Color
{
BorderColor "black"
DefaultBackground "white"
DefaultForeground "gray60"
TitleBackground "white"
TitleForeground "gray60"
MenuBackground "white"
MenuForeground "gray60"
MenuTitleBackground "white"
MenuTitleForeground "gray60"
IconBackground "white"
IconForeground "gray60"
IconBorderColor "black"
IconManagerBackground "white"
IconManagerForeground "gray60"
}
MoveDelta 3
Function "move-or-lower" { f.move f.deltastop f.lower }
Function "move-or-raise" { f.move f.deltastop f.raise }
Function "move-or-iconify" { f.move f.deltastop f.iconify }
Button1 = : root : f.menu "defops"
Button1 = m : window|icon : f.function "move-or-lower"
Button2 = m : window|icon : f.iconify
Button3 = m : window|icon : f.function "move-or-raise"
Button1 = : title : f.function "move-or-raise"
Button2 = : title : f.raiselower
Button1 = : icon : f.function "move-or-iconify"
Button2 = : icon : f.iconify
Button1 = : iconmgr : f.iconify
Button2 = : iconmgr : f.iconify
menu "defops"
{
"Twm" f.title
"Iconify" f.iconify
"Resize" f.resize
"Move" f.move
"" f.nop
"Focus" f.focus
"Unfocus" f.unfocus
"Show Iconmgr" f.showiconmgr
"Hide Iconmgr" f.hideiconmgr
"" f.nop
"CDPlayer" f.exec "exec xplaycd &"
"Gdict" f.exec "exec gdict -a &"
"Netscape" f.exec "exec netscape &"
"Xterm" f.exec "exec xterm &"
"" f.nop
"Kill" f.destroy
"Delete" f.delete
"" f.nop
"Restart" f.restart
"Exit" f.quit
}
IconManagerDontShow
{
"xclock"
}
I like fonts although I'm not typographer. After reading the HOWTO, I managed to get both normal X and VNC (pass '-fp unix:/7100' to vncserver if you're using redhat 7.1 or xfs) to see verdana, one of the pretty things to come out of Microsoft. Even crusty old xfontsel can see all my new fonts. Delic!
Oh yeah. Use.perl.org is now in verdana for me. Coolio.
Here's the deal: you're on win32, you need to fork() and you're using Perl. Since version 5.6, win32 Perl has had fork() emulation. That's good. It's emulation (accomplished at the interpreter level with threading), so standard Unix tricks like using a signal handler to reap children and control the number of child process don't work. That's bad.
Here then is my Win32 Perl recipe for the week: Limiting forked child "processes" under Win32 Perl (versions 5.6 and higher).
use strict;
use POSIX ":sys_wait_h";
use constant MAX_KIDS => 4;
my (%children, $quit) = ((), 0);
print "[$$] Entering main loop\n";
while (!$quit) {
reap(\%children);
if ((keys %children) <= MAX_KIDS) {
print "[$$] Forking child $_\n";
if (my $pid = fork()) {
$children{$pid} = 1;
} else {
do_child();
exit;
}
} else {
$quit = 1;
}
sleep(1);
}
# reap existing kids
while (keys %children) {
reap(\%children);
sleep(1);
}
print "All children reaped\n";
#--------
# sub
#--------
sub reap {
my ($kids) = @_;
for (keys %{$kids}) {
print "[$$]child '$_' reapable?\n";
next if waitpid($_,WNOHANG()) != -1;
print "[$$]child '$_' reaped\n";
delete $kids->{$_};
}
}
sub do_child {
sleep(1) for 0..10;
print "[$$] done\n";
}
Note that you can do other work in the main reaping loop.
The main loop is really the service state loop for a Win32 service. Also note that reaping the child "processes" isn't strictly necessary. Not only are the children not real processes (they are interpreter threads), but also no zombies can be created if the parent process exits without calling wait(). All child "processes" will be terminated with the parent. So you've got that going for you.
See David Roth's web site, books or articles for more excellent details on the devilish art of creating Win32 services with Perl.
[Original use.perl.org post and comments. Minor cleanup on 11/30/2007.]
UPDATE: It's nice to see that this script still works in 2009, four years after I wrote it. Must have done something right.
Whether pornography is psychologically damaging, socially corrosive or just plain nasty, it is clear that at least for men, it is a strong attractor and motivator. It is time we left behind antiquated nineteenth century morality and boldly embraced this core driver of male behavior to fill the more serious deficit of quality programmers in the U.S.A.
To this end, I submit my own work, a perl script that fetches the publicly available gallery at IShotMyself.com. The real benefit of this script (well, at least secondarily) is that it is my first concerted use of Andy Lester's WWW::Mechanize. Because of the positive reinforcement generated by this script, I am more likely to use this module in other, less pornographic work -- and so I move the Wheels of Industry forward! Adam Smith's Invisible Hand of Capitalism guides me!
Briefly, the script fetches the front page and looks for a link labled "FOLIO [\d+]". This page is then fetched and links that start with "javascript:" are culled. By adding on "&m=img" to these links, it is possible to get to the actual picture in question. These images are stored in a directory on my Winders box, hence the funny path names.
Based on the impressive success of this experiment, I recommend that we teach children to read by giving them digests of Penthouse Forum. We must always be thinking of the children!
I further recommend that we ease the suffering of the impoverished by eating their babies.
More Good Ideas (™) to come...
use strict;
use WWW::Mechanize;
my $base_url = qq[http://www.ishotmyself.com];
my $main_page= qq[/public/main.php];
my $dest_dir = "ishotmyself";
print "Fetching $base_url/$main_page\n";
my $B = WWW::Mechanize->new;
$B->get(qq[$base_url/$main_page]);
unless ($B->success()) {
die "Couldn't fetch main page: ", $B->status();
}
print "Main page fetched\n";
print "Dumping folio links\n";
my $todays_gallery;
my $gallery = "unknown";
foreach my $l ($B->links()) {
next if $l->text() !~ /folio \[\d+\]/i;
$todays_gallery = $l;
($gallery = $l->url) =~ /([^=]+)$/;
$gallery = $1;
print "\t", $l->text(), " : ", $l->url(), "\n";
}
# find today's folio name
print "Fetching ", $todays_gallery->url, "\n";
$B->get($todays_gallery->url);
unless ($B->success()) {
print "Couldn't fetch '" . $todays_gallery->url(),
"' : ", $B->status(), "\n";
print $B->content, "\n";
}
#open OUT, ">out.txt";
#print OUT $B->content, "\n";
#close OUT;
print "Dumping folio links\n";
my @found;
foreach my $l ($B->links()) {
next if $l->url !~ /^javascript:popupLandscape/;
print "\t", $l->text(), " : ", $l->url(), "\n";
# unjavascript!
(my $url = $l->url()) =~ /\('([^']+)'\)/;
if ($1) {
push @found, "$1&m=img";
} else {
warn "Couldn't unjavascriptify!\n";
}
}
unless (-d $dest_dir) {
mkdir $dest_dir;
}
$dest_dir .= "\\$gallery";
unless (-d $dest_dir) {
print "Creating $dest_dir\n";
mkdir $dest_dir || warn "mkdir $dest_dir failed: $!";
}
print "chdir $dest_dir\n";
chdir $dest_dir;
print "Fetching public images\n";
my $cnt = 1;
for my $l (@found) {
$B->get($l);
unless ($B->success()) {
warn "picture fetch failed: ", $B->status, "\n";
next;
}
my $imgfile = sprintf("${gallery}_%03d.jpg", $cnt++);
if (-e $imgfile) {
print "\tOVERWRITING $imgfile\n";
}
if (open(OUT, ">$imgfile")) {
binmode(OUT);
print "\tWriting $imgfile\n";
print OUT $B->content();
close OUT;
} else {
warn("open $imgfile failed : $!");
}
}
print "done\n";
Here's another boring programming note to myself. Please note that I had to break lines of code for formatting reasons. Please use this syntax as a guideline. Don't try to cut and paste it.
Ajax is all the rage now. To me, it's just web services done with Javascript and Javascript blows bigger chucks than Java (and we all know how I feel about that).
In playing with the simple code on Mozilla's site, I realized that the browser complicates using Ajax as a simple RPC mechanism. The reason? Browsers are inherently multithreaded applications and so all the RPC Ajax calls must be asynchronous. That means that you can't directly port something like XML-RPC or SOAP (both of which are more or less synchronous) to Ajax. Browsers run javascript on events. When the Ajax response finally comes back to the browser, a callback function is needed to continue processing the response.
That's not so bad, but what if you really want to do this:
<script>
var uid = ↵
RPCRequest('server.php?action=getUID?username=jjohn');
</script>
The problem is that RPCRequest returns as soon as the request is made, not when the response comes back.
Here's the typical Ajax code (via the Mozilla article) that makes the request and parses the response:
// file: ajax.js
// make the request, register the callback
function makeRequest(url, cb) {
var http_request = false;
if (window.XMLHttpRequest) { // Mozilla, Safari, ...
http_request = new XMLHttpRequest();
if (http_request.overrideMimeType) {
http_request.overrideMimeType('text/xml');
}
} else if (window.ActiveXObject) { // IE
try {
http_request = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
}
}
if (!http_request) {
alert("No HTTP object!");
return false;
}
http_request.onreadystatechange = function() {
alertContents(http_request, cb); };
http_request.open("GET", url, true);
http_request.send(null);
return http_request;
}
// the callback to handle the response from the server
function alertContents(http_request, cb) {
try {
if (http_request.readyState == 4) {
if (http_request.status == 200) {
// parse out response
var resp = http_request.responseXML;
var val = ↵
resp.getElementsByTagName("response"). ↵
item(0).firstChild.data;
cb(val);
} else {
alert('There was a problem with the request.');
}
}
} catch (e) {
alert("Exception: " + e.message);
}
}
In this code, I expect a rather meager XML response that has only a response tag.
I got the notion that what I should do with the response handler is send it a callback from the caller to handle the parsed response. Something like the following:
<!-- file: test.html -->
<html>
<head>
<script type="text/javascript"
language="JavaScript" src="ajax.js">
</head>
<body>
<span
style="cursor: pointer; text-decoration: underline"
onclick="makeRequest('server.php', ↵
function(x){ alert(x) })">Make a request</span>
</body>
</html>
Again, this works fine when the anonymous function has a globablly defined
function like alert(). But what if you want to define a function
in the caller's HTML page and have it called from code in the ajax.js page?
I haven't figured that out yet (but see below).
The problem is that the scope of functions defined on the calling page is not visible (it seems) to the functions in the ajax.js page. I think I could get around this using fully-qualified DOM names ore something nutty like that.
I want to use objects, but of course, Javascript has classless objects which don't help here. I guess I could define the callback behavior for each call, but that seems dumb.
I need to sleep on this a bit.
UPDATE: What a difference eight hours of sleep makes!
After digging around the javascript books I have, I came up with this more object-oriented version of the ajax code that removes the callback function in favor of a user-overridden method. That nicely divides that generic RPC tasks from the caller-specific ones. The code appears to run in both Firefox and Internet Explorer on my XP box, but I'm sure it will break in Opera and Safari.
Let's look at the new ajax.js file. This defines an "class"
call RPCClient.
/* Much of this is from
* http://developer.mozilla.org/en/docs/AJAX:Getting_Started
*/
function makeRPCRequest(url) {
if (window.XMLHttpRequest) { // Mozilla, Safari, ...
this.http_request = new XMLHttpRequest();
if (this.http_request.overrideMimeType) {
this.http_request.overrideMimeType('text/xml');
}
} else if (window.ActiveXObject) { // IE
try {
this.http_request = ↵
new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
this.errstr = e.message;
}
}
if (!this.http_request) {
alert("No HTTP object!");
return false;
}
// Can't use 'this' in callback.
// Gets confuses with http_request.
var thisObj = this;
this.http_request.onreadystatechange = function() {
thisObj.watchResponse();
};
this.http_request.open("GET", url, true);
this.http_request.send(null);
}
function parseRPCResponse() {
try {
if (this.http_request.readyState == 4) {
if (this.http_request.status == 200) {
// parse out response
var resp = this.http_request.responseXML;
var val = ↵
resp.getElementsByTagName("response") ↵
.item(0).firstChild.data;
this.handler(val);
} else {
alert('There was a problem with the request.');
}
}
} catch (e) {
alert("Exception: " + e.message);
}
}
// define the RPCClient object
function RPCClient () {
this.errstr = false;
this.http_request = false;
}
// navigator 3 bug workaround for prototype
new RPCClient();
RPCClient.prototype.send = makeRPCRequest;
RPCClient.prototype.watchResponse = parseRPCResponse;
RPCClient.prototype.handler = function (x) ↵
{ alert("Default handler.n" + x) };
The idea of this code is that the user overrides the handler
method to do something useful with the value that comes back from server.
Most of this code just replaces procedural elements with OO mechanisms. However, notice this stupid looking assignment:
var thisObj = this;
I had to do this because of scoping behavior that, as a Perl coder, I find
nutty and wrong. The trouble comes when I want to use the current object in
the http_request callback/method onreadystatechange. Like most
method declarations, I use an anonymous function to override the default
action. When I try to use the this in the callback/method, it
then refers to http_request object, not the RPCClient object.
Ugh.
I guess JavaScript has to work this way, since it doesn't have proper
namespaces or Perl closure rules. How can I tell JavaScript which object
this refers to? The answer is not to use this, but
a reference to the current RPCClient object.
This is a pretty subtle point and it's easy to get forget. Of course, if you only use JavaScript, it probably makes perfect sense to you.
Time for the calling HTML page.
<!-- file: test.html -->
<html><head>
<script type="text/javascript" language="JavaScript1.2"
src="ajax.js"></script>
<style type="text/css">
.fakeURL { text-decoration: underline; cursor: pointer;}
</style>
<script type="text/javascript" language="JavaScript1.2">
var rpc = new RPCClient();
rpc.handler = function(x) { alert("From the caller: " + x)};
function dump (obj) {
document.write("<div><p><b>Object properties</b></p><dl>n");
for (var p in rpc) {
document.write("<dt>property: <code>" + p
+ "</code></dt>" + "<dd>type: <code>"
+ typeof(rpc[p]) + "</code></dd>n");
}
document.write("</dl></div>n");
}
</script>
</head>
<body>
<script type="text/javascript" language="JavaScript1.2">
//dump(rpc);
</script>
<span class="fakeURL"
onClick="rpc.send('ajax.php')">Make a request</span>
</body>
</html>
There's a bit of debugging code in there that dumps the properties of
the RPCClient object. I left that code here for future debugging needs.
A new RPCClient object is instantiated as you'd expect. The default
handler is overridden with a trival method.
Now the RPC mechanism looks a little cleaner! It's still asynchronous, but all the caller side code can be stored with the caller.
For the record, here's the PHP server, which is just echoes the values passed to it:
<?php
header("Content-Type: application/xml");
header("Cache-Control: no-cache");
print "<?xml version='1.0'?>";
?>
<response>
<? foreach($_GET as $k => $v) {
print "$k => $vn";
}
?>
</response>
This PHP is pretty trivial, but there are a few gotchas.
You have to set the MIME type correctly and disable browser caching.
You also have to emit XML, which is no big thing, except that the XML declaration looks like PHP code to PHP, so I had to be a little crafty in emitting it.
Finally, I just echo the key-value pairs I receive. Simple!
The next iteration of the code will pass the response back as a base64 encoded string. I don't need to pass complex data to the browser, but I may want to pass HTML. From my years of XML-RPC hacking, the best way to do this is by encoding the HTML. Otherwise, you'll go mad.
Have I mentioned that XML sucks too? It does.
Happy hackin'.
Mary, Joseph and the Spook!
If I never see another goddamn inconsistent filesystem on a linux box, it will be way too soon.
Jeez-suhs...
Here's a quick note to me on how to create a CGI script that streams a process to the browser in such a way as to prompt the user for a "save as" box. This is technique is older than dirt, but still useful.
#!/usr/bin/perl --
use strict;
use CGI;
use POSIX q[strftime];
my $q = CGI->new;
if (open my $in, "/usr/bin/zip -r - logs/ |") {
my $filename="lhp_logs-".strftime("%Y%m%d", localtime()) . ".zip";
print $q->header({ "-type"=>"application/x-unknown",
"-Content-Disposition" =>
"attachment; filename=$filename",
}
);
my $buf ="";
while (read($in, $buf, 4048)) {
print $buf;
}
} else {
die "Oops: $!";
}
The error checking isn't particularly robust here. Please gold-plate this code as needed for your next review.
A project for work required me to program in that most rigid and litigious
of languages, Java. I used Apache
Axis to make SOAP calls over SSL, thus making the project buzzword
compliant. This post is about how, with the help of John Cho at VMware, I
was able to coax Java in talking with SSL hosts whose X509 certificates have
no recognized identity by the client. Typically, this means the hosts are
self-signed and are not know to trusted root certificate servers. Without
this hack, Java will refuse to use the SSL connection until the user imported
the client certificate from the self-signed server, using the
keytool utility from the JRE.
NOTE: Disabling authenticating of SSL certs is not compatible with security-sensative projects. Please understand that by using this hack, you are reducing the effectiveness of SSL security (although the actual network traffic through the SSL tunnel is still encrypted). The decision to use this hack is typical of the design choices made on the security-versus-convenience axis.
Although this code is specific to Axis, the careful reader may find this code applicable to their next Java project.
Apache Axis 1.4 (and earlier) uses the standard java.net.ssl package that is distributed with Sun's JDK 1.5 (it's in older JDKs too). The Axis class that creates the SSL socket factory that contacts the class that actually does the cert authenication is:
org.apache.axis.components.net.JSSESocketFactory.java
What we need to do create a special X509TrustManager that accepts all certs and install it as part of the SSL Factory. There's a lot of abstraction and action at a distance code here, so let's start with how to implement an all-trusting X509Certificate class.
class TrustyTrustManager implements javax.net.ssl.X509TrustManager {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] c,
String authType) throws CertificateException {
// do nothing, accept by default
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] c,
String authType) throws CertificateException {
// do nothing, accept by default
}
}
Recall that X509TrustManager is just an abstract class. It requires three
methods to be implemented. Fortunately, these methods are easy to deal with,
if you don't care about authenication. The first,
getAcceptedIssuers(), is supposed to return an array of certs
for trusted authority servers. Since we aren't going to consult any, we
can return null there. The last two methods throw an exception if the
described certificate cannot be authenticated by building a "trust path" from
the root authorities to the given one. Since not throwing an exception means
that the trust path was succesfully constructed, we do absolutely nothing in
these methods. If only all code were so easy to write!
The only catch here is that we do need to import
java.security.cert.CertificateException to handle the exceptions
that we'll never throw, otherwise we get a compiler error. Thanks for the
extra hoop, Java.
I added this class as an inner, private class to the JSSESocketFactory.java file, but you may want to isolate this code into its own file for use in other projects.
Now we need to install this X509Certificate class. This can be done by
change the code in initFactory() to something like the following:
protected void initFactory() throws IOException {
// Inspired by John Cho
try {
javax.net.ssl.TrustManager[] trusty =
new javax.net.ssl.TrustManager[] {
new TrustyTrustManager()
};
javax.net.ssl.SSLContext sc =
javax.net.ssl.SSLContext.getInstance("SSL");
sc.init(null, trusty, new java.security.SecureRandom());
sslFactory = (SSLSocketFactory) sc.getSocketFactory();
} catch (Exception e) {
throw(new IOException("SSLFactory: " + e.getMessage()));
}
}
Let me step through the code backwards, since it will help to know where
we're going with this method. The whole point of initFactory()
is to initialize the protected class member sslFactory with a
valid object. SSLSocketFactory objects are returned by
javax.net.ssl.SSLContext objects, which the documentation describes like this:
«Instances of this class represent a secure socket protocol implementation which acts as a factory for secure socket factories. This class is initialized with an optional set of key and trust managers and source of secure random bytes.»
I emphasized "trust manager" here, since that's what we implemented with the TrustyTrustManager class. Now, we just need to create an SSLContext object for the SSL protocol (not sure why there are other options here), and shove in our TrustManager.
Well, not quite. It turns out that SSLContext expects to get an array of TrustManagers.
init(KeyManager[] km, TrustManager[] tm, SecureRandom random)
We don't need KeyManagers at all and the random seed can be generated from
a method in java.security. What we do need is an array of
TrustManagers with one TrustyTrustManager object in it. Once this is
assembled, init() can be called correctly and the
SSLSocketFactory obtained.
There you are! It's as simple as rocket science, but not as hard as brain surgery.
Some advanced Java coders will probably decry this code as
cheating. It's not very OO to manhandle an existing class like this. Surely,
there must be a way to subclass JSSESocketFactory and override the behavior of
initFactory() more cleanly. The answer is "yes," but that means finding all
those places that instantiate JSSESocketFactory objects and subclassing
those classes. And, of course, you can see how this subclassing will
quickly ripple through the entire object heirarchy of Axis to destroy all
my free time. In Perl, I might have simply poked through the package namespace
at some point to overwrite initFactory(). I'm not aware how to
do that in Java effectively, but I welcome your suggestions.
UPDATE: For additional information on doing this with Apache's XML-RPC lib for Java, check this out. I think this technique may work with axis too, since it is built of the standard Sun SSL stuff. But, I haven't tried.
SOAP::Lite is a suite of Perl modules for doing SOAP RPC. SOAP::Lite was originally written by mad Russians. Its incredible flexibility is also it's main drawback. Debugging isn't as obvious as it could be with this library.
Most of the time you the would-be SOAP scripter want to know what the request and response XML messages look like. The SOAP::Trace doc doesn't make this clear (since the sample code is filled with mistakes). Here is one way to get SOAP to spew the XML messages to STDOUT they way you'd expect from a more humble module like Frontier::Client.
use SOAP::Lite "trace" => ["transport" => \&log_it];
sub log_it {
my ($in) = @_;
if (ref $in && $in->can("content")) {
printf "**GOT: %sn", (ref $in);
print "-"x60, "n";
print $in->content, "n";
print "-"x60, "n";
}
}
There's a lot of fun things going on in the log_it() subroutine. Notice the use of the oft-forgotten can() method, which all Perl objects have. Yes, this is a very special code review.
Should SOAP::Lite have a simple flag like "trace_xml" which does all this for you? Sure it should. But until then, you've got this humble blog and your old Uncle jjohn to help you.
Now get the hell off my lawn, you kids!
I acquired an old DL360 (PIII 1.4/1G RAM/36GB SCSI (2)) from eBay for relatively short money (under $650). I needed a machine to develop Leostream apps for ESX. I've installed 2.5.3 on it now. I'll probably put ESX 3 on it when the final version is in my hot little hands.
When it's on, it sounds like a small vaccuum cleaner. It's my first dual-proc machine in chez Johnston. This machine replaces the ESX 2.1 "cluster" I had cobbled together with PIII 500 machines.
Since ESX 3 is supposed to support NASes, I hope that I can use the open source FreeNAS project for my common VMFS storage. Of course, having common storage suggest buying another DL360. That may or may not be needed. We'll see.
In the meantime, new toys!
Today's installment of Ask an Internet Foggy concerns the standard Windows background color. For the modern versions of Windows, the RGB values are, in uncool decimal:
red: 58 green: 110 blue: 165
It used to be dark cyan but then it switched to a more professional shade of blue.
Yes, I'm blogging this because I expect to look this up later in my life.
I'm so jazzed about my new monitor!
It's been a long time since I've raved about any computer equipment, mainly because I've been buying cheap crap from newegg.com. While that stuff is functional, it's not going to challenge Apple for any design awards.
To deal with some increasingly painful back aches (that I believe are all posture related), I decided to invested in a quality LCD monitor for my bedroom/office. For those keeping track, this is my second LCD monitor for my bedroom. The original 14" model is still in use for my music workstation.
My first choice for a monitor was the unbelievably awesome Apple Cinema display. Starting at $800, these devices are a luxury item that will have to wait until I make my tech start-up millions. Instead, I spent less than half of that (thanks to the aforementioned newegg) on an HP 1955 19" LCD monitor. At 24 pounds, it's a solid piece of equipment that reminds me of HP's glory days of manufacturing industrial-strength PC equipment.
Although I could have spent around $250 for a similar monitor, I wanted a display that would very likely work with a Mac Mini, which I think I will buy sometime this year. I can then get a KVM switch to go back and forth between the laptop and the Mac. I reasoned that Apple was more likely to support HP and some nutty Pacific Rim manufacturer. I'll let you know if this gambit pays off.
In any case, spending dough on a monitor makes more sense to me that spending cash on a computer. My monitors typically outlast the hosts. Since monitors are a huge part of the computer experience, it seems reasonable to put some loot out for a good one.
As far as I can tell, there are no dead pixels on this screen and the picture is very sharp and bright. Because the screen sits at eye level, I won't be hunched over looking, scanning for items on the desktop -- like a God-damned monkey! I've got opposable thumbs and the will to use them!
I'll run through the short list of problems. None of them are show-stoppers and a few of them actually help my back.
- Built-in USB hub in the monitor isn't recognized by XP pro on my laptop
- The monitor runs at 1280x1024
- I have to used an external keyboard and mouse
- My desk has a lot more wires on it
- I had to remove my iMac from the desk
I think #1 could be solved if I worked the problem long enough. But I just plug my USB keyboard (that I retrieved from the trash!) and my optical mouse (which I paid for [bah!]) directly into the laptop. #2 and #3 actually forced me to create an ergonomically healthier workplace and that was the point of the excerise anyway. As for removing the iMac, that's not going to affect my day-to-day productivity at all, since I rarely turn it on.
Really, my only complaint is about the wires on my desk. Which I admit is a pretty weak lament. So, it's all very, very good for jjohn.
Capitalism! Whodda thunk it would work?
After a good deal of hardware problems, I believe I have stabilize my digital infrastructure at home. Using the very excellent True Image from Acronis, I imaged the new and failing Samsung hard drive in my XP box. I then restored that image to a new and not failing Western Digital disk. The machine booted without incident and so I could put by living room back into order, instead of having look like a computer workbench.
I also restructured the "computer lab" in my bedroom so that it is neater, cleaner and properly wired. This includes some clever reworking of two machines running VMWare ESX 2.X and sharing two small SCSI disks across the same SCSI ribbon. This simulates a SAN very nicely for me. The disks run off a hacked power supply unit. It's all very sketchy, but it works.
Also, on permanent loan from Leostream is an old IBM Netfinity server. After the current project is done, I may attempt to install ESX 3.0 onto it.
To recap: my XP box in the living is now stable. It's my digital media convergence box that is my TV, DVD player and stereo. My bedroom lab hosts both my digital audio workstation (Sonar 4) and my machines for Leostream development.
Of course, my Linux soft-RAID1 box is in the hall closet.
Recently, I had to wade deeper into the murky, fetid waters of JavaScript. Like an encounter with a cheap hooker, a session coding with JavaScript makes me want to shower and weep. What I had to struggle with on this occasion was JavaScript's notion of associative arrays, which are implemented as Objects. All you Perler, Pythons and Rubyists out there, hold on! It's not that associative arrays are a type of object in JavaScript, it's that Objects are associative arrays (have I BLOWN your MIND yet, Java?).
Normally languages which provide associative arrays also build in
routines to list the keys, the values or both from an instance of these
variables. Not JavaScript. It's true that you can iterate through an Object's
attributes with a for/in loop, but that's a bit like using a
hammer to repair a watch, which is an analogy that also holds for parsing
query strings in URLs with JS. What a nightmare. It's always 1996 in
JavaScript's world.
Weirded out yet? Wait! I've held the most offensive bit of JavaScript classes for last. In nearly every other God-fearing lanaguage I've used, Objects are declared. That is, the object's attributes and methods are defined (usually) before an instance of that class can be used. So JavaScript, whose name clings to the fame of that paragon of OO-design Java, should have some kind of class declaration, right? I mean, declaring a class would make inheritence and object inspection easier and thus leverage the the major benefit of Object Oriented (OO) programming, right? Therefore, JS must have class declarations. But sadly, this isn't the case. JavaScript sports Classless objects. Objects in JS are defined at run time procedurally, in what seems to be an attempt to make the brains of OO zealots melt.
Classes without an inheritence mechanism are like pants without bottoms and only David Lee Roth could get away with wearing assless pants.
How does JavaScript allow users to define their own classes? You make a generic Object and start assigning methods and attributes to it, as if it were a dictionary, which it really is! How does object inheritence work? Not very well, but you can try assigning to the pseudo-attribute .prototype. Or not. Apparently, prototype is sort of busted.
Another consequence of not having classes is that you can never find out
what kind of Object you're dealing. That is, you can't ask an object,
"what class do you belong to, little fella?" The confused bastard will
answer "I'm an Object Object," which sounds a little desperate -- like the
JS object really wants you to believe it's a first class object, which
it isn't. This reminds me of a common folklore tenet in which all things
have a Truename that, if uttered, will give the speaker power over that thing.
Are JS objects superstitious? It's true you can "override" (or overwrite)
the .toString method with something about the class name, but
that's hackiferic.
Anyway, why bother with class heirarchies? Most JS scripts last only a short while and have so limited a scope. Then again, why bother pretending to have objects at all? Let's call a hash "a hash" and be done with it.
Classless classes are the assless pants of the Internet.
Ego surfing, in which you try to find other online references to yourself, is a time-honored tradition of the net that became ubiquitous with web search engines. I suspect that ancient proto-netizens used to look through archives of newsgroups to find themselves too, but the fossil record is unclear on this point. In any case, I'd like to push the start of the art in ego-surfing to the new frontier of social bookmarking.
Some of you will already know about del.icio.us, the clever site that lets you share your bookmarks with other people. One of the interesting features of the site is that it tells you how many other people have bookmarked the same URL. In this way, you can get a sense for the general popularity of a site. For instance, 3708 people have bookmarked google.com (for reasons that are unclear to me). Slashdot.org has been bookmarked 13,473 times. The Funeral Quest product page has been noted by only three people, including myself.
In my copious spare time, I began to wonder if anything I've written has been noted by anyone (outside of family). Surprisingly, the answer is yes. Here's how del.icio.us users remember me:
- why PHP 0wnz mod_perl: 15 other people
- Journal of jjohn: Snow Fudge, a recipe: 1 person
- Intelligent Design critique: 1 person
- Markov Blogger code: 1person
According to del.icio.us, I'm best remembered for slagging Perl, ID and blogging. Super! I'm pretty happy that someone found my fudge recipe interesting enough to remember.
There's a lot of money to be made feeding people's egos. Think of social bookmarking sites like del.icio.us and social networking sites like orkut and friendster and more generic hangouts like myspace. Clearly, there's some kind of business opportunity in ego massaging.
Must remember that next time I talk to VCs.
I've been working on building a new game since before last Christmas. I'm not ready to announce it yet, but I think it will be simpler to learn (and build) than State Secrets proved to be. As a warm-up to the real thing, I wanted to acquire the Flash skills that I think I'll need for game, which I think will have a flash interface that connects to a PHP script (a la, Funeral Quest.
Surprisingly, I think that in just a few days of watching online tutorials about Flash and some reading of the docs, that I've got the core skills that I need. Flash is a weird environment to work in, but I appreciate that Macromedia has hidden away threads and forking from me. In any case, I have created this very easy-to-defeat Tic Tac Toe game in flash. The client doesn't store any game state information, but uses the super-weird LoadVars class to make RPC calls to a PHP script which plays like a drunk co-ed. This is a technology exhibition rather than a very enjoyable toy. You may download the amateur-ish flash source code here. There are problems on the server-side that I don't care to fix. Sessions aren't properly implemented either. Good thing the price is right!
Those suffering from low self-esteem will appreciate the near-impossibility of losing this game.
I've begun again to learn how to use Macromedia's Flash. It's hard for me to pick up because the tool that authors flash documents uses an animator's paradigm, which is very unfamiliar to me. However, I did manage to put this crappy demo together of a bouncing ball married to a crappy MIDI cover of scarface's "no tears."
By God, if that's not entertaining, I don't know what is!
Perl offers many ways to commafy a number. That is, insert commas every three number for integers larger than 999. Here is my commafy routine:
sub commafy {
my $num = shift;
my $new = "";
while ($num =~ s/(?<=d)(d{3})$//g) {
$new = ",$1$new";
}
$new = "$num$new";
return $new;
}
It works backwards from right to the left. It uses the "new" look behind assertion in the regex. Works fine in Perl 5.8, and I think Perl 5.6.
I assembled my new P4 2.4Ghz machine yesterday only to find that the machine crashes. It starts to boot XP, but before going into graphical mode, it reboots. Started the machine in safe mode, and it just shuts off. Tried booting the machine with the XP install disk; crash. With W2K3EE; crash. With linux; boots into X, then crashes. With Win98; crash.
OK. I get the hint.
I call Intel tech support, who are both very foreign, very nice and very competent (a refreshing change). They explain that intel does not test chips reported as DOA, but that it's up to the customer to prove the chip is defective. Oddly enough, their biggest competitor does test chips reported as defective and replaces them as needed. I know this to be the case because my buddy Zorknapp had just such a defective AMD chip. His experience with ADM's support system was much better than mine with intel.
ADM, knowing how a CPU should behave and what the expected voltage outputs should be for given inputs, can very quickly and efficiently determine if a CPU is faulty. Intel expects me to drop this P4 into another P4 box (which I don't have) and see if the problem persists. This is a good test for a field engineer, but a bad test for a large chip manufacturer. Intel surely has the equipment that AMD has. They simply refuse to use it.
Intel chips are much more expensive than one's from AMD. Intel is a lot wealthier than AMD. Intel should test their defective chips. How gallingly aggrogant of them not to. How unpardonably parsimonious of them.
Intel, you have not only lost this customer, but all those I advise on computer purchases. Hold on to that cash, tightwads. You should be ashamed of yourselves.
Well, the Dell Dimension 4400 is dead and I've ordered a replacement from newegg. I'll reuse the storage systems from the Dell and put them into a P4 2.4Gz prescott PC Chips mobo (boo) with 1GB of RAM. Amazing, the CPU, mobo, RAM and case with shipping is less than $300. Wow.
I'll stuff the Hauppauge TV tuner card in the new box. It will be a general terminal and will have more than enough horsepower for the games I play.
Thanks to TorgoX, I've been listening to the Kleptones' Night at the Hip Hopera. Oh my!
Let's do this New Year right!
My Dell XP box is not booting, even after rebuilding the box. It barely gets past the POST and then hangs. A coworker cleverly suggested that the power supply may be hosed. I was thinking the problem might be in the DIMMs. I frequently forget that power supplies die. I have an old 240V Dell power supply that I'll use for testing. Dell apparently uses custom PSUs with non-standard wiring for vendor lock-in. Boo.
They probably got a deal on them from the supplier.
I'm nearly convinced to turn all of my computers into fish tanks, door stoppers and mod furniture.
I've got a Redhat 9.0 box running a software RAID1 array with two 160GB IDE. It works well and I can't complain about the performance. Like the idiot that I am, I failed to set up notification to tell me when a disk in the array fails (UPDATE: use smartd to check the health of hard drives and even mdmonitor to watch for a failed disk in array sets).
This happened recently (within the month), and the array degraded gracefully to using the remaining disk. But I still had to replace the "broken" one (which I don't believe to be broken at all).
To do this, install the new disk as planned. When you boot, you'll be shoved into a root shell.
- fdisk the new disk.
- create a new partition with partition type 0xFD.
- write the new partition table out.
- edit the /etc/raidtab file. Promote the remaining disk to being the first in the array.
- start raid with mdadm:
mdadm --assemble --run /dev/md0 /dev/hd - add the new disk into the existing array:
mdadm --add /dev/md0 /dev/hd - The new disk will sync up to the old.
Verify with
cat /proc/mdstat. This process took about five hours on my system. - Reboot your new, happy system.
Just to make this more concrete, my /etc/raidtab looked like this:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
chunk-size 4
persistent-superblock 1
nr-spare-disks 0
device /dev/hdb1
raid-disk 0
device /dev/hdc1
raid-disk 1
Recently, /dev/hdb1 failed (although it seems ok. fsck revealed no problems, but it was out of sync with /dev/hdc1). I then replaced the drive (Slave HD on the primary IDE channel). I booted the box and changed /etc/raidtools:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
chunk-size 4
persistent-superblock 1
nr-spare-disks 0
device /dev/hdc1
raid-disk 0
device /dev/hdb1
raid-disk 1
I then started the RAID: mdadm --assemble --run /dev/md0 /dev/hdc1. Then, I added the new disk: mdadm --add /dev/md0 /dev/hdb1.
Hope this limited and cursory treatment of a complex topic helps.
UPDATE: Also see this primer on LVM and RAID tools in modern RedHat. This system was running RedHat 9 and so used the older raidtools stuff. You really only need mdadm, which can create, restore and repair RAID disk sets. Configure /etc/mdadm accordingly.
Good times: I'm dealing with another hard drive crash. This one is less traumatic than previous ones, but it still sucks. It's a 160GB drive formated for NTFS that was a sort of extra data drive for my Windows box. I have managed to CHKDSK.EXE the drive several times, but it's not happy. However, something of the filesystem lives and I think the leostream converter floppy disk appears to be able to read the raw disk. So, I'll buy some new drives tomorrow and image the old disk across.
This is much less fun than writing software. Happy friggin' birthday.
(No, I'm not tired of that stupid camel picture yet.)
UPDATE: Most of my digital infrastructure is lying in pieces on the floor of my apartment. Here's the cascading failures:
- Second HD on my WinXP box has some kind of HW failure -- locks up machine
- HD removed
- WinXP reconfigure for one drive
- After few hours, it too locks up
- Couldn't rip CDs on my linux box, cdrom apparently not available
- More digging on the linux box, one of the disks in my two disk SW RAID1 has gone tits up
- Bought two new 160GB Western Digital drives (with $80 rebates for each)
- Replaced failed RAID1 disk, rebuilt raid, am syncing the disks (slowly)
- Have installed other 160GB drive on WinXP box, partitioned into 40GB/120GB, reinstalled XP, slaved the old C: drive for file access
Need a real backup plan. I didn't lose any important data this time. Probably will get a USB drive to hang off linux box for "offline" storage.
I'd love to know what Bill Gates does for this crap. He must have a private IT staff.
Also, there's a blog post sitting on my old C drive waiting to be posted.
UPDATE 2: Please see this post for the latest working version of this module.
UPDATE 1: This code works with XML::RSS version 1.05 or so. The newest versions of this library removed the encode() method for reasons beyond my reckoning. You can either use this code as a starting point for porting to the new XML::RSS module (and tell me how you did it!) or simply use the older version, which is still available on CPAN.
Like searching for Bigfoot, creating a podcast feed that's recognized by Apple can be an elusive, furtive and lonely process. Most know that podcasts are really just RSS 2.0 feeds with some extra tags. This seems like something perl should handle. The perl module XML::RSS nearly has everything necessary, but there's always a catch: the itunes namespace.
All is not lost, because XML::RSS is a class that can be inherited from. With a little overriding goodness, you too get make valid feeds that even Apple's iTunes music store will accept. Here's my module that inherits from XML::RSS. While it's not a complete solution, it works well enough for me.
package XML::RSS::Podcast;
use XML::RSS;
@XML::RSS::Podcast::ISA = qw[XML::RSS];
sub as_string {
my $self = shift;
return $self->as_podcast_rss;
}
sub as_podcast_rss {
my $self = shift;
my $enc = $self->{encoding};
my $output = <<EOT;
<?xml version="1.0" encoding="$enc"?>
<rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
version="2.0">
EOT
$output .= $self->podcast_start_channel;
for my $i (@{$self->{items}}) {
$output .= $self->podcast_item($i);
}
$output .= $self->podcast_end_channel;
return $output .= "n</rss>n";
}
sub podcast_start_channel {
my $self = shift;
my @fields = qw[ttl title description link language
pubDate lastBuildDate creator
webMaster copyright
];
my @image_fields = qw[title url description link
width height];
my @itunes_fields = qw[subtitle author summary
image];
my $output = "<channel>n";
for my $f (@fields) {
if (length($self->{channel}->{$f})) {
my $s = $self->encode($self->{channel}->{$f});
$output .= "t<$f>$s</$f>n";
}
}
my $seen_image = 0;
for my $f (@image_fields) {
if (length($self->{image}->{$f})) {
unless ($seen_image) {
$output .= "t<image>n";
$seen_image = 1;
}
my $s = $self->encode($self->{image}->{$f});
$output .= "tt<$f>$s</$f>n";
}
}
if ($seen_image) {
$output .= "t</image>n";
}
# Owner name/email not handled
for my $f (@itunes_fields) {
if (length($self->{channel}->{itunes}->{$f})) {
my $s=$self->encode($self->{channel}->{itunes}->{$f});
$output .= "t<itunes:$f>$s</itunes:$f>n";
}
}
# FIXME: Doesn't handle sub cats.
if (ref $self->{channel}->{itunes}->{category}) {
for my $c (@{$self->{channel}->{itunes}->{category}}) {
my $s = $self->encode($c);
$output .= qq[t<itunes:category text="$s" />n];
}
}
return $output . "n";
}
sub podcast_end_channel {
return "</channel>n";
}
sub podcast_item {
my $self = shift;
my $item = shift;
my @fields = qw[title guid pubDate description];
my @itunes_fields = qw[author subtitle summary
duration keywords explicit];
my $output = "t<item>n";
for my $f (@fields) {
if (defined $item->{$f}) {
$s = $self->encode($item->{$f});
$output .= "tt<$f>$s</$f>n";
}
}
if (ref $item->{enclosure}) {
$output .= "<enclosure";
for my $f (qw[url length type]) {
if (defined $item->{enclosure}->{$f}) {
$output .= qq[ $f="$item->{enclosure}->{$f}"];
}
}
$output .= "/>";
}
for my $f (@itunes_fields) {
if (defined $item->{itunes}->{$f}) {
$s = $self->encode($item->{itunes}->{$f});
$output .= "tt<itunes:$f>$s</itunes:$f>n";
}
}
return $output .= "t</item>n";
}
A word about the RFC822 pubDate. This seemingly arbitrary date format
can be easily generated with a call to strftime(). The
format string is "%a, %e %b %Y %H:%M:%S %z". You might think
that you can use mysql's DATE_FORMAT() to replicate this, but you'd be wrong.
Instead, generate mysql queries with UNIX_TIMESTAMP(), feed the result of
that to localtime() and feed that to strftime(). Simple, no? No, but such
are the challenges in programming.
Here's a sample of how I use this for pseudocertainty.com.
use strict;
use DBI;
use POSIX qw[strftime];
use MP3::Info;
my $rssfile = shift || "./ps-pod.rss";
my $dbh=DBI->connect("dbi:mysql:pseudo", "pwrUser", "s3cr3t")
or die "connect: $DBI::errstr";
my $shows = get_shows($dbh);
$dbh->disconnect;
my $rss = XML::RSS::Podcast->new(version => "2.0");
my $rfc822_fmt = '%a, %e %b %Y %H:%M:%S %z';
my $iMeta = { "author" => "Joe Johnston and Mike Lord",
"summary" => 'UFOlogy, Cryptozology and
the people who love them are
discussed on this
internet-only radio show',
"subtitle" => "Don't be Certain.
Be PseudoCertain.",
"category" => ["Talk Radio"],
};
$rss->channel(title => 'PseudoCertainty',
"ttl" => 60, # time to live
link => 'http://www.pseudocertainty.com/',
language => 'en-us',
description => 'UFOlogy, Cryptozology and the
people who love them are discussed
on this internet-only radio show',
copyright => "Copyright Joe Johnston and Mike Lord",
webMaster => "jjohn@pseudocertainty.com",
pubDate => strftime($rfc822_fmt,localtime()),
"itunes" => $iMeta,
);
for my $r (@$shows) {
# no more than 30 words
my @words = map { s/(<[^>]+>)//g; $_; }
split /s+/, $r->{about};
my $desc = "not set";
if (@words > 30) {
$desc = join " ", @words[0..29], "...";
} else {
$desc = join " ", @words;
}
my $finfo = get_mp3info("/path/to/shows/$r->{mp3_filename}");
my $pl = qq[http://pseudocertainty.com/$r->{mp3_filename}];
my $enc = { url => $pl,
length => -s "/path/to/shows/$r->{mp3_filename}",
type => "audio/mpeg",
};
my $itunes = {
explicit => "N",
keywords => "UFO aliens zorknapp",
summary => $desc,
duration => $finfo->{TIME},
}
$rss->add_item( title => $r->{title},
link => $pl,
pubDate => strftime($rfc822_fmt,
localtime($r->{pretty_created})),
enclosure => $enc,
permaLink => $pl,
description => $desc,
"itunes" => $itunes,
);
}
$rss->save($rssfile);
#------------------------
# subs
#-------------------------
sub get_shows {
my ($dbh) = shift;
my $sql = qq[
SELECT *,UNIX_TIMESTAMP(created) as pretty_created
FROM shows WHERE publish = 1 ORDER BY created DESC;
];
my $sth = $dbh->prepare($sql);
die "get_shows: '$sql': " . $sth->errstr unless $sth->execute;
return $sth->fetchall_arrayref({});
}
And they called me a fool for wanting to make XML::RSS spew podcast RSS. But I showed them! I showed them all! Bwahahhaha!
(Note 1: This is the third and last part of a series on the taskboy CMS. It's the least informative of the three, substituting opinion for fact in a thuggish way.)
(Note 2: I stole this picture of TV cook and personal hero Alton Brown from the Internet Wayback Machine, since he no longer blogs. He had an ugly incident with the public, it seems. Personally, I think it became too much of a time sink for him. I hardly have a life at all and I can't add content here daily.)
XML, SOAP and other windbags
This is a very long post already, but I'll chime in some recurring trash talk about XML-RPC that irks me. Before I start, I want to clarify one thing: this screed isn't directed against Mr. Harold, who is to be praised for making his manuscript available for free. That's something I'd like to see more of from all quarters. Mr. Harold has done his part to make the world a little brighter by providing a free and quality resource for java programers.
OK. Time for the shit storm.
In Processing XML with Java, Elliotte Rusty Harold asserts the following while introducing the section on SOAP:
«XML-RPC was in large part invented by a single person who really didn't know a lot about XML. Consequently he made many very questionable choices...
Whereas XML-RPC was a quick hack by one developer, SOAP has been developed by a committee of XML experts from various companies including IBM and Microsoft.
You've undoubtedly heard the old saw about a camel being a horse designed by committee... SOAP is a much more robust protocol than XML-RPC. It is much better designed from an XML standpoint as well. It takes advantage of numerous features of XML such as attributes, Unicode, and namespaces that XML-RPC either ignores or actively opposes. (my emphasis) XML-RPC is adequate for simple tasks. However, if you get serious with it you rapidly hit a wall...
The biggest conceptual difference between SOAP and XML-RPC is that XML-RPC exchanges a limited number of parameters of six fixed types, plus structs and arrays. However, SOAP allows you to send the server arbitrary XML elements. (my emphasis) This is a much more flexible approach. »
This is a very common argument made against XML-RPC and here's why it's dumb.
The beauty of web services doesn't lie in XML; it lies in the simplicity of the process. Yes, XML (sort of) helps this simplicity, but the real value of a web service in the work it faciliates. It's the platform neutral procedure calls that make web services interesting. Too many SOAP advocates are thinking in terms of only one language (*cough*java*cough*) and so expect a very close mesh between their language and an RPC mechanism. News flash: XML-RPC ain't RMI. I've used XML-RPC in at least a have dozen languages. SOAP is just painful, even in java (which I used for a cell phone application). Sometimes I expect an array of strings just be an array of strings.
XML-RPC is caveman simple and that is a colossal advantage. It's restriction of datatypes is an advantage that many OO-heads should wake up to. Not only are most things in life not Objects or Classes, many things for which there are Objects and Classes shouldn't be. Sometimes a plain old ASCII string is Good Enough, despite political posturing.
I dislike XML and barely tolerate it in XML-RPC. All of the XML features mentioned in the quote, particularly namespaces and user-defined datatypes make SOAP a digital Babel. I hardly like attributes in XML. SOAP's flexibility is a liability. How many times of Perl hackers been taken to task for their language's robust syntax? WSDL is little help to taming the complexity of web services here. It contributes to the traffic jam. I LIKE that there's no offical DTD for XML-RPC. DTD's are sooo SGML. Well formedness is all you need. Everything else is a data validation problem for the application.
That's right, I said: the app needs to validate data. Does your language make this easy? If not, get one that does.
The KISS principal of XML-RPC goes very, very far here. The features that SOAP has are for a future web of web services that may never come. XML-RPC is here today; stable with a boatload of implementations in many languages. SOAP is one document format too far and too far out.
The End: "Well, I'm back."
(This article continues my thoughts on the taskboy CMS.)
How the taskboy CMS works
Once I decided that content would be managed through emacs (to as large a degree as possible), the rest fell mostly into place. The blog, the music section, the polls and the ratings would all be stored in mysql and accessed through a XML-RPC API. I would use PHP to define the layout, pulling the content from the database where needed. Templates as such are not used. To my thinking, a PHP page is the template. I also decided against database abstraction classes, since I'm unlikely to move from mysql any time soon. I do have a collection of PHP utility functions (like, sql_insert, sql_delete, sql_select) to make database access less painful. Each PHP page calls the same header and footer pages. Much of this code was developed along side State Secrets. Together, this makes the PHP stuff pretty easy to modify.
Getting from emacs to PHP is a little circuitous, so please bear with me. It is straight forward to write a perl script that's an XML-RPC client using the Frontier::RPC2 library. So that's what I did first. I verified that I could talk to the PHP page that processes XML-RPC requests. Emacs is an extentable editor using the macro language lisp. The creator of Perl, Larry Wall, said of lisp that it had all the visual appeal of "porridge with toenail clippings" and I agree. However, I did learn just enough lisp to write the current emacs buffer to standard out to be read by a perl script which could then make the appropriate XML-RPC request and make some snappy response that emacs could deal with. This solution is what I wrote about on use.perl.org.
Gnu Privacy Guard
The new wrinkle for taskboy is security. The XML-RPC messages go across the network in clear text. The primary risk I wanted to address is not that someone will see my blog before it's posted, but that an unauthorized fool would mess with my XML-RPC service. Whatever authorization mechanism I choose would have to work over clear text. It's true I could have used SSL with HTTP Authentication for the web services PHP page, but I didn't want to. Fortunately there is already a solution for this kind of problem, but for a different form of internet messaging.
Back in the mid-80's, Phil Zimmerman had a problem: he couldn't prove he was him. That is, email that claimed to be from him could have been forged by some joker only claiming to be him. How could those receiving email from he be assured that the sender Phil Zimmerman was the Phil Zimmerman? The answer became known as Pretty Good Privacy and it involved some very scary math. But you can think of it as something like a lock and key mechanism. When an email is sent out, a Very Big Number is computed with the content of the message and your private key. Your private key has a sibling called a public key that the recipient of mail will already have (and verified). When the recipient gets this message, pgp uses the public key on file to decode the message (or signature). If nothing has been changed in the message, the math will work out (via magic) and you can be pretty sure that Phil indeed has told you to "go pound sand."
The important concept here is that PGP was meant to guarentee the identity of a sender using a message that anyone could read, but not change. Now in web services, I also have messages that anyone could read, but I want the server to accept only requests from me. Although it's not a seemless fit, PGP turns out to be a good authenication method for private web services. Here's how I modified Edd Dumbill's XML-RPC PHP library and Ken MacLeod's Frontier::RPC to use Gnu Privacy Guard (any open source version of PGP) to look down my web service. The strategy in both cases is that requests should be signed, not responses. It would be staight-forward to implement response signing too, but I don't deem it necessary for my application.
Tweaking the PHP server
This class merely extends the xmlrpc_server class found in xmlrpcs.inc. I need to intercept the content, verify the signature, remove it if the message checks out and pass the rest of the XML doc to the parent class for handling. Hats off to Edd and the boys for getting the class partitioned so that I needed to override only one method.
One PHP tip: name your class files with .php. That way, you can point a browser to them and check the syntax. After all the syntax typos are gone, the page will appear blank. The the contents of files with .inc extensions are typical just displayed by the web server without parsing.
VerifyRequest($data)) {
return $this->RPCError("Couldn't verify request");
}
$data = $this->RemoveSignature($data);
}
# pass off to parent
return parent::parseRequest($data);
}
#-----------------------------------------
# Look at the body of the request. Does it have
# a signature to verify?
function VerifyRequest ($data="") {
# BTW: I hate this solution
# write out to a tmpfile
$infile = "/tmp/" . posix_getpid() . ".vrf";
if ($fh = fopen($infile, "w")) {
fwrite($fh, $data);
fclose($fh);
} else {
return 0;
}
# is this signed by someone I trust?
$cmd = "/usr/bin/gpg --homedir=/path/to/gpg "
. "--verify <$infile 2> /dev/null";
$retval = 1; # default to failure
if (file_exists($infile)) {
system($cmd, $retval);
} else {
return 0;
}
unlink($infile);
return $retval ? 0 : 1;
}
#-------------------------------------------
# remove signature header/footer
function RemoveSignature ($data="") {
# for GPG
# strip of the GPG stuff to get the basic XML back
$preamble = "/-----BEGIN PGP SIGNED MESSAGE-----r?n"
. "Hash: SHA1r?nr?n/";
$footer = "/-----BEGIN PGP SIGNATURE-----r?n"
. "Version: .+r?nr?n(S+r?n)+"
. "-----END PGP SIGNATURE-----/";
$data = preg_replace($preamble,"", $data);
$data = preg_replace($footer,"",$data);
return $data;
}
#--------------------------------------------
# wrapper for easier (and non-granular) error reporting
function RPCError ($msg=0) {
return new xmlrpcresp(0,500,"Bad request: $msg");
}
}
?>
A few notes on this amateurish PHP code. First, any security wonk will tell you not to create temp files with PID names. In my case, I trust the other users on my server and don't feel compelled to improve the security here. You may want to. I'm using the fact that gpg process has an exit value of 0 if the verify succeeds. The only way I saw of getting the exit value of a process in PHP is by using system(). There are a couple of other process handling functions, but those didn't seem to give me this simple result to check (I could have used popen() and grepped through the output, but that seemed painful [although I might have done that if this were a perl module]).
parseRequest() is called by the parent class to unpack the XML request. Here, I look for the GPG signature and if all goes well, I pass just the XML string to the parent parseRequest() for processing.
Keep in mind that PHP runs as whichever user Apache runs as. This affects GPG. You have to set up the file ownership for the keys so that Apache can read and write to a directory. You should create keys specifically for this web service and not reuse your own GPG stuff. You were warned.
This class is used identically to the xmlrpc_service class defined in xmlrpcs.inc. No, I don't know what the "da_" stands for in the class name. I though I wrote "ds_", which would have stood for "digital signature."
Expanding the Frontier
For the perl client, I simply defined to classes at the start of the program. Keep in mind, this is a win32 perl program.
package RPCEncoder;
use Frontier::RPC2;
@RPCEncoder::ISA = qw[Frontier::RPC2];
sub encode_call {
my ($self) = shift;
my $request = $self->SUPER::encode_call(@_);
# sign it. 2-way opens hurt my brain
my $outfile = "C:/blog/tmp.txt";
unlink $outfile;
my $cmd = qq[|C:/blog/gnupg/gpg.exe --homedir=/blog/gnupg ]
. qq[--clearsign > $outfile];
open GPG, $cmd or die "Can't proc open: $!";
print GPG $request;
close GPG;
open IN, $outfile or die "Can't open signed $outfile: $!";
undef($request);
while () {
$request .= $_;
}
close IN;
unlink($outfile);
return $request;
}
sub decode {
my ($self) = shift;
my ($string) = shift;
my %args = ('Style' => 'Frontier::RPC2',
'use_objects' => $self->{'use_objects'},
);
$self->{'parser'} = XML::Parser->new(%args);
return $self->{'parser'}->parsestring($string);
}
#-----------------------------------------------------
package RPCClient;
use Frontier::Client;
@RPCClient::ISA = qw[Frontier::Client];
sub new {
my ($self) = shift->SUPER::new(@_);
my %args = ('encoding' => $self->{'encoding'},
'use_objects' => $self->{'use_objects'}
);
$self->{'enc'} = RPCEncoder->new(%args);
return $self;
}
The perl is a little weirder because of the way the Frontier Client works with XML::Parser, itself a horrible creation of Cthulhu. The Frontier::Client constructor needs to be overrided so that I can insert my custom RPCEncoder class, which is a thin coating over Frontier::RPC2. All the XML encoding and decoding happens in Frontier::RPC2 and that's what I need to intercept.
When making a request, I need to sign the XML string before it goes on the wire. All things being equal, I'll like to open the gpg process for reading (to feed it the string I've got in memory), but also read from it to get the output. This is a kind of double pipe, which is easy to do in shell, but weird to do with perl and especially so on Windows. Once again, I write a temp file and I don't even pretend to give security a mind. Windows boxes are typically single user machines and mine doubly so. Also note that I don't need to worry about running as a different user when I make the XML-RPC request. I'm in emacs (which runs as the current user); it spawns a shell to run perl; perl spawns a shell to run gpg.exe). All these processes run will run as me.
I had to also override decode(), because the parent uses ref($self) to determine the class name of the XML callbacks (n.b. BAD MONKEY!). This
really should have been hard coded to 'Frontier::RPC2' since the callbacks all
have hardcoded class names (see the code for the real scoop). I think this was
an attempt to make child classes easier to write, but this trick backfired.
A Quick Note on GPG setup
Getting up to speed on how GPG works took longer than integrating it into
the taskboy web service. I cannot go in to all the set up details here, but
if you are familiar with ssh key mananagement, you will be well ahead of the
game in GPG. If ssh keys make your brain hurt, GPG is a veritable migraine.
But it boils down to this: you must make a GPG key pair for the source machine
with the perl/emacs setup. You must copy the public key to the server. You
must import that key into GPG and verify it (with gpg --edit).
If you don't do all of these steps, this digital signature for XML-RPC hack
won't work and you'll be mystified at what went wrong.
Verify your GPG at all stages using test files, so that you can get the
GPG errors.
Note to jjohn: Move the *gpg files to wherever gpg want to find them. It will make things go easier on you.
The next three posts are about how XML-RPC makes taskboy go, why XML-RPC is better than SOAP and how PHP and perl excel in their own domains. I wrote these as one long article, but, taking pity on my readers, I broke the leviathan into three parts. I expect to be rewarded for my gesture of mercy with crazy-go-nuts xmas stocking stuffers this year.
Web Logs
Although web logs, more popularly called blogs, have recently come to most people's attention during the 2004 US presidential election, the idea of having an easy mechanism to post ideas large and small onto a web site is not new at all. After writing the first dozen or so HTML pages by hand, most of us starting thinking there must be a better way.
Enter the Content Management System
The content management system (or CMS) is a software package that sits on the web server (and perhaps has a special client for your local machine) which faciliates publishing web content and maintaining a consistent look and feel from page to page. The core technology for a consistent look and feel is the use of a templating system. That is, a pattern of code that content can be paired with to generate the final HTML page that visitors to a site view. A natural fit with the use of templates is an underlying database in which to keep the core, user-defined content (and possibly the templates too). So much for web design history up to 1998.
Early CMS tools were written in a variety of languages. Back in the day, Perl was the king of dynamic web applications and was used to build many CMS systems. The trouble with Perl is that it's fidgity in spots to learn and (for some people) it's complexity isn't welcomed as a templating language.
Enter PHP
PHP is a dynamic, perl-like language specifically designed for the web domain. It's both fast and feature-full (and not a little unwieldy at times from problems stemming from it's early lack of namespaces). Knocking out simple, but powerful dynamic web sites with PHP is criminally easy and fast. PHP even has a Windows help file, for those of us workin' on Uncle Billy's farm.
But not everything needed to make a web site go happens in CGI land. There are plenty of system-level things and non-http tasks for which PHP isn't suited (despite some heroic hacks from the PHP community). Wouldn't it be nice to have a way to get the best of both worlds: to use PHP for web front end stuff and perl for backend management?
Enter XML-RPC
XML-RPC is way for programs written in different languages to talk to each other. This concept has the buzzword-compliant name of "middleware" and it's a far from new idea. Programs are, almost by definitions, fiefdoms onto themselves in which data gets trapped. It's hard enough to make computers do what you want to today, let alone try to build software in such a way that it will be useful tomorrow. XML-RPC is the method by which one program's procedure is serialized as XML, passed over a network using HTTP, executed on the receiving machine who then returns the response as XML to be unserialized back into a format mostly easily consumed by the caller. It is a "web service wire protocol" and is the forerunner of the more feature-rich (but strangely less useful) SOAP. This complex dance allows Perl and PHP to use each in appropriate contexts. But what are those contexts? The answer lies with existing CMS.
Thin clients? Fat chance!
The web browser is arguably the most important application to be written in past ten years. In network architecture terms, browsers are thin clients that handle a variety of display issues and a small subset of user input tasks. This is the key to the success of the web. The browser, as envisioned by Tim Berns-Lee, was for reading and changing web pages. By latching onto the read-only aspect, browser makers forestalled what would have been a virtually endless debate about how to implement security for changing web pages. Web page security became the domain of web servers. Which is why CMS systems must be installed on web servers and can, for convenience or to implement specific features, have a client piece on a web author's desktop. Many CMS just used the browser for their client.
And this is exactly the thing that most blogging software does: use the browser to compose entries and manage the content. Blogs, often being simply one page, wouldn't have gotten so popular if each required some kind of special input client on each author's machine. Blogs then are a very specialized form of CMS.
The problem is: I hate composing prose in web browsers. Being a programmer, I've invested a lot of time learning to use a programmer's editor (emacs, in my case). What I want from a blog CMS is to compose, publish and manage my blog entries from emacs editor on my desktop. Now many a blogging CMS have some kind of web service APIs that allow you to hack up glue for your editor of choice to do the same thing that I've done with taskboy. Much of the taskboy system derives from what I did for my use.perl.org journal.
When I decided to move my first blog off use.perl.org, I knew that I wanted to continue using emacs to compose and edit my entries. However, I now had to deal with the front end display logic for the blog. Having written and modified several perl CGI/mod_perl HTML generating programs, I wanted to avoid that solution. Instead, I wanted to learn more about PHP and I'm glad I did.
Next: The dirty details
Here's a perl tip for those trying to report progress on external programs that don't report that kind of information. The case this hack was designed for was gzip, but you'll think of many other examples of this class of problem.
Perl is an incredible flexible language. Of all its wonderous features, the ability to get a file handle to a process is the most arcane and little appreciated. It is, however, the key to reporting how much input is consumed by a process.
Here's a concrete example of what I'm talking about. The compression utility gzip is a stream-oriented program that works on chunks of data that it receives typically from stardard input (STDIN). You can therefore feed gzip a file of any size and it should work, given enough disk space. The larger the file, the longer gzip takes to run (I suppose this makes the runtime a Big O of (n), linear time [so much for using my comp sci degree]).
Occassionally, you'd like to know how far along gzip is in compressing a large file. Gzip does not report this, but does give you the compression ratio at the end of the run, if you called it with the -v flag.
Without hacking gzip, you can create a perl wrapper around gzip in which you can report how many bytes gzip has consumed of the source file. The idea is that the source file is read by perl and feed to gzip. Keep tracking of how many bytes are read in the perl script is simple. Here's some code.
my $infile = shift @ARGV || die "$0n"; open GZIP, "|/bin/gzip -c > out.gz" or die "can't open process to gzip: $!"; # disable output buffering to see the progress report $|++; open IN, $infile or die "Can't open $infile: $!"; my $original_size = -s $infile; my ($buf, $sum); my $chunk = 200; while (read(IN, $buf, $chunk)) { $sum += $chunk; print GZIP $buf; printf "progress: %02.2fr", ($sum/$original_size)*100; } print "n"; close GZIP; close IN;
This short script expects to be called with the name of the file to compress. The output file name is hard coded to be "out.gz", but it's a simple matter of programming to make this more flexible. The magic begins when we open the process to gzip. Here, the GZIP file handle will be written to. The source file is then opened for reading. I choose to read the source file in very tiny chunks to clearly see the progress indicate work. Here, 200 bytes are read from the source file and then feed to gzip. The number of bytes read is tracked and reported in a straight forward way.
Two penetrating glimpses into the obvious. One: this script is built
for some flavor of UNIX. Some modifications would be needed for Windows,
including the use of binmode(IN), binmode(GZIP).
Two: this is really just a specialized echo loop. While I'm not one to
yammer on about coding patterns, I would say that nearly 90% of the code I
write is some kind of echo loop, when you take away the business logic, error
checking and other distractions.
If you only learn one thing for a programming class, it should be the humble echo loop.
In my professional life, I'm often tasked with making a help system of somekind to help users understand the application I'm building. This is notiously difficult to do. However, certain technologies have come along to make showing users what to do possible, even over dialup connections. One of those technologies is Flash, which combinds lightweight vector graphics with mp3 support.
Unfortunately, the Flash studio app costs mucho dinero and I wouldn't use the tool enough to get the full value of my investment back. Also, what I want to do is capture stuff I'm doing on my desktop along with a voiceover of what's going on. This, to my knowledge, would require an additional package.
Fortunately, there have been a few open source projects that can manipulate the Flash file format to some degree. But what I need is to capture a Windows desktop session.
The solution comes in the python version of vnc2swf. This appears to be a platform-neutral solution, which is remarkable.
Very. Good. Hack.
I wish perl had done this first, but hats off to the python hackers that made this happen. I applaud the use of a scripting language I can hack. I particularly like the edit.py feature, which allows me to add an mp3 soundtrack. I have a lot of equipment and software to manipulate sound, including Sonar for the base recording and WAV editting and RazorLame for the mp3 encoding (which I did at 32kb sample rate monophonically). I did have to time shift the original VO track to 85% of the original. That's why the demo sounds like I just snorted crystal meth. I'm not sure if there's a better trick for syncing the video and audio.
Click on the picture for the full demo of my wonderful site.
For my long-term gaming project, State Secrets, I had to write this gem of SQL code:
SELECT c.*,length(session_id) as online FROM characters as c LEFT JOIN users as u ON u.current_char=c.id
LEFT JOIN sessions as s USING (username) WHERE u.username != "admin" ORDER BY c.savvy DESC
Even now, that double join makes my eyes bleed.
Some would site the following device as an example of overengineering. It converts vinyl LP records into mp3s via an iPod.
However, those critics would be entirely wrong. Many of my generation only vaguely remember that electronic devices such as TVs and stereos used to be sold in very large and heavy wood furniture frames. They were meant to reflect affluence as much as fulfill their primary objective. I'm not one that pines for the old days of vinyl (I think digital music is just fine), but I'm tickled by the elabrate fussiness that transformed the relatively disposable iPod back into an old-school furniture-appliance.
Despite my glee at this hack, my aging back does approve of the comparitive weightlessness of today's devices. In the old days, you tended to plan your interior design around the TV and stereo because you sure as shootin' weren't moving them.
To the makers of this fanciful iPod mod: I salute you!
Update: For reasons unknown, my ruby iMac that I found in the trash is booting again! Happy days! Who said that ignoring a problem won't make it go away? He or she (or maybe even it) was wrong!
Are you jealous of all those cool ascii art titles at the beginning of game FAQs? Be envious no longer because this CGI app will make the banner for you, in a variety of "fonts."
Here are a few examples:
--.-- | |
| ,---.,---.|__/ |---.,---., .
| ,---|`---.| | || || |
` `---^`---'` ``---'`---'`---|
`---'
Sleek European design
:::==== :::==== :::=== ::: === :::==== :::==== ::: === :::==== ::: === ::: ::: === ::: === ::: === ::: === === ======== ===== ====== ======= === === ===== === === === === === === === === === === === === === === ====== === === ======= ====== ===
See? It's like the letters are little American flags! Just what every jingoist FAQ needs!
_____ _ _
|_ _|_ _ ___| | _| |__ ___ _ _
| |/ _` / __| |/ / '_ / _ | | | |
| | (_| __ <| |_) | (_) | |_| |
|_|__,_|___/_|__.__/ ___/ __, |
|___/
Vintage
==================================================== = ================ ===== ================== ==== =================== ===== ================== ==== =================== ===== ================== ==== ====== ==== === = == ====== === = = ==== ===== = == = == === === == = = ==== ======== === ==== ==== = == = === = ==== ====== ==== === === = == = ===== = ==== ===== = == = == = == = == = == = = ==== ====== === === = == ==== ==== == ====================================================
Reverse video, on the web!
I'm pretty sure this is the second or third to last page on the web.
[ed: due to a technical glich, I had to delete and re-add the previous entry.
Computers: feh!]
About this blog
The taskboy blog is a exploration of computer technology by Joe Johnston. Topics of posts include practical examples Perl, PHP, Python and Java as well as book reviews, industry insights and miscellaneous good stuff.
Current Status
Watching _Brass Latern_. Ah IF, your coyness is your charm.
Posted: Sun Sep 05 16:02:15 +0000 2010
Latest Feedbag
- Stadiums vanish, but their debt lives on
- Hillary Clinton on America's future: US retains role as world leader
- Stephen Hawking looks at the cosmos in 'The Grand Design'
- Need Niche Network Group Buying Deals? Meet ChompOn
- Q&A: Five key questions about midterm elections in Congress
- Microsoft intros Kinect bundle
- European Parliament All But Rejects ACTA
- Grain Sack Doubles Up As A Water Purifier Kit
- BMW Takes Internet Car Reveals To A Weird New Level
- Monocolumn: Imelda Marcos, Mark 2
Generated: 10:45 on 08/Sep/2010
Recent posts
- Very quick git primer for basic functionality
- Tips for spammers: don't insult me
- CakePHP vs. Symfony: a quick note
- Creating events for Yahoo and Google calendars
- SANs on a budget: iSCSI under Ubuntu
- iPad, iTouch and Kindle: Which is the better mousetrap?
- Rise of the Ad-Hocracy, Part II
- Rise of the Ad-Hocracy, Part I
- Small Hiatus
![[RSS feed icon]](/blog/img/feed-icon-14x14.png){kind=icon}
RSS Feed