The Upper Hand

When coding I quite often find myself having to setup some state temporarily limited to just the block I’m in and the routines the block calls.

For way of a simple example, imagine we have a password reset utility on our website. In our example resetting works by sending a user a url with a token in it to email address associated with their account, and then when the user clicks on that url they’re sent to a a page where they can send us the token and a new password.

The core of the code to do the actual password resetting might look something like this:

sub reset_password_from_email_token {
  my $self = shift;
  my $token = shift;
  my $password = shift;

  # temporarily disable security checks as this user isn't the one
  # the session is logged in as
  $self->disable_security_checks;

  if (any { $_ eq $token } $self->recent_reset_tokens) {
    $self->set_password($password);
    $self->remove_reset_token($token);
  }

  # turn the checks back on again
  $self->enable_security_checks;
 }

This is fairly reasonable code, but prone to subtle bugs.

What if there’s a problem with setting the password? For example, set_password could have easily been written to raise an exception if the password isn’t long enough:

sub set_password {
  my $self = shift;
  my $value = shift;

  if (length($value) {value} = $value;
  return $self;
}

And it’s fairly reasonable for someone therefore to write something like this:

eval {
  $user->reset_password_from_email_token($token, $password);
};
if ($@) {
  if ($@ =~ /too short/) {
    return render_bad_password_page();
  } else { die $@ }
}
return render_password_reset_page();

Have you spotted the problem yet? Yep, enable_security_checks never got called. render_bad_password_page is running with security off!

What we want is to ensure that security is always turned back on when we exit from reset_password_from_email_token no matter how we do that.

We want something like this psudocode:

sub reset_password_from_email_token {
  my $self = shift;
  my $token = shift;
  my $password = shift;

  # this user isn't the one the session is logged in as
  ...temporarily disable security checks somehow...

  if (any { $_ eq $token } $self->recent_reset_tokens) {
    $self->set_password($password);
    $self->remove_reset_token($token);
  }
}

Now how to write that? One very hacky way of doing it would be to localise a state variable that Perl will automatically restore to the original value as it exits the current scope, i.e. as it exits the subroutine:

sub reset_password_from_email_token {
  my $self = shift;
  my $token = shift;
  my $password = shift;

  # this user isn't the one the session is logged in as
  local $self->{security} = 0;

  if (any { $_ eq $token } $self->recent_reset_tokens) {
    $self->set_password($password);
    $self->remove_reset_token($token);
  }
}

Of course, this has several obvious drawbacks. Firstly, it requires the reset_password_from_email routine to understand how security works; If we ever change the security implementation of the module we’re going to have to alter this code too. Heaven help us if we try this approach on a third party module!

Secondly, it assumes that the implementation of security is sufficiently trivial that it can be controlled by a simple variable. This isn’t often the case. You may have to end up localising a whole collection of variables, or even running complex logic to work out what to do. Very very messy.

Quite simply, just resetting variables back to their original state isn’t powerful enough of a mechanism. What we actually would like to do is define some code that will be run on the exit of the subroutine.

One way to do that is to use the End module from the CPAN:

use End qw(end);

sub reset_password_from_email_token {
  my $self = shift;
  my $token = shift;
  my $password = shift;

  # this user isn't the one the session is logged in as
  $self->disable_security_checks;
  my $temp = end { $self->enable_security_checks };

  if (any { $_ eq $token } $self->recent_reset_tokens) {
    $self->set_password($password);
    $self->remove_reset_token($token);
  }
}

As long as $temp stays in scope nothing happens, but as soon as the subroutine exits and $temp goes out of scope the code we passed in will be executed.

How does that work? The End module is a way to create an instance that runs some code when it’s garbage collected. In the above example when $temp goes out of scope its DESTROY method will be called which will in turn call the anonymous subroutine that we passed in which calls enable_security_checks.

Great! We’ve solved the problem. We’ve almost invented a kind of backwards try / catch / finally syntax al-la Java and friends.

The problem with is it still requires me to write code every time I disable security, and therefore think, and therefore have a chance to introduce bugs. What I really really would like to do is write this:

sub reset_password_from_email_token {
  my $self = shift;
  my $token = shift;
  my $password = shift;

  # this user isn't the one the session is logged in as
  $self->temporarily_disable_security_checks;

  if (any { $_ eq $token } $self->recent_reset_tokens) {
    $self->set_password($password);
    $self->remove_reset_token($token);
  }
}

And it to essentially do the same thing, call disable_security_checks immediately and enable_security_checks at the end of scope. Is this possible? Yes, with the help of another CPAN module, Scope::Upper:

use Scope::Upper;

sub temporarily_disable_security_checks {
  my $self = shift;

  # disable security checks immediately
  $self->disable_security_checks;

  # and when the scope that called us exits, re-enable them
  reap sub {
    $self->enable_security_checks;
  }, UP;
}

Whoa! What happened there? Like end, the reap function exported by Scope::Upper allows us to to define an anonymous subroutine that will be called when a scope exits – but rather than the current scope, we can say when any scope in our call-chain exits. In this example we’re saying “UP”, which is a constant exported by Scope::Upper which means “in the scope that called us”, i.e. run this code when reset_password_from_email_token exits.

As you can imagine this is a really powerful mechanism that can be used to encapsulate complex logic. It’s useful for all sorts of things from cleanup exercises like I’ve shown here, to being really helpful in defining new keywords…

Testing times

One of my all time favourite things about the CPAN and Perl is that we ship test suites with our distributions that are run before installation. This means that we can be darn sure that not only does the code run on the developer’s machine, but it also runs on the end user’s machine correctly too – otherwise, it won’t be installed.

This obviously puts quite a burden on the developer to ensure that his or her code functions correctly if the test pass (this is what utilities like Test::Coverage help with.) I’d like to make a second point however: It puts a burden on the developer to ensure that if the tests pass if the code functions correctly.

For example, one thing that people do (myself included) is insist that their code is documented correctly by installing a test in their test suite that runs Test::Pod and Test::Pod::Coverage. These test suites will fail if the documentation isn’t in valid pod format, or you haven’t documented a method, respectively.

But hang on a minute, what if it’s not? Sure, the tests should fail on the developer system, and fail hard: You’ve got more work to do. Should your code now be uninstallable by end users? Does your code no longer work on the end user’s machine? Arguably yes, but at the same time arguably no. This kind of failure is a kind of “soft failure” that is often caused by differing test infrastructure on the developer and user machines, not by the previously passing pod somehow mutating after your bundled your machine.

I try and avoid these problems by disabling my pod checks unless someone has either explicitly asked for pod checking to be run by setting the POD_TESTS environment variable.

Here’s my standard 001pod.t file:

#!perl
############## STANDARD Test::Pod TEST - DO NOT EDIT ####################
use strict;
use Test::More;
unless ($ENV{POD_TESTS} || $ENV{PERL_AUTHOR} || $ENV{THIS_IS_MARKF_YOU_BETCHA}) {
    Test::More::plan(
        skip_all => "Test::Pod tests not enabled (set POD_TESTS or PERL_AUTHOR env var)"
    );
}

eval "use Test::Pod 1.14";
plan skip_all => "Test::Pod 1.14 required for testing POD" if $@;
all_pod_files_ok();

I also provide support for the PERL_AUTHOR environment variable which patchers can turn on while their patching my code and documentation (but keep off while installing modules from CPAN.) And because I’m really lazy I also put a test for the environment variable THIS_IS_MARKF_YOU_BETCHA in my code that is always enabled in my .bashrc on my development machine, meaning I don’t have to remember to turn these tests on for my own modules.

What does “ownership” of a module mean?

So, I was contacted by someone yesterday with patches for a module I wrote during my employment at Fotango. I thought it might be a good idea to release these patches, but looking at the module in question, I quickly realised that I didn’t actually have rights to upload an updated version, because the module in question is owned on CPAN by the FOTANGO CPAN user.

This complicates matters no end. You see, Fotango was shut down by it’s parent company quite a while back. There is no Fotango anymore. I no longer have access to the FOTANGO CPAN account, nor do I know any person that does.

I contacted Alias. He’s trying to sort it out.

I know, for example, in theory the property rights for Fotango revert to the parent company (though the exactly who owns that IP is something that the lawyers probably know – I’m not sure myself.) Obviously the code isn’t the issue here; The code is open source and licensed in such a way that it can be forked by me quite happily. The real question here is, is the CPAN ID intellectual property? Are the CPAN namespaces? Do they have value in the same way that a domain name does?

If the answer is yes, then we need to get then we’ve got bigger issues at foot than who owns the Fotango modules. Anyone who has released company code using their own CPAN ID rather than a company one (which is currently considered best practice) is registering IP that belongs to the company in their own name. That can’t be right.

If the answer is no, then the namespaces are just things that someone (presumably the PAUSE admins and/or the modules mailing list) control and “ownership” of a module is just a handy tag to let the automated systems do automated tasks so a real human doesn’t keep having to make judgement calls every five minutes. This doesn’t mean the real human doesn’t need guidelines and rules to work within however. What’s the correct thing to do when a company (or for that matter, human) dies? I think my argument stands that the “ownership” doesn’t transfer like other property rights.

I’m sure that Greater Minds than myself have already debated and come to conclusions on this; Anyone want to point me towords the guidelines?

Testing Perl 5.10.1 RC 1

UPDATE: Since this blog post was written, this got a lot easier with perlbrew. I wrote about this here

There’s a new stable release of perl in the works – Perl 5.10.1 – that’s reached “Release Candidate” stage.  In other words, we’ve got a shiny new version of the perl interpreter and associated core Perl modules that the pumpking (the person in charge of the release) thinks is ready, but he wants us to all test it on our systems with our code to make sure that there’s no problems he can’t spot himself before it’s finally released.

So how do we do that? Well, we can follow gugod’s instructions to build a version in our home directory. Below is what I did for my system (a Mac OS X 10.5 box, with the developer tools already installed from the CD that shipped with the computer)

First we download and extract the distribution:

bash$ wget http://search.cpan.org/CPAN/authors/id/D/DA/DAPM/perl-5.10.1-RC1.tar.gz
bash$ gunzip -c perl-5.10.1-RC1.tar.gz | tar -xvf -
bash$ cd perl-5.10.1-RC1

Then we then configure it.

bash$ Configure -de -Dprefix=${HOME}/local

-de means “Accept the defaults” and “Go on accepting the defaults”. -Dprefix tells the installer where to install – in this case in “local” in our home directory where it won’t interfere with the system Perl.

We then can tell it to build, test, and install itself.

bash$ make
bash$ make test
bash$ make install

This all will take some time. Either go make a cup of tea, or just get on with something else.

Whenever I want to test the new version of Perl I can use the export command to modify my path to put this new perl first.

bash$ export PATH=${HOME}/local/bin:${PATH}

This means that when I type perl from my shell (or use any of the other perl utilities in this shell) from that point on it’ll run the version of perl I’m testing rather than the system perl that came with my Mac.

bash$ perl -v

This is perl, v5.10.1 (*) built for darwin-2level
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2009, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

We can now start installing modules from the CPAN in order to test out our new perl. I must admit that I jumped straight for the big guns: Task::Kensho.

“Tasks” are psudo-packages that you find on the CPAN that don’t really contain any code of their own, but just provide documentation and depend on a bunch of other modules. They’re a common way to load a collection of modules. Task::Kensho is one of the big ones here – it’s a large-ish collection of modules that are commonly used by modern Perl programmers, and installing it basically will tell cpan to install all the modules that I need to get up and running and start being productive.

It took me about an hour to install all of these modules from a local on disk cpan mirror I had previously created with minicpan, but quite a lot of that was delays caused by me configuring my cpan shell to prompt me before doing anything (because I wanted to keep track of what it was doing, as I was testing the new release candidate.)

All in all it was a fairly painless experience. I look forward to the release proper of 5.10.1 in the near future!

say “Hello World”

This is a blog.  There are many like it but this one is mine.

– badly paraphrasing Full Metal Jacket

Hi.  You might know me.  I’m Mark Fowler.  No, not the fictional guy from Eastenders or the deceased serial killer, but the Mark Fowler whose best known on that there Internet in connection with the Perl programming language.

And I’ve decided to do a very silly thing.  Matt Trout, king of the rants and opinions, is trying to encourage more people to write Perl blogs.  And I said that not only that I would but I’d also post at least once a week as some part of a crazy iron man challange.  Ooops.  That sounds like a lot of work.

Why have I decided to do that?  Well, I do a lot of Perl stuff that’s worth talking about, but I don’t really write about it any more, which is a shame because I think it’s interesting at least.

I used to enjoy keeping a very active blog on use.perl.org, but I got fed up with the whole primitive nature of use.perl.org. It was obviously time to move somewhere else.  And being a geek, I though it might be fun to write my own blogging engine to do that.  This is Yak Shaving at it’s best. How long could it possibly take to write your own blogging software?  Well, if all you want to do is knock out a few pages of text, not long at all.  If you however want rich GUI editors, open ID support, trackbacks, comments, anti-spam filters, gravatar support, previewing and draft posts, workflow management, and about a gazillion other things that make up a modern blog it might take a smidgen more than not long at all.

This is even more of a problem if you’ve in fact got less than a smidgen of spare time because:

  1. You’re a professional Perl programmer who spends all his time doing much more serious Perl development or running teams doing professional Perl development, so you don’t actually feel like doing boring aspects of projects in your spare time.
  2. You hence look for shiny and fun projects to play with in your spare time because it’s your spare time, so you want more instant payback.
  3. You took on a big project half way through the rewrite that’s much much more fun, but very demanding on your time and can’t be put down.  Little baby geek in training, I’m looking at you ;-)

In other words it went the way of all other programming projects:  The requirements were not properly scoped and estimated, and the fact that other, more pressing projects had to be delivered in the same timescale wasn’t accounted for.

You see, I forgot to apply one of the virtues of a Perl programmer;  Laziness. Or, put another way, I should have applied a little agile methodology and done the least possible thing I could have done to get the desired result.  What I should have done is said “What software have other people written that I can use, and if need be, alter to do what I need.”.  Or, in the days of cloud computing, I should have said “What hosted service can I use that does everything for me, from deployment to working out how to make it scale.”  So now after spending literally minutes filling out forms online I have a wordpress blog and I can actually start blogging again.

So, having started the new blog describing how much of a Class One Idiot I am, why should you listen to me?  What am I likely to discuss?

Posts that are in the draft folder currently include

  • A bunch of discussion of Perl modules I’ve started using in the last few years.  I used to review 25 modules a year for the Perl Advent Calendar and I miss writing about cool modules.  (I don’t miss the grueling post-per-day schedule however)
  • How Devel::Declare is allowing redevelopment of the Perl language directly from the CPAN, and how we’re even using in production now
  • Something on how Ash Berlin’s TryCatch module is the improvement to Perl I’ve been waiting for my entire programming career.
  • Waffle on XML::Easy, my new favourite XML module of choice, and how I used it to create Test::XML::Easy.
  • Improvements to my Perl modules;  Where I’m heading with Test::Builder::Tester after leaving that code base dormant for so long.
  • How I’m moving all my code to github, and what changes I’m making to the packaging of all my CPAN modules as a matter of course as I do so.

And eventually I’m going to talk about my super secret Perl project when it comes out of stealth mode.  But I’ll leave that for another post…

Follow

Get every new post delivered to your Inbox.