Tell me more, tell me more

A new version of Test::DatabaseRow has just escaped onto the CPAN, with a minor new feature verbose data to make working out what went wrong in the database test even easier when the worst happens

In a nutshell this feature allows you to output in diagnostics all the results the database returned (not just listing the first thing that didn’t match, which is the default thing row_ok does in a similar way to Test::More’s is_deeply does.)

In other words instead of writing this:

    all_row_ok(
       table => "contacts",
       where => [ cid => 123 ],
       tests => [ name => "trelane" ],
       store_rows => \@rows,
       description => "contact 123's name is trelane"
     ) or diag explain \@rows;

You can just write this:

    all_row_ok(
       table => "contacts",
       where => [ cid => 123 ],
       tests => [ name => "trelane" ],
       verbose_data => 1,
       description => "contact 123's name is trelane"
     );

Or even just write this:

    all_row_ok(
       table => "contacts",
       where => [ cid => 123 ],
       tests => [ name => "trelane" ],
       description => "contact 123's name is trelane"
     );

And turn on verbosity from the command line when you run the tests

TEST_DBROW_VERBOSE_DATA=1 perl mytest.pl

Mavericks, XCode 5 and WWW::Curl

Sometimes you go through a lot of pain trying to get something to work, and you just need to write it down and put it on the internet. You tell yourself this is useful; You’re doing it so that others can find your words and solve the problem too. In reality, it’s just cathartic. And who doesn’t like a good rant?

So What Broke?

I recently updated my Mac to Mavericks (OS X 10.9). Overall, I like it…but it broke something I care about. It broke libcurl’s handling of certificates., and in so doing broke my install of WWW::Curl also (since it’s a wrapper around the system libcurl.) This is remarkably hard to diagnose, because it just seems like the servers have suddenly stopped trusting each other (when in reality libcurl has just started effectively ignoring the options you’re passing to it.)

Now, WWW::Curl (and its working certificate handling) is a dependency of some of the code I’m using. Worse than forcing me to disable certificate security, I actually need to identify myself to the server with my own cert. Just turning them off and running insecurely won’t work.

Installing libcurl

I’m not a great fan of installing software from source on my mac after having been bitten by fink and it’s kin numerous times in the past. This said, homebrew is actually pretty darn nifty, and can be used to install libcurl:

brew install curl   

Wrrr, clunk, clunk and zap, I have a non broken curl in /usr/local/Cellar/gcc47/4.7.3/bin. Hooray! This one actually deems to listen to the command line options you pass it!

Installing WWW::Curl

Now all we have to do is install WWW::Curl and link it against this libcurl, right? Well, I did this:

wget http://cpan.metacpan.org/authors/id/S/SZ/SZBALINT/WWW-Curl-4.15.tar.gz
gunzip -c WWW-Curl-4.15.tar.gz | tar -xvf -
cd WWW-Curl-4.15
export CURL_CONFIG=/usr/local/Cellar/curl/7.33.0/bin/curl-config
perl Makefile.PL

And things started to go wrong:

The version is libcurl 7.33.0
Found curl.h in /usr/local/Cellar/curl/7.33.0/include/curl/curl.h
In file included from /usr/local/Cellar/curl/7.33.0/include/curl/curl.h:34:
In file included from /usr/local/Cellar/curl/7.33.0/include/curl/curlbuild.h:152:
In file included from /usr/include/sys/socket.h:80:
In file included from /usr/include/Availability.h:148:
/usr/include/AvailabilityInternal.h:4098:10: error: #else without #if
        #else
         ^
/usr/include/AvailabilityInternal.h:4158:10: error: unterminated conditional directive
        #if __MAC_OS_X_VERSION_MIN_REQUIRED >= __MAC_10_6
         ^
/usr/include/AvailabilityInternal.h:4131:10: error: unterminated conditional directive
        #if __MAC_OS_X_VERSION_MIN_REQUIRED >= __MAC_10_5
         ^
/usr/include/AvailabilityInternal.h:4108:10: error: unterminated conditional directive
        #if __MAC_OS_X_VERSION_MIN_REQUIRED >= __MAC_10_4
         ^
4 errors generated.
Building curlopt-constants.c for your libcurl version
Building Easy.pm constants for your libcurl version
Building Share.pm constants for your libcurl version
Checking if your kit is complete...
Looks good
Writing Makefile for WWW::Curl
Writing MYMETA.yml and MYMETA.json

That Ladies and Gentlemen is the sound of Apple’s compiler sucking. You see, not so long ago I upgraded to XCode 5 and apparently this no longer ships with gcc. Uh oh.

(I don’t show the errors you get when you make and make test, but be assured it’s all downhill from here)

Installing gcc

I suspect I could have just re-downloaded XCode 4 and then used xcode-select. But I didn’t. Let me know if that works, okay? This is what I actually did:

brew install gcc47

Then I waited a long time (while doing something else productive instead.) Finally when I was done I had to manually edit the Makefile.PL to use the right preprocessor:

perl -pi -e 's!cpp"!/usr/local/Cellar/gcc47/4.7.3/bin/cpp"!' Makefile.PL
perl Makefile.PL

(yeah, it’d be nice if they’d been some sort of option for that.) Then I altered the resulting Makefile for good measure too:

perl -pi -e 's{CC = cc}{CC = /usr/local/Cellar/gcc47/4.7.3/bin/gcc-4.7}' Makefile

And then I could build it all.

make
make test

Of course, it still got errors. But they’re not new errors. So I pretend I didn’t see them and install anyway.

make install

I hate computers sometimes.

5.18 Hash Key Ordering Changes In A Nutshell

$ perlbrew switch perl-5.16.3
$ perl -E '%h=map {$_=>1} (1..10); say join ",",keys %h for 1..3'
6,3,7,9,2,8,1,4,10,5
6,3,7,9,2,8,1,4,10,5
6,3,7,9,2,8,1,4,10,5
$ perl -E '%h=map {$_=>1} (1..10); say join ",",keys %h for 1..3'
6,3,7,9,2,8,1,4,10,5
6,3,7,9,2,8,1,4,10,5
6,3,7,9,2,8,1,4,10,5
$ perlbrew switch perl-5.18.0
$ perl -E '%h=map {$_=>1} (1..10); say join ",",keys %h for 1..3'
10,2,8,3,6,9,5,4,1,7
10,2,8,3,6,9,5,4,1,7
10,2,8,3,6,9,5,4,1,7
$ perl -E '%h=map {$_=>1} (1..10); say join ",",keys %h for 1..3'
4,6,9,1,10,3,2,7,8,5
4,6,9,1,10,3,2,7,8,5
4,6,9,1,10,3,2,7,8,5

Now each execution has its own hash key ordering (but hash key ordering is the same for the duration of the execution until you insert a new key.)

perl-5.18 increases the chance that keys will be reordered when inserts happen:


$ perlbrew switch perl-5.16.3
$ perl -E 'for (1..20) { $h{$_}=1; say join ",",keys %h }'
1
1,2
1,3,2
4,1,3,2
4,1,3,2,5
6,4,1,3,2,5
6,4,1,3,7,2,5
6,3,7,2,8,1,4,5
6,3,7,9,2,8,1,4,5
6,3,7,9,2,8,1,4,10,5
6,11,3,7,9,2,8,1,4,10,5
6,11,3,7,9,12,2,8,1,4,10,5
6,11,3,7,9,12,2,8,1,4,13,10,5
6,11,3,7,9,12,2,14,8,1,4,13,10,5
6,11,3,7,9,12,2,15,14,8,1,4,13,10,5
11,7,2,1,16,13,6,3,9,12,14,15,8,4,10,5
11,7,17,2,1,16,13,6,3,9,12,14,15,8,4,10,5
11,7,17,2,1,18,16,13,6,3,9,12,14,15,8,4,10,5
11,7,17,2,1,18,16,13,6,3,9,12,14,15,8,4,19,10,5
11,7,17,2,1,18,16,13,6,3,9,12,20,14,15,8,4,19,10,5
$ perlbrew switch perl-5.18.0
$ perl -E 'for (1..20) { $h{$_}=1; say join ",",keys %h }'
1
2,1
1,2,3
2,4,1,3
4,1,2,5,3
2,4,1,5,3,6
4,1,2,6,5,7,3
7,8,1,3,5,6,2,4
4,2,6,3,5,8,1,9,7
9,7,8,1,6,3,5,10,4,2
9,7,11,8,1,6,5,10,3,4,2
11,8,1,7,9,2,12,4,5,10,3,6
4,13,12,2,6,3,5,10,8,1,11,9,7
11,14,8,1,7,9,2,12,4,13,5,10,3,6
4,13,12,15,2,6,3,5,10,14,8,1,11,9,7
9,16,7,8,14,3,13,12,15,2,1,11,6,5,10,4
5,10,17,6,4,11,1,3,2,12,15,13,16,7,9,8,14
4,17,6,5,10,1,11,13,12,15,2,18,3,8,14,9,16,7
17,6,5,10,4,1,11,19,3,13,12,15,2,18,9,16,7,8,14
11,1,4,5,10,17,6,8,14,16,7,9,2,18,20,12,15,13,3,19

This can be controlled with the PERL_PERTURB_KEYS environment variable


$ perlbrew switch perl-5.18.0
$ PERL_PERTURB_KEYS=0 perl -E'for (1..20) { $h{$_}=1; say join ",",keys %h }'
1
2,1
3,2,1
3,2,1,4
5,3,2,1,4
5,6,3,2,1,4
5,6,3,2,1,4,7
5,2,3,6,8,1,4,7
5,2,3,6,8,9,1,4,7
5,2,3,6,10,8,9,1,4,7
5,2,3,6,10,11,8,9,1,4,7
5,2,3,6,10,11,12,8,9,1,4,7
5,2,3,6,10,13,11,12,8,9,1,4,7
5,2,3,6,10,13,11,14,12,8,9,1,4,7
5,2,3,6,10,13,11,14,12,8,15,9,1,4,7
3,14,12,16,8,15,9,4,7,5,6,2,10,11,13,1
17,3,14,12,16,8,15,9,4,7,5,6,2,10,11,13,1
18,17,3,14,12,16,8,15,9,4,7,5,6,2,10,11,13,1
18,17,3,19,14,12,16,8,15,9,4,7,5,6,2,10,11,13,1
18,17,3,19,14,12,16,8,15,9,4,7,5,6,2,10,11,13,20,1

Posted in Uncategorized

Permalink 2 Comments

Another Feature Perl 5 Needs in 2012

In Features Perl 5 Needs in 2012 chromatic asks the question What’s on your list?  Well, my list is long.  But the thing at the top is structured core exceptions.

While I don’t think Perl 5’s exception throwing syntax is completely solved by Try::Tiny (just as it’s time for Perl 5 to have it’s own MOP it’s time for Perl 5 to have its own exception handling syntax to solve that problem in a well-defined way too) that’s not the biggest stumbling block to having awesome exception handling in Perl 5.  The main problem is that Perl 5 still throwing plain old strings as exceptions.

Often I find myself wanting to catch all IO errors, but not runtime stupid-coding errors (My error code is going to handle  a dumb filename, but it’s totally not going to know what to do if I’ve typoed a method name.)  This is hard because my code has to parse the exception string – essentially throw a bunch of regular expressions at it – until it can hopefully figure out what’s going on.  And, of couse, I say hopefully because the strings can (and should, as improvements are made) change between versions of perl.

The correct solution is for Perl to throw structured exceptions.  Where all IO errors are a subclass of the main IO error and all bad method calls are a subclass of the DispatchError error or somesuch.  Then I only have to check that $@ isa IO Error and the whole problem becomes simple.

Now in theory someone could build a version of Try::Tiny that had all this logic built in – did all this parsing for me – so it would seem like I had structured exceptions for all the inbuilt errors.  And as long as this module was updated for each version of perl this would probably work and maybe just be good enough.  But it’s not the correct solution!  It’s going to be slow (oh so slow) and brittle and…bad and right.  I want (no, I demand!) a good and right solution.  Enough with the band-aids!

So structured core exceptions goes to the top of my list.  Which of course means very little since I’m not a core Perl hacker (nor do I play one on TV) and I’m not going to have time or skills to do it myself.  But if the TPF wants to take some of that money I’m paying for core development every month and pay someone to do it, I’d be a very happy Perl programmer. 

Posted in Uncategorized

Permalink 9 Comments

On Diversity in Tech Communities

One of the advantages of working from home is that I can do more interesting things with my lunch break that simply eat lunch. So, every week for three years I took my daughters to a local sing-a-long group.

Now as you can imagine, this group was primarily made up of mothers and their children. The people who run the sessions are women. On the odd occasion another father would turn up, but it was mostly just me and thirty women and their children.

No-one ever implied that I shouldn’t be there. No-one ever made jokes about men being useless. People didn’t try to have “Cosmo” type conversations with me that would make me blush. No-one even made any comment implying it was un-manly for me to sing along with nursery rhymes like all the other parents did.

All in all you could say it was great. I’d never accuse anyone at any of these events of being sexist.

But then again, every so often the group would sing a song about Bobby Shaftoe. For those of you not familar the lyrics go:

Bobby Shaftoe went to sea,
Silver buckles on his knee.
He’ll come back and marry me,
Pretty Bobby Shaftoe.

I never – in three years – spoke up about how uncomfortable these lyrics are for a straight man to sing. In the end I just stopped singing them.

Did this really bother me that much? Not really, otherwise I would have said something. But it let me experience in the tiniest possible little way what it’s like to be suddenly reminded that you’re different to everyone else in the group and to find out that you can’t join in what everyone else is doing because it’s not designed for you.

So, with this in mind I wish that people would understand that when I’m suggesting a code of conduct for a tech community my primary objective is not to suggest a list of things you can and can’t do. Nor am I suggesting that people are deliberately being nasty. I’m just trying to encourage everyone to think a little wider about the other people in their community that aren’t just like them – because even the best of us sometimes can have a blind spot. You know, it’s not always about the big things.

Sometimes I just don’t want community member to have to sing songs about their desire to marry a sailorman. And if they do find themselves in a situation where someone is asking them to declare their desire for silver buckled knee wearers that they feel like they can politely point out that they shouldn’t have to.

That is all.

Oh Function Call, Oh Function Call, Wherefore Art Thou Function Call

Ever wondered where all places a function is called in Perl? Turns out that during my jetlagged ridden first day back at work, I wrote a module that can do just that called Devel::CompiledCalls.

Use of the module is easy. You just load it from the command line:

shell$ perl -c -MDevel::CompiledCalls=Data::Dumper::Dumper myscript.pl

And BAM, it prints out just what you wanted to know:

Data::Dumper::Dumper call at line 100 of MyModule.pm
Data::Dumper::Dumper call at line 4 of myscript.pl
myscript.pl syntax OK

While traditionally it’s been easy to write code using modules like Hook::LexWrap that prints out whenever a function is executed and at that point where that function is called from, what we really want is to print out at the time the call to the function is compiled by the Perl compiler. This is important because you might have a call to the function in some code that is only executed very infrequently (e.g. leap year handling) which would not be simply identified by hooking function execution at run time.

In the past developers have relied too much on tools like search and replace in our editors to locate the function calls. Given that Perl is hard to parse, and given that many of the calls might be squirreled away in installed modules that your editor doesn’t typically have open, this isn’t the best approach.

What Devel::CompiledCalls does is hook the compilation of the code (techically, we hook the CHECK phase of the code, but the effect is the same) with Zefram’s B::CallChecker. This allows the callback to fire as soon as the code is read in by Perl.

All in all, I’m pretty happy with the module and it’s a new tool in my bag of tricks for getting to grips with big, unwieldy codebases.

Posted in Uncategorized

Permalink 1 Comment

Coming To America

Big changes are afoot

Not many people know this yet, but Erena, myself and the girls are in the process of emigrating. Yesterday I completed purchase of our new house in New Lebanon, New York, in the United States of America. Yes, this means that I’m going to be attending London.pm meetings even less than I did when I moved to Chippenham.

One of the consequences of my move is a change of jobs. At the end of April I’ll be leaving the wonderful Photobox and starting to do some work for the equally wonderful OmniTI.

Photobox is a great place to work, with a world class Perl team. In the last four years I’ve worked for them I haven’t complemented them enough. What other place can I work next to regular conference speakers? Core committers? People with more CPAN modules than I can shake a stick at? Perl Pumpkins themselves! To be blunt, if I was to stay in the UK I’d can’t imagine I’d want to work anywhere else.

It’s been an interesting four years, seeing the company grow and grow and dealing with the scalaing problems – both in terms of the number of customers and the challenges of growing the team – and I’ve really enjoyed it. But new adventure beckons, and I needn’t sing the praises of OmniTI to tell everyone here what great company they are and I’m going to be honoured to work with them.

So, once I’ve completed the gauntlet of my Green Card application (which is scheduled to take many months yet) it’ll be off with a hop, skip and a jump over the pond for a new life. Can’t wait.

Posted in Uncategorized

Permalink 3 Comments

Follow

Get every new post delivered to your Inbox.