Damian Conway on Perl and its future

Three friends, Dave Cross, Greg McCarroll and Josette Garcia, met at YAPC Europe in Lisbon wanting to update the world on the status of Perl. As Damian Conway, one of the key designers of Perl 6 was giving a talk, we thought we should ask him a few questions. The following questions took a long time to put together, possibly in sympathy with Perl 6.

— Perl Dead

Dave: We often hear people saying that Perl is dead. I assume that you don’t agree, but what do you say to people who try to tell you that?

I usually say “I agree. Sad, isn’t it. Ah, well.” This causes the crazy person to go off quietly, so I can get back to uploading new or improved Perl modules to extend CPAN’s 7GB of open source software (as over 700 other developers also do every month), or attending several of the dozen or more Perl conferences and workshops held all round the world every year, or visiting some of the over 250 Perl user groups in 58 countries around the world (www.pm.org), or interacting with the thousands of like-minded people helping each other study at Perl’s premier learning website (www.perlmonks.org), or participating in other online Perl social and study activities (such as the Perl Iron Man Challenge), or upgrading to the new releases of Perl 5 and Perl 6 (both of which are now on a monthly release cycle), or contributing to the active development processes for either of the two implementations of Perl 5 or any of the four implementations of Perl 6, or teaching the dozens of major new features that have been added to Perl 5.10 and Perl 5.12 over the past three years, or getting involved in the various Perl renaissance activities (www.enlightenedperl.org and www.modernperlbooks.com), or just reading Tim Bunce’s excellent summary of how dead Perl has never, in fact, been more alive.

I can only hope that, when I’m dead too, I’m as vibrant and active and still growing as strongly as Perl is now.

Greg: Many people are faced with managers who see Perl as a dying language – apart from “get a new job,” what advice would you give to them to evangelise Perl?

Well, first of all, see above.

Also, they need to look at the issue from their manager’s perspective. Many Perl developers see their goal as being to develop in Perl; their managers see their goal as being to develop cheaply, effectively, reliably, robustly, maintainably, quickly, and with as little risk as possible. In whatever language achieves all that.

So Perl advocates need to focus their arguments for using Perl on those needs, rather than on their own desire to use Perl. In other words, they need to make a business case for Perl, rather than just a technical case.

Such a business case might include the high number of extremely good programmers available for Perl development, the familiarity and skills of their existing team with Perl, the huge resource of free software that is CPAN, the large number of mature, powerful and scalable frameworks now available for Perl, the speed with which Perl can be used to implement and deploy systems, how the high-level nature of Perl eliminates common low-level problems, the huge amount of high-quality (and often free) support that’s available for Perl, the number of large organizations who have successfully delivered mission-critical systems in Perl (for examples see: www.perltraining.com.au/whyperl.html#who, www.perlfoundation.org/perl5/index.cgi?companies_using_perl, www.london.pm.org/advocacy, www.proudtouseperl.com, or blog.listcentral.me/2009/05/22/companies-that-use-perl)

Selling Perl to your manager is about explaining Perl in terms that your manager cares about: stable, reliable, powerful, efficient, cheap, maintainable, well-supported, and future-proof.

Josette: Recently Martin Drashkov wrote “Why Perl lost It” (http://martin.drashkov.com/2009/11/why-perl-lost-it.html). What do you think of the final statement: “Whether Perl will continue to decline or not is hard to say, though I highly doubt it’ll ever return to its former glory – neither Perl5 nor Perl6 seem to offer much for developers who do not already love Perl.”?

With due respect to Martin, whose other work I admire, I think that his initial premise (that Perl is declining) is extremely dubious and that his final observation is strongly contradicted by the reactions I get whenever I tell people about Perl 6.

I think it’s true that recent versions of both Perl 5 and Perl 6 do have a huge amount to offer those who *do* love Perl, but I also think that Perl 6 at least has a huge amount to tempt those who has previously found Perl 5 unsatisfactory. From its immensely sophisticated OO model to its extraordinarily powerful built-in grammars to its easy-to-use native concurrency and data parallelism to its immensely handy multiple dispatch semantics to its cleanly integrated first-class macro facility to its incredible support for creating domain specific languages, I think Perl 6 has a huge amount to offer *all* developers. We’ve spent a decade stealing the very best idea from the best programming languages, and making them simple and practical for mortal developers to use. I think the result is going to appeal to a lot of people.

By the way, the article you quote is still well worth reading…especially the comments at the end of it, which more or less demolish every point Martin makes.

Greg: One thing a lot of Perl programmers face is large old codebases, full of different coding styles, often difficult to write unit tests for, drive-by refactoring is a common solution, but do you have any silver bullet ideas or even just ideas for understanding and working with these aging codebases?

No silver bullets, but one simple and blindingly obvious principle: “Start Evolving…Today!”.

Refactoring a large heterogeneous codebase is usually neither feasible or even helpful (the word “enbugging” leaps to mind), but at least you can stop adding to the problem as you continue adding to the code.

The simplest approach is to get your team to agree on a coding standard and testing process and then start applying those constraints to any new (or frequently maintained) portions of the codebase. At very least, you’re not making the problem worse and, as the software expands, its percentage of cruddy code will necessarily diminish.

It’s no magic wand, but it does work. Evolving the code style and introducing testing frameworks as you maintain the code focuses your efforts on the parts of the codebase where improvements will most benefit the team; namely, those parts where they most frequently interact with the code. And if they’re consistently using their new coding style and testing frameworks in new development, it will be easy apply them correctly when reworking existing code.

Finally, because they’re refactoring and restyling code that they’re actively maintaining, they’ll be more likely to reimplement it correctly.

— Perl 6 Delay

Dave: In retrospect, is there anything that you and Larry would have done differently with the Perl 6 design process? Is there any way that it could have taken less than nine years?

Sure. We could have done a worse job. We could have simply patched Perl 5 with the most popular requests from the RFC process. We could have accepted the less-clean, less-well-integrated, less-clever design that we had five years ago. We could have let the good be the enemy of the best. But I’m glad we didn’t.

The evidence is that most major new programming languages take about a decade to reach a stable and useful design. C++ did, Java did, Perl 5 did, Haskell did, Python 2.0 did, Standard ML did. ANSI C arguably took two decades to get right, and Lisp took either two or four (depending on whether you think Scheme or Common Lisp Scheme was the final “correct” incarnation).

So when people point to the fact that the Perl 6 design process has taken 10 years, I consider that to be a sign that we did it right.

The only real difference between Perl 6 and most other languages is that we had the courage (or perhaps the foolhardiness) to carry out that inevitable decade-long design process in public. In retrospect it would have been politically cleverer to have worked quietly on it from 2000, announced the project in 2008 and delivered it this year.

Except, of course, that without the help, insight, and feedback of the dozens of highly talented people the public process has attracted, Perl 6 might still have taken ten years to design, without being nearly as good.

Greg: A lot of work has gone into regular expressions and parsing in Perl 6, with XML/YAML available for data and with ‘little languages’ being a niche technology, do you think this focus has been worth it?

Interesting question. You might have asked exactly the same question about Perl 5, whose regex mechanism has had two decades of improvement lavished on it. Has that proven worthwhile? I think so.

As for Perl 6, I think it’s impossible to accurately estimate just how valuable a built-in grammar mechanism is until you actually have one. It’s true that XML and YAML/JSON cover a lot of data representation needs, but there is a vast amount of data that isn’t available in those formats and will never be unless it’s parsed and converted. And a grammar is unquestionably the best way to do that.

As for the niche of DSLs, I think it’s been a niche precisely because the necessary tools simply weren’t available to implement little languages cleanly, easily, and efficiently. One of the reasons we’ve worked so hard on Perl 6’s regexes and grammars is to make developing those DSL interfaces very much easier.

The other reason we’ve spent so much time on regexes and parsing is that Perl 5 regexes, which are easily the most powerful and sophisticated in a mainstream language, also now have a near-fatal problem. As they’ve evolved from the simple syntaxes of the 1970s, they’ve become “trapped” by the ever-diminishing syntax available for new constructs.

Virtually every major improvement to Perl 5 regexes over the past decade has a syntax of the form “(?<something>)”. A modern regex that’s making use of the full power of Perl 5 regular expressions looks more like Lisp than Perl! It’s as if the long-term evolution of the regex syntax had produced 10 useful organs and 30 veriform appendixes.

It was time to redesign the entire organism, so that the simplest syntax could be assigned to the most useful constructs (instead of being monopolized by those constructs that happen to arrive first). We think of this as analogous to finding the optimal Huffman coding for regexes, so that commonly used regex features become easier to write and easier to read.

And when you see how very much easier Perl 6 regexes and grammars are to work with, and even just to read, it’s clear to me that the result was worth the considerable effort required.

— Perl 6 Released

Dave: At YAPC in Lisbon, Patrick announced that a first version of Perl 6 (called Rakudo Star) will be released in Spring of 2010. What are you looking forward to most about seeing Perl 6 out in the wild?

That’s easy: what I’m most looking forward to is not being asked “When will Perl 6 be released???”

But seriously, I’m really looking forward to being able to write production systems in Perl 6. The language is just a joy to code in, and it’s been enormously frustrating to have been able to deploy it only in my head for the last few years. Now that Rakudo is usable, I’ve switched most of my development to Perl 6, and it’s like stepping out of a prison I didn’t even know I was in (exactly as it was when I first moved from Pascal to C, and then again from C to Perl).

The other thing I’m really looking forward to is seeing what the wider Perl community does with this new tool. I have big plans for new projects myself, but I’m far more excited to see the amazing, brilliant, and utterly unexpected wonders our community will create on top of Perl 6.

Greg: Unofficially, but with an educated guess which year do you think Perl 6 will be used in a business critical application in say 10 reasonably sized (5 million+ Euro turnover) companies, and will it be greenfield development or a conversion from existing Perl code?

Whenever it is, it will almost certainly be a fresh codebase (though it might easily be a clean reimplementation of an existing Perl 5, Java, or Python app). I think the vast majority of Perl 6 development will be the creation of new code, rather than the migration of existing Perl 5 code. Simply because Perl 5 will still be there, and still kicking butt, so there won’t be much incentive to migrate existing software.

As to when, I’d say we’re probably looking at three to five years for Perl 6 to achieve that level of penetration, acceptance, and trust.

Greg: Which do you think are the best features Perl 5.x has gained through
Perl 6 work?

Tough question. Personally, the one I love most is by far the least sophisticated: the “say” function that was retrofitted to Perl 5.10. It solves the commonest annoyance in Perl 5: the perennial need to add a newline at the end of most “print” statements. Not very sexy, but it scratches an incessant tiny itch that used to niggle me at least twenty times a day.

I’m also very partial to smartmatching and the given/when construct. I find I use them surprisingly often in my current Perl 5 development.

Dave: When will Perl 6.1 be released? And what will it contain?

Probably at least a year after Perl 6.0, which is itself still six to twelve months away. I’m afraid that’s not really a question I’m competent to answer, it being entirely the bailiwick of the implementers.

As to what we’re likely to add to Perl 6 for the next major increment, hopefully not very much. The whole point of the Perl 6 design is that we’ve tried to add enough power so that anything we missed can be added from within the language, rather than having to add it onto the language itself.

That said, I think the focus of our design efforts from this point will be on Perl 6’s threading and concurrency models, and I expect that Perl 6.1 will be much more sophisticated and complete in that respect.

— Perl / Computing in general / Anything Else

Dave: Which of your many CPAN modules do you think is the most useful? And which is the least?

The most widely used seem to be Parse::RecDescent, Text::Autoformat, and Regexp::Common. Whether that means they’re the most useful, I’m not sure.

From another point of view, you could also argue that the Switch module has been the most useful since, despite its obvious deficiencies (or perhaps because of them!), it eventually led to the addition of native versions of given/switch and smartmatching in Perl 5.10 and Perl 6.

The least useful? That’s actually much harder to determine than it should be. Lingua::Romana::Perligata (Perl in grammatical Latin) and Coy (error messages translated into haiku) are obvious candidates, except that I know that both have actually been used in production.

So it’s probably Acme::Bleach (Perl encoded in pure whitespace). Especially because I know that people have also mistakenly tried to use that module to “protect” their source code. Sigh.

Josette: I keep being told that programs written in Perl are often impossible to read by other Perl programmers and that this problem does not occur with programs written in other languages such as Python. Do you agree with this statement and can you explain why this happens?

I hear this all the time, and it’s just not accurate. I see a lot of different Perl code in many different styles, and I can read most of it without trouble. I see a lot of codebases with self-inconsistent coding styles, which other Perl programmers nevertheless manage to maintain without undue difficulty.

Of course, a standard set of coding conventions is an excellent idea and should be everybody’s first choice. But coding consistently in a readable style does require a certain discipline that doesn’t come naturally to everyone. There’s a necessary learning curve there: for the individual programmer, for programming teams, and for the entire Perl community. I think as a community we’re making good progress, but that the message of consistency and writing-for-readability is still filtering down to some individual developers.

And, yes, I often hear the claim that code written in Python (or Java or Eiffel or any other language less flexible than Perl) is ipso facto more readable. But that claim neglects a fundamental truth: that syntax is only one dimension in which people will express their individuality and inconsistency. And, in many respects, it’s the least important dimension.

Sure, all Python code has to look superficially pretty much the same, but that just shifts the essential complexity of the code to some other dimension. Typically it will manifest elsewhere, in an inconsistent variable or method naming scheme, or in the use of hard-to-comprehend data structures, or in subtle non-standard algorithms, or in weird library APIs, or in overlaying a functional or procedural style on the intrinsic Python OO worldview.

The bottom line is that bad programmers will program unreadably in *any* language. Perl just lets them do so at the simplest and most easily overcome level: syntactically.

Greg: If Perl didn’t exist which language would you be drawn to?

If Perl didn’t exist I think I would have found it necessary to invent it. Probably badly.

That’s in fact more-or-less what I was doing at the time I first discovered Perl: working on the design of a dynamic language called OOK (“Object-Oriented Kernel”). But I abandoned it as soon as I realized that Perl already provided all of the power, convenience, and flexibility that I had been striving for in my own design.

If I weren’t allowed to create my own replacement for a non-existent Perl (and assuming there was consequently no Ruby either), I suspect I would have been forced to choose Python. Though I’m not at all sure I would have been a force for good in that community.

Or perhaps I’d have gravitated towards something like Lua or Sather. Both of them had huge potential, that hasn’t quite been translated into popularity or market share. There might have been real opportunities to contribute to their communities.

Greg: What non-computing books would you recommend programmers to read?

Programming is an intrinsically creative task, so it’s critically important to feed your creativity from outside the discipline. My own interest has always been in new models and metaphors for computation and better ideas for interfaces, so I try and read as widely as I can in the hard sciences (especially physics and mathematics) and in the literature of general design. But that’s me, and most people wouldn’t find inspiration in those same places.

So the general answer, I think is that you need to find books that stretch your brain in unexpected ways, that break you out of your habitual ways of thinking and of viewing the world, that challenge your assumptions and your certainties. Some great example of such books are “The Design of Everyday Things” by Donald Norman, “Freakonomics” by Stephen Dubner and Steven Levitt, “Guns, Germs, and Steel” by Jared Diamond, “The Prince” by Machiavelli, “Catch Me If You Can”, by Frank Abignale, “Lost in the Cosmos” by Walker Percy, or just about anything that Douglas Hofstader has written (sadly, most people seem to stop at “Godel, Escher, Bach”).

Now, I don’t say that I agree with every idea or theory in those particular books, or even with most of them, but I do think that every one of them issues a direct challenge to our entrenched expectations and beliefs. And I think that’s the critical thing.

So much of everyday programming is monotone. You need to transcend that sameness if you want to become a better programmer. And learning to think outside the box (and even just that there’s a box to think outside of!) is essential to that growth.

I think that’s also why so many programmers naturally gravitate to science fiction. Really good SF takes you outside your assumptions in exactly the same way.

Dave: In Lisbon this year you branched out into another kind of training – personal training in the gym. Which type of training do you enjoy more?

I’m a teacher first. It doesn’t matter whether I’m teaching computer science or Perl or Vim or presentation skills or weight training or martial arts or anything else: I just love to teach, love to watch people discover new knowledge in the world and new capacities in themselves.

That said, I’ve been a gym rat longer than I’ve been a hacker, so there’s a certain “first love” aspect to helping people improve their lifting skills and, more importantly, to develop the confidence and mental discipline needed to achieve their physical goals.

Greg: In the spirit of desert island disks, we are sending you off to an island where you can only program in C, but we will allow you take one luxury language feature from Perl, what would it be?

Definitely memory management and garbage collection. I actually quite enjoy the hands-on nature of programming in C, except for the tedious necessity to malloc, realloc, and free my storage … especially for strings.

Josette: Most important, will you revise your old books and write new ones, if so when can we expect something new?

I’m just starting work on a new book…on Perl 6. I hope it will appear some time next year.

As far as second editions go, I’d certainly like to revisit “Perl Best Practices”. I’ve learnt so much more myself about good programming in the past five years. The community’s notion of “best” – and its available tools – have also developed considerably in that time. But, if I were to look at a second edition, it would certainly be a few years further down the track. There are just so many other projects and so few available tuits.

Josette: On August 11th you agreed to give a tutorial on ‘Understanding Regular Expressions’ in London. Can you tell us who would benefit attending such a tutorial? How will it enhance their career?

Regular expressions are a tool that just about every programmer should master, and certainly that every Perl programmer should. Regexes, especially Perl’s, are just too useful, powerful, and efficient to ignore if you have to deal with any kind of text manipulation task.

Yet Perl’s regex mechanism is so powerful that it very often bites the hand that’s trying to employ it. Many developers are actually scared of using regexes and lack any confidence that they can do so correctly or effectively.

I lay most of the blame with the way that regexes are traditionally taught or, rather, with the fact that regexes are often not taught at all…you’re just expected to pick them up as you go along.

That means that most people have a very poor grasp of what regexes actually are and no idea how to design them. The very idea of designing a regex (as you might design a program) seems odd to most people.

So that’s the focus of this forthcoming class: taking current (ab)users of regular expressions back to the fundamentals of what regexes are, how they work, and how they can be designed and optimized (rather than just cloned from existing applications and semi-randomly adapted to new tasks).

Who would benefit from that greater understanding of regexes? Every Perl programmer, because regexes are so integral to getting work done efficiently in Perl. Especially those who have to deal with, recognize, or manipulate any kind of textual or string-based data.

How will it help them? By arming them with a deep understanding of this powerful tool. By reducing their uncertainty and trepidation in deploying or maintaining regexes within their existing code. By increasing their repertoire of programming skills and techniques. In other words, by helping them to become better, more rounded, and more competent programmers.

Josette: I understand that you are travelling between April and August – As the Perl community would love to meet/listen to you, can we publish your itinerary?

Of course. It’s always at http://damian.conway.org/Events/