Overview

This is the first of three blog posts about lean software development. Most material on the subject focuses on the what and the how – what to do and how to do it.

These posts will focus on the why – the thinking behind it – and to illuminate the why with practical experience and, critically, data. Lean production is about metrics as much as anything.

About the Author:
Gordon Guthrie is the CEO/CTO of hypernumbers.com,
a cash-flow positive startup in Customer Development.
Hypernumbers are a Seedcamp company.
Gordon previously wrote Erlang: The CEO’s View for GMT.

The first post will look at some background about lean production, and try and synthesise some of the various viewpoints on the subject into a coherent world view.

The second post will focus on particular technical aspects of lean and look at particular issues in the cost of producing software – with metrics.

The third post will look at business aspects of lean, and bring examples of how lean production techniques – the rigorous elimination of waste – have had practical impact on a number of business sectors. Again this post will bring data to the table.

Introduction

Over the last year or so the whole tech world has gone lean crazy. As usual, it is a mixture of hype and fashion wrapped around a steely core of good practice.

Let’s start with some data. Rob Fitzpatrick at the Startup Toolkit¹ did a survey of people who were signed up to lean mailing lists – in other words the self-proclaimed stormtroopers of lean – and the overwhelming majority of them had read none of the canonical texts about Lean Production. This is not a good thing or a bad thing: some people are signed up because its trendy and some are intrigued and want to learn – and many are both.

A side-effect of the lean scene being immature is that a lot of nonsense is being spouted about lean. These articles will be practical, aimed at people who want to learn more, and will try and help sort the wheat from the chaff.

My obsession with lean production dates back to the heady days of Web 1.0. I was Chief Technical Architect at if.com – we had 500+ developers and spent money accordingly (think drunken sailors).

It was a real eye-opener – an effectively unlimited budget – one particular day I spent more or less $1m before lunch with no budget, no studies, nothing more than “I need this, today”. For a short period we were starting 35 developers a week; and yet we had a famine amidst plenty. We were struggling to make progress despite all our ‘advantages’.

We struggled, and went live, and took 10% of the UK mortgage market in 18 months. It was a success, but it left me with clear feeling that there was something very wrong with my understanding of production.

In the rest of this article I will talk through some key issues in lean and then in subsequent articles I will try and illustrate them with practical examples from my ‘fat’ years at if.com and my famine years at hypernumbers.com.

The father of quality, W Edwards Deming once said “In God We Trust, all others must bring data” – I will bring as much data as I can reasonably share.

What Is Lean?

Let’s begin by putting in place a framework to discussion lean software development. There is an emerging business stack which is shown below:

When do we do things?	Methodology	Stephen Blank2/Eric Reis3	a statement of dependencies – if you don’t know what you are selling yet how can you sell it, etc, etc
What are we trying to achieve?	Objectives	Sean Ellis‘s4 Startup Pyramid	a statement of the conditions that, if met, let you move onto the next stage of the methodology
What are the customers doing?	Metrics	Dave McClure5	an idealised representation of the customer lifecycle that you can test against
Why are the customers doing what they’re doing?	Analytical Methods	Dave McClure	Quantitative, Qualitative, Comparative, Competitive
How do we know why the customers are doing what they are doing?	Tools and Techniques	A million different companies
Who does what?	Tasks	Organisational Structure

Lean is the methodology in this stack. A methodology is simply ‘this is how we do stuff’ – it combines formal processes and procedures with informal or cultural ones.

People say stuff like “we don’t do procedures, we just do the right thing” and other statements which play up the elite start-up persona. That’s guff – you need to know the methodology – even if that only means reading Four Steps To The Epiphany.

It’s called ‘lean’ for a reason (it is short for lean production), but lean is a relative and not an absolute term. If someone says “we have a lean start-up” they don’t. If they say “our start-up is lean relative to…” they might – provided they have an appropriate comparator and data.

It is important that you are comparing apples to apples. Consider that poster boy of the MBA circuit SouthWest Airlines. Their customer proposition was making it cheaper to fly from Des Moines to Oklahoma City than to drive – the customer proposition is that it is cheaper. But because lean production addresses production costs, the lean comparator for SouthWest Airlines is other airlines.

SouthWest Airlines is a success because its lower production costs enabled it to make more obscure airline routes cheaper and more convenient than driving whilst still remaining profitable.

Lean is not cheap, lean is cheaper – but only in the aggregate. Lean production doesn’t consist of doing everything cheaply. It mostly consists of not doing stuff and doing what you do at the most appropriate price.

Principles Of Lean

Overview

Fred Wilson had a great article⁶ on the key tasks of a CEO:

• don’t go out of business
• hold the product vision
• build and motivate a great team

The first point is the first principle of lean. Boiling Stephen Blank’s Four Steps To The Epiphany down until you can write it on a beermat:

Customer Development is spending as little time and money until you can repeatedly and reliably walk into someone’s office and say ‘I can save you X, let me show you how’

Executing customer development is ‘only’ knocking down all the unknown-unknowns until you get there. So when it comes to estimating time and money to get there – you are on your bloody own.

How long will it take? Unknown. How much will it cost? Unknown. What happens if we run out of money? That’ll be a known.

Don’t Do Work

The key principle of lean is don’t do work – eliminate waste. That is both a positive and negative thing.

The (easy) first half of it is slash and burn, find’n’kill, old fashioned cost squashing, negotiation, shopping around and so on.

The second half is the more complex – it is identifying activities which accumulate to cost. Taiichi Ohno of Toyota used to use a technique called the Ohno circle to do this. He would draw a chalk circle on the floor in the middle of a workshop and get people to stand in it for the whole day and watch what people actually do. In the context of a car assembly line that means walk from here to there and back, reach for this and stack that. Then they would work on reducing that cycle time at a micro-task level.

The third half is more complex again. Eliminate this task at the micro level by changing that other micro task or tasks. The problem here is that the totality of a car assembly plant production line is too complex to be held in a single head. Toyota has a complex culture. On the one hand workers, at a task level, appear to be dehumanised automatons – the tasks are choreographed like ballet. On the other hand, employees are valued and cared for and rotate through different tasks on the line. The skilled empowered knowledge worker is then encouraged to say “if we added this task here we could eliminate 5 tasks that we do later on in another station”.

Physical labour can have a performance aspect to it, in a way that typing on a computer doesn’t. When I worked offshore as a lad, I used to watch the roughnecks grabbing an 80 foot drill string hanging in the derrick with tongs, stabbing it onto the rest of the drill string, casting a chain onto it and spinning it in: a team operating almost at an autonomic level. Toyota aims to have a combination of that ‘work as performance’ combined with a critical intelligence about the whole system in as many people as possible in its workforce.

The fourth half of this process is yet more complex – eliminating micro-tasks by reconfiguring the whole overall flow of the work pattern – having understood the micro in sufficient detail it becomes possible to ‘refactor’ a complex workflow.

You’ll notice that the halves of this process are mounting up – do less stuff is a bit harder than it seems.

The fifth half is knowing when to increase costs to reduce costs. The Japanese notion that the way to lower costs is to pay higher wages spits in the face of the cruder more reductionistic end of Anglo-Saxon capitalism: which is a shame because they learnt it from Henry Ford. Likewise is increasing the training budget – or Facebook’s policy of paying people higher salaries if they live within a mile of the office: staff are more valuable if they don’t commute.

The sixth (and final) half is to embrace insanity – organise your factors of production to bring problems to the attention of management as quickly as possible – this is known in the trade as production-levelling.

Consider a car assembly line that produces 200 cars a day – you have to make 100 four doors and 100 two doors. So the ‘easy’ way is to do 100 two-doors in the morning and 100 four-doors in the afternoon. That way you can ‘optimise’ things for each production run. Well it turns out the right way is the hardest way, so let’s do them four-two-four-two. Then throw in each run should be half-red and half-black, and half with air-con and half-without, and half with walnut dashboard and half without. Yup, a black two-door with walnut and no aircon follows a red four-door without walnut but with air-con.

The point being if you can do that then you have got to have solved a gazillion problems about workstation setup time, and colour swapping time, and component ordering and restocking and loads of other wasteful stuff that you haven’t thought or identified yet.

And the interesting part of this is that in fixing the detail, and reducing costs, you uncover new, previously unviable business models. We will see practical examples of that in the third post.

The point is not to make life difficult because it makes you morally better (let’s make all developers work standing on one leg) but choosing to organise your factors of production to make problems surface as early as possible and then fixing them as soon as possible. If there is a hurricane between you and where you want to go to, then sailing straight at the hurricane is the right thing to do.

The end result of this is that when you walk into a showroom in Japan and talk about buying a car, you run through the options with the sales staff. And when you complete the order, your specific car starts on the assembly line and, as it rolls off, it is driven to your house.

Don’t Build Things People Don’t Want

The YCombinator slogan is build things people want. Lean production is the double negative of that: don’t build things people don’t want.

This also has its root in Japanese production methods – the focus is on customer-pull rather than product-push – known in the trade as kanban.

The best way to explain this is to consider a tale of 2 computers.

In the first case you go into PC World (or whatever the big computer and hardware superstore is in your neck of the woods) and browse the computers on display – a sleek tower, a chunky desktop, a slimline netbook. The manufacture has churned out a set of machines and pushed them to the shop for you to buy.

In the second case you go online to a firm like Dell and order a new machine online. As soon as your order is placed the actual machine configuration that you specified (this much memory, these size disks with this spindle speed, that specification of optical drive) is then assembled to order on the line. And as a consequence replacement parts are pulled from the supplier based on your demand.

Lean Customer Development on the internet is about moving software development from push to pull.

Instead of bundling up some new features, pushing them to live and having a version 2.0 party, you try and get customer signals by a variety of mechanisms that ‘pull’ features out of your team.

Don’t Write Code

Again ‘don’t write code’ has many sides. The first (and easy part) is to be aggressive in your use of already available software, ranging from:

• operating systems
• cloud provision
• infrastructure software
• databases
• webservers
• etc, etc
• libraries
• ajax
• authentication
• etc, etc
• software as a service
• analytics
• application/infrastructure monitoring
• OAuth signon
• etc, etc

The principle here is the old one of ‘write what gives you competitive advantage’ and ‘buy what doesn’t’.

The second part of ‘don’t write code’ is considerably harder. A key reference here is Paul Graham’s essay Beating The Averages⁷. Any bloody idiot can write code – the secret is to build a team of developers who specialise in not writing code and use all any tools available to help you not write code.

By that I mean a team with a strong architectural sense who can decompose your business problems into the smallest and cleanest set of sub-systems with the minimum numbers of edge and special cases. Aggressive use of techniques that work at higher and higher levels of abstraction – meta-programming, domain specific languages and so on and so forth – the whole notion of using the most powerful tools available to the best effect.

But critically it also means building a business team who can ruthlessly strip business processes to their bones. An example of this would be tripit.com who reduced their service to emailing your itinerary e-mails to them – they would then create an account, register you and process your stuff just from that. The sheer brilliance of that, measured in lines of unwritten code, is immense. The less work the customer has to do, the less code you have to write to support them.

There is a third and much less glamorous part is the whole question of quality. There is an old industry saw ‘there’s never time to do it right, but there’s always time to do it twice’.

Lean start-ups are positively encouraged to build up ‘technical debt’ – indeed to go heavily into technical debt – but little is written about how and when to go into technical debt, and which debt, and to whom, not to incur.

The economics of technical debt are well understood. It is about 3 times as expensive to do something properly as it is to bodge it the first time and about twice as much again to fix it after the fact.

Technical debt only makes sense in two circumstances:

• when someone else is going to pay for it
• ‘if we can achieve this we can execute a funding round that will make an order of magnitude change in our resources’
• when you intend to default on the debt

Consider if you have 10 features you want to try out with can each be bodged in for $1k (with a clean-up cost of another $5k each) or done right first time for $3k.

You do all properly and it costs you $30k and then you throw away 90% of your work.

Or you bodge all 10 in for $10k, throw away 9 of them and then spend $5k fixing up the bodged winner – you are $15k up.

If your analysis shows that nobody else will pay back your technical debt, and you can’t default on it, you are always cheaper to do it once, do it right.

(In an exploding competitive market you have to throw in market risk in the equation – but that is just a special case of someone else paying – in this case the customer base you would otherwise lose.)

Why Has Lean Emerged So Strongly?

The great transformation of the industry over the last 10 years has been the availability of already-written (and battle tested) software which has slashed the size of a team that can build a functioning product and the crash in price of hardware.

Back in Web 1.0 at if.com we had a pair of Sun E10K’s as our core machines – which rolled in a £1.5m a pair. We weren’t VC backed, we canned £60m from Halifax, but we had some doings with Paypal who were VC backed. They had an E10K as well⁸.

The entry cost to a web 1.0 firm then was about $1m in hardware plus staff to build things. The structure of the VC industry simply reflected that – there’s a simple biological metaphor. If the cost of a baby is extremely large (compared to the parent) you can’t produce many, so you gotta love and protect them a lot (hence mammalian parenting).

But entry level hardware is now buttons. We also had 500 developers at if.com in 25-odd teams. Many of those did jobs that would now be done by software libraries or SaaS products – we hand-rolled the best part of a 1,000 Soap endpoints (back when Soap was the wee boy). So the cost of a baby is now extremely low compared to the parent and we have a frogspawn ecology. The funders only care that some frogs get out of the pond and they get to kiss them to see if they turn into a princess – so most tadpoles die, so what.

Additionally, in the old days it was hard to become a VC and get the opportunity to lose a lot of other people’s money (most VC funds under perform the market) but now it is super-easy to become an angel and lose your own money, so the funding side of the game has gone crazy as well.

Interestingly because the industry has so many network effects that lead to competitive advantage we are starting to see insect-like cousin-group selection – start-ups that are loosely part of a bigger group who work together and where the members of unsuccessful start-ups can see second careers as early employees of the successful ones. The YCombinator alumni are probably the best known of these, but Dave McClure is building his own swarm and various other well known early investors are making sectoral investments.

My gut instinct is that this will formalise. There are other hit-based industries (think of 60’s pop groups out of the Brill Building, or Motown, or the way magazine publishers work). People work for the group for salary (and upside) and resources (editors, band members, backing singers) are relocated from flops to push hits. If I was running 500 Startups I would share risk and reward across my base and build a crop of ‘future CEO’s’ that I would rotate in junior roles through my teams to build experienced executives. As long as folk are heavily tied as ‘founders’ to their start-up that will be hard. Different capital structures would be needed.

Zed Shaw has identified another trend⁹ that I think will also grow over time. Many of the new crop of B2B start-ups are ‘departments of Web 1.0 startups turned into a SaaS’.

At if.com we had paid staff who did the work that is currently done by companies like:

Linode -> our data centre staff

Kiss Metrics -> our MIS teams

io: -> survey contracted market research people

Heroku -> our platform teams

Google Analytics -> our MIS teams

etc, etc

Essentially whole teams done gone and privatised themselves as RESTful services.

The beauty of selling starting-up SaaS to start-ups is that all of them will buy it – the successful and the unsuccessful ones. It is hard to think of a startup I know that won’t leave a coupla thousand dollars stuck to a YCombinator SaaS start-up or two. But starting a SaaS company now that offers a service which only mature companies can use? It would almost certainly die from lack of customers.

As SaaS grows over the years I think you will find the opportunities for SaaS companies will continue to keep growing. If I was stuck as the developer in a day job with the Department of Paperclips in Monster Corp I would be taking a hard look around me and thinking:

• does this job exist in every company?
• what would this job look like if it were running over a REST interface?
• what would truly excellent customer service look like for the Department of Paperclips?
• what are Monster Corp’s drivers in relation to this department?

The great problem with young founders is that they don’t have the experience to build products for people outside their circle of experience. The world of open source is full of text editors because the builder is also the customer. Likewise start-ups that build tools for start-ups are a no-brainer. Well a couple of years in a big corp now will provide you with critical Subject Matter Expertise for your SaaS start-up to come.

State Of Play In Lean

When the lean movement took off there was a distinct tendency to overplay lean technical techniques (agile, scrum, test-driven development and continuous deployment). This has mellowed down a bit, but the point is well made: lean technology is only useful in as much as it supports lean business processes.

Like all hot concepts there is a lifecycle of hype. When people realise it is useful they first start by re-categorising everything they all ready do as the flavour de jour. Then it blooms and blooms and starts being applied quite wildly.

An example of this would be the rise of paper prototypes as MVP’s. Paper prototypes are a great and cheap way of working out design and interface issues but they need to be tempered with capability.

I could knock up a paper prototype of an iPhone teleportation app and get great customer stats (‘96% of 14-25 year olds would def-in-et-ely pay $1,000 for this app, right here, right now!’). But without capability it is just piss and wind.

Patrick Vlaskovits and Eric Ries would define an MVP loosely as ‘enough product to test a hypothesis’ – and would insist that if it isn’t testing a hypothesis, then it isn’t an MVP.

I am all in favour of testing hypothesis by the cheapest mechanisms possible, but my gut instinct is that you probably should not call it an MVP unless it is a product – a product that you are actually trying to sell, right here, right now. An MVP, after all, is a Minimum Viable Product. If your users are not your users (but your product) you should be thinking how do I get enough users to ‘sell’ to whoever your customers are (lead generation, advertisers, market information, whatever).

One of Usability Guru Jacob Neilson’s maxims¹⁰ is:
To design an easy-to-use interface, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior (sic).

allow me to introduce Guthrie’s Corollary:
Nothing cuts through the fog of customer intentions better than a palm crossed with silver.

I’m not a reductionist who believes that all human relations can be reduced to the bare cash nexus, or that the glories of human companionship in its myriad complexities is just about getting your bones jumped, but there is a delicate point at which these central topics must be addressed – and it seems to be that the MVP is right at that nitty-gritty.

One of the reasons I think we should insist on this MVP point is because of people’s behaviour towards income.

Tolstoy once said:
Happy families are all alike; all unhappy families are unhappy in their own way.

A quick look at some demand curves is instructive here:

It seems to me that:

the freemium ends of demand curves are all equally uninteresting; the premium ends are all differently fascinating.

I think start-ups should take a hard look at exactly what they are testing when they launch free or freemium products. How many times do we really need to test the proposition “do people like free stuff?” If you think that is still an interesting question, my Nigerian friend would like to sell you some ursine scat-porn.

If your business is ‘selling users’ then free is obviously great. But if it is not, then freemium is a marketing strategy that you should explore once you have product-market fit. When you understand your fulfilment costs, and your customer acquisition costs, and the drivers that motivate your customers, then you must consider exploring freemium¹¹.

A fundamental point is that, for a particular organisation, lean technology involves Donald Rumsfeld’s useful category – the known-unknowns (people know how to do continuous deployment, it has been solved, we as a company simply lack the skills) whereas lean business development involves unknown-unknowns (will anyone ever buy this?).

The thing about known-unknowns is that they are (thankfully) reducible to a ‘cookbook’ approach – the problem with the unknown-unknowns is that they are all down to judgement, culture, organisational structure and other more opaque skills.

I suspect the glamour boom of lean is fast approaching an end and it is settling down into a mature discipline of the hard grind out. A good indicator that ‘the market in lean is at the top’ would be this discussion¹² on Hacker News right now as I write. Mr Waxman boldly poses the (rhetorical) question “The Beatles, The Original Lean Start-up?” (question mark interpolated by me). The answer is inevitably “No”. Those of you with sharp hearing should be able to discern the gentle rustle that an single eyebrow makes when it is raised archly.

Practical MVP’s And Launching

There is a certain cargo cult approach to launching a company – launch early, launch often, grow up in public, fail fast, ABL (always be launching).

The phrase ‘cargo cult’ comes from the Second World War. American troops would arrive on some obscure island. They would clear the jungle, flatten the ground, put up a small tower, and lo “big silver birds would come and debouche good things”.

So after the war, the locals would clear some jungle, flatten the ground, make a small tower and wait for the silver birds and their precious cargo, which, hélas, never came.

In the absence of a theory of launching, people simply cargo culted – confusing “achieving what someone else had achieved” with “doing what someone else does”.

Successful start-ups launched early and iterated, therefore we should launch and iterate. Note the difference – the critical adverb early. When is it early enough?

A theory of launching is emerging: the components are falling into place.

We have purpose – the purpose of an early stage start-up is to find a repeatable business model.

We have an operational methodology – customer pull, launching is the only way to get the customer to pull product from us.

What has been missing is a concise statement of customer motivation to participate in this. Paul Graham has supplied a concise summary of this in his recent essay¹³:

Usually we advise start-ups to launch when they’ve built something with a quantum of utility—when they’ve built something sufficiently better than existing options that at least some users would say “I’m glad this appeared, because now I can finally do x.” If what you’ve built is a subset of existing technology at the same price, then users have no reason to try it, which means you don’t get to start the conversation with them. You need a quantum of utility to get a toehold.

Turns out that the theory of launching is simply reciprocal gifts: “I will give you some of my time if you can give me something that I can’t do at the moment”.

So you can launch an MVP when you have something you can sell which is better enough than the alternatives to make someone think “I will give that a try”.

Note the contradiction, nay healthy tension, between customer and user in that statement.

There’s a lovely quote from Goethe (it was one of Lenin’s favourites, but don’t let that put you off):
Grau, theurer Freund, ist alle Theorie,
Und grün des Lebens goldner Baum

It translates as:
Gray, dear friend, is all theory, and green is life’s golden tree.

In the subsequent posts we will pull away from gray theory and embrace, the prickles, thorns and branches of Yggdrasil.

Footnotes