There's plenty of hard topics in building software, but if you asked me what's the most challenging one, I'd answer without doubt: scaling (engineering) organizations.
Why so?
- first of all: economies of scale do not work for building software (I've written an article about that few years ago, if you're interested in more details)
- the role of contextual knowledge is more important than in other industries (people/roles are not so easily replaceable), so learning curve for new people is steeper & adaptation takes them more time
- organizations can grow much faster than the sharpest & most successful people within these orgs - in other words: orgs can easily outgrow their top-, mid-level leadership; what is more, there's a common saying about people growing up to the limits of their competence - they will inevitably hit this wall, but it's rarely apparent immediately
- software building orgs with an engineering headcount of 20, 100 and 1000 are diametrically different - they need different roles, responsibilities, processes, communication routines; what's more - they need completely different profiles of people, with radically distinct skills (more on that here)
- the faster you scale, the less time you have to apply learning & fix mistakes at the early stage - hence, scaling generates all sorts of debt (tech, process, model, ...); practically speaking, the bigger the org grows, the more it harms its "unit efficiency" - in more clear terms: efficiency DOES NOT grow linearly with the headcount! Org has to grow faster & faster to maintain any reasonable benefits of growing
- the one of organizational dimensions that deserves to be named explicitly is communication: the number of communication paths can grow exponentially, so does the number of inter-dependencies; keeping this in check requires very mature & well-synced approach to simultaneous scaling of organization, product, architecture & processes
What it all means is that your journey to having 1000 engineers on board will require you to pretty much REBUILD the organisation iteratively - at least few times - on the way. Of course this process is much more turbulent & wild if you scale quickly (or even blitzscale) - e.g. by doubling the headcount every 6 months. This of course means a massive risk to your carefully cultivated culture ...
So why bother? (when it's so risky & potentially inefficient)
Well, even if there's no such thing as "safe scaling", some companies do not have any other option - modern, Internet-facing markets rarely allow compromises - the fastest & most determined pretender gets it all , there's no consolation prize for runners-up. But that's not the only reason ...
- ambitious leaders lust for more & more causative power to tackle even the greater challenges - and in some cases (even if it sounds crazy ...) it's faster just to grow inefficiently than to optimise by fixing structural/motivational/technical problems (at least in a short-term perspective)
- people need space for growth - yesterday's regulars are today's seniors & tomorrow's leaders; providing sufficient room for growth in a successful organisation can be a challenge itself
Interestingly, there's no prescription for scaling. No framework. No "Ten Commandments". It's almost like some sort of art where you make your balance-preserving decisions based on highly-contextual awareness, deep understanding of the work environment (as a system - according to the System's Theory) and built up clarity of where & when you want to make your next move.
