There's plenty of hard topics in building software, but if you asked me what's the most challenging one, I'd answer without doubt: scaling (engineering) organizations.

Why so?

  1. first of all: economies of scale do not work for building software (I've written an article about that few years ago, if you're interested in more details)
  2. the role of contextual knowledge is more important than in other industries (people/roles are not so easily replaceable), so learning curve for new people is steeper & adaptation takes them more time
  3. organizations can grow much faster than the sharpest & most successful people within these orgs - in other words: orgs can easily outgrow their top-, mid-level leadership; what is more, there's a common saying about people growing up to the limits of their competence - they will inevitably hit this wall, but it's rarely apparent immediately
  4. software building orgs with an engineering headcount of 20, 100 and 1000 are diametrically different - they need different roles, responsibilities, processes, communication routines; what's more - they need completely different profiles of people, with radically distinct skills (more on that here)
  5. the faster you scale, the less time you have to apply learning & fix mistakes at the early stage - hence, scaling generates all sorts of debt (tech, process, model, ...); practically speaking, the bigger the org grows, the more it harms its "unit efficiency" - in more clear terms: efficiency DOES NOT grow linearly with the headcount! Org has to grow faster & faster to maintain any reasonable benefits of growing
  6. the one of organizational dimensions that deserves to be named explicitly is communication: the number of communication paths can grow exponentially, so does the number of inter-dependencies; keeping this in check requires very mature & well-synced approach to simultaneous scaling of organization, product, architecture & processes

What it all means is that your journey to having 1000 engineers on board will require you to pretty much REBUILD the organisation iteratively - at least few times - on the way. Of course this process is much more turbulent & wild if you scale quickly (or even blitzscale) - e.g. by doubling the headcount every 6 months. This of course means a massive risk to your carefully cultivated culture ...

So why bother? (when it's so risky & potentially inefficient)

Well, even if there's no such thing as "safe scaling", some companies do not have any other option - modern, Internet-facing markets rarely allow compromises - the fastest & most determined pretender gets it all , there's no consolation prize for runners-up. But that's not the only reason ...

  • ambitious leaders lust for more & more causative power to tackle even the greater challenges - and in some cases (even if it sounds crazy ...) it's faster just to grow inefficiently than to optimise by fixing structural/motivational/technical problems (at least in a short-term perspective)
  • people need space for growth - yesterday's regulars are today's seniors & tomorrow's leaders; providing sufficient room for growth in a successful organisation can be a challenge itself

Interestingly, there's no prescription for scaling. No framework. No "Ten Commandments". It's almost like some sort of art where you make your balance-preserving decisions based on highly-contextual awareness, deep understanding of the work environment (as a system - according to the System's Theory) and built up clarity of where & when you want to make your next move.