Humans have been in the business of engineering since the beginning of our existence. Software engineering hasn’t been around for very long in this context; however, our collective learning about the process has increased exponentially nevertheless. The tools and technology behind everything from source code to computing machines have evolved rapidly. One static, fundamental principle will continue to shape the tech economy and how future businesses will run. This series discusses how the Modularity Principle will force all of us to find a specialty and perfect its network API.
All systems are simply a sum of their parts. When I see someone look at a monstrous problem and become flustered, they’re usually flustered because they don’t even know where to begin. Other times, they know where to start, but they balk at the amount of time they perceive it takes to solve the problem. This balking is a valid defense mechanism that prevents us from wasting our time on things that do not matter to us.
For most matters we take on personally and view as a challenge we love, we underestimate the time it will take to solve the problem. This phenomenon is called the planning fallacy by psychologists. Daniel Kahneman and Amos Tversky introduced the concept in 1979. On the other hand, if we look at a problem as an observer and view it as someone else’s idea, we overestimate the time. Our estimates are never correct. I’m six years into a problem I thought I would’ve moved on from in eight months.
What causes us to underestimate the complex problems we want to solve? I believe it’s because we can simplify it in our minds better than those trying to save us from our own insanity. When we approach a complex problem that we are so willing to take on with superhuman energy, how do we go about simplifying it? Let’s take a typical, complex machine we are all familiar with, cars. Most people see cars as complex. A Cooper Tire study published in 2018 estimated that 25% of Americans are overwhelmed by their vehicles, and 36% are not confident that they can even fix a flat tire.
What allows a car mechanic to take on the task of fixing these complex machines with such ease? They can break this complex machine down into its simple parts because the car mechanic thoroughly understands the components that make up a car. When most of us have a “vehicle breakdown,” we see it as a broken vehicle. A complex machine that simply no longer works correctly. Our trusty mechanic does not see it as a problematic car but understands the problem as one of those simple parts.
What Is Complexity?
Everything complex, which is most things, is a sum of simpler components. All of us understand something complex better than others. When we flex our brains in these areas, others see us as superhuman smart. This understanding exists because we know the simple individual components and understand how they work together to create the complexity everyone else tends to see.
Software is a sum of simple processes. Let’s think about an authentication system. If a request is retrieving private data, then we must authenticate. If there is an authentication token in the request, then check its formatting. If the formatting is correct, then check its value. If the value matches a known identity, then pass it to the authorizing system. In all other cases, fail the request. If there are different authentication protocols used, each one will be the same process with different data structures and interfacing requirements.
In total, they will add up to a “complex” authentication system built from simple processes running based on logic gates. Although the authentication system adds up to be a complex system, this system is not helpful on its own. Authentication systems tend to be the first layer of a more extensive system sitting behind it. This composition means that our complex system is also one modular piece of an even more powerful, more complex system.
There are many organizations in the business of delivering complex software. There was a time where monolithic companies built monolithic ‘stacks’ and developed everything in-house. We’ve all looked in awe at the troubles early computer-based companies went through to deliver, what we would consider now, simple data. As Computer Science education formalized, it became clear that engineering complex systems required organized and self-contained modules that did one logical thing well.
Monolithic builds gave way to smaller modules working together through communication protocols. Nowadays, these can be interprocess messages, or they are messages sent and received over a network. Even when a system is “complete” with clear, internally built sub-systems, technology inevitably requires the addition of new modules and the need to replace existing modules. Take finance; for instance, there is no doubt we have stable, mature, and complete banking systems. Nothing is missing for these systems to service their intended purpose today as trillions of dollars in value flows through personal spending accounts, investment accounts, and business accounts.
Yet, even these financial systems face the reality that there is a new system in town that is gaining traction at a rapid pace, the cryptocurrency system. There is no debate that cryptocurrency systems bring a completely new approach and a new complexity to financial systems.
Hopefully, you can begin to see that, while we live in a world of increasing complexity, there is a framework by which we can manage this complexity. I’m looking forward to diving into the world where complex systems will take us as we walk through this series. We will see how the intersection of bleeding-edge technology, centuries-old engineering principles, and the modern economy will force us to work together globally. No one is safe because no one person or entity can compete on their own in the future. We will see how interconnectivity is inevitable. It is the best teams that operate within this paradigm that will win in the end.