What’s changed in 20 years of software engineering?

In the past two decades, technology and the choices we have as developers have moved forward very quickly.

It’s mind boggling to reflect upon the dramatic changes in software engineering at Chariot Solutions, founded in 2002, as well as in the overall industry in that time. At the beginning of the new millennium, smartphones were just a twinkle in engineers’ eyes.

What has changed in 20 years?

Consider the landscape when Chariot started. Our team was writing a lot of Java 1.4 (or earlier) code, developing in IDEs such as JBuilder, NetBeans and Eclipse, deploying to proprietary application servers (Tomcat and open source was a luxury for companies that embraced the rapidly spreading open source movement), and using APIs like Hibernate and Struts when we could get them approved.

There was lots of proprietary software out there, from databases like Microsoft SQL Server and Oracle and Sybase, to expensive application servers such as WebSphere and WebLogic, to commercial version control systems like ClearCase, PVCS and Source(un)Safe. Books and manuals were everywhere. There was no Stack Overflow. In short, you had to do a lot of reading and experimenting to level up.

Some highlights to remember:

• Java 2 Standard Edition 1.4 had been recently released
• The Jakarta Commons and Tomcat were established and quite busy
• Hibernate was new — only a year old!
• Mac OS 10.2, AKA Jaguar, had just been released
• Jira was just introduced
• Amazon sold books online; AWS was still four years away
• People still tried to create Java GUIs, but web (and eventually mobile) applications made that irrelevant
• The smartest phone available was the Handspring Treo — a PalmPilot and cell phone monstrosity!

I’ve been with Chariot for 15 of the past 20 years, and although it’s impossible to completely summarize, my time has done some reflecting about the big changes.

Project estimation and management

Most software engineering teams don’t run software development projects like they did in 2002. They don’t use Microsoft Project GANTT charts much anymore, because it’s very hard to know how long individual tasks will take up front. You need to understand the requirements well enough to estimate the effort, and that is rarely possible, so you’re constantly updating a very detailed Microsoft project plan, and it quickly diverges from reality, to the point of being useless.

Small teams can develop more useful software by repeatedly releasing small updates to tackle smaller stories. As the project chips away at the larger requirements, we reach a better understanding, and customers actually get what they need.

Today, most organizations use some sort of iterative development process with lightweight methodologies like Kanban, Agile or Scrum. Estimation is still hard, but large-scale efforts don’t generally pick one final delivery date as much anymore, since teams that develop incrementally gain feedback and understanding along the way. That is a huge improvement.

Open source is now the norm. It wasn’t always this way

Back in 2002, we still had to justify open source on our projects. We even had a series of blogs and podcasts about it, well into 2010. Open source was just gaining a  foothold in some projects, and it only began replacing proprietary servers (remember WebSphere?) and APIs (I’m looking at you, WebLogic Integration) a few years later.

In Java, the Spring Framework accelerated open source adoption beginning in 2004 (for background, check out a brief history lesson of Spring Framework with Rod Johnson and his work on Scala in 2012). These days, we don’t even blink when selecting an open source API over a commercial one. The main concern is picking the appropriate ones. (Check out this podcast featuring Joel Confino in 2010 on picking an open source project. Some things never change.) It’s important to find one that is stable and has enough support and usage that you won’t have to worry about it dying just after you begin using it.

Automate all the things (where possible!)

 The best engineering teams are multi-discipline, involving developers with a variety of skills, testers, business experts and stakeholders regularly. Most teams also automate builds, tests, application deployments and monitor platforms to scale based on load. A great talk about being a pragmatic programmer was given at our Philly ETE 2017 by Andy Hunt and it dives into the criticality of automation, testing and CI.

It’s a polyglot world

In 2002, most of the industry focused on Java and .NET.

Today, we have a proliferation of languages, frameworks and platforms, with deployment in the cloud becoming de rigueur. Old-school Virtual Machines were common, but now containers in Docker/Kubernetes isolate software into services, rather than full virtual machines, making it easy to scale parts of your platform where needed. For a great rundown on some of the options at the time — since then Lambda performance has increased and cold starts can be mitigated a bit easier — check out Chariot’s Keith Gregory on picking the right AWS computer infrastructure from 2019.

The serverless movement promises to take platforms out of the way and just allow you a place to deploy code without managing the platform itself. As an example of how we approach emerging technologies, we see value in places where it makes life easier on the developer or customer (services such as Serverless Databasing with the Aurora Data API, 2021 Serverless RDS on AWS is a great example), and with eye toward where it is most practical, not as a hammer to drive every nail (see my own Serverless, Schmerverless skeptical view of serverless for everything in my 2021 Suburban Philly DevOps meetup talk).

Oh, and interestingly, LISP isn’t dead. The language lives on, but has been reimagined by Rich Hickey as the Clojure Platform: a LISP-inspired language and platform that runs in the Java Virtual Machine (among other places). For those who want to use 50 year-old functional programming techniques that are relevant even today, check out Original Posthilly-ete-2021-clojurescript-in-the-age-of-typescript-david-nolen/" rel="noopener noreferrer">ClojureScript in the age of typescript, a Philly ETE 2021 talk by the legendary David Nolen on ClojureScript.

Mobile applications didn’t even exist in 2002

In 2002, phones were mostly phones, and if anything they might have had an address book, calendar and to-do list on them.

The iPhone, iOS and then Android blew the doors wide open when they arrived on the scene. Chariot jumped in early, with a team of developers who embraced, learned, and then ultimately consulted first with iOS, then Android, to produce applications for our customers.

Those platforms have evolved a tremendous amount over the years, switching their languages (iOS from Objective-C to Swift, Android from Java to Kotlin), undergoing many changes to their design language, tooling, SDKs and features), which has kept our mobile team on their toes for more than a decade now.

We’re keeping track of non-native mobile web application platforms such as React Native and Flutter, and have used tools like PhoneGap and Ionic where lightweight, simple user interfaces are needed to interact with specialized hardware.

The next 20 years …

This brief look back can help current developers and the businesses they serve to understand the landscape of software development in the past, present and future. Whether you are a developer or helping your developers make tech decisions, change is happening at high speed. As Steve Jobs famously said in his Stanford commencement speech, “You can’t connect the dots looking forward; you can only connect them looking backwards.”

-30-