Sunday, July 19, 2009

Is Software Engineering Really Dead?

I recently read a very interesting post on a well trafficked software blog claiming that Software Engineering is Dead. After having just spent 4 semesters at a large university to earn a master of software engineering, I was scared. Did I just waste my time and money? Could I have just continued to hack on code to get it to work like I did prior to my educational venture? Knowing first hand that I'm a better engineer for the experience, I can say that this isn't true.

Following is the comment that I posted to the blog:
What scares me about this blog post is that it's sending the message to budding computer scientists and software developers that they don't need to pay attention to software engineering practices. In the absence of any software engineering practices, what is the next best alternative? Glorified hacking with venerated gurus working in dark rooms churning out beautiful code when they're ready to grace the world with their creations?

I think it might be more accurate to say that "CMMI Level 5 project management is inappropriate for developing blogging platforms" instead of the general statement, "software engineering is dead." Several other comments have addressed this, that rigorous software engineering is appropriate for different kinds of projects like compiler development, so I won't belabor that point.

A true software engineer has a tool belt of best practices, techniques, and approaches that can be applied to different types of software projects. Are you working on aileron control software for Boeing? Then you'll apply the heaviest practices possible, with complete software specifications and sign offs possible to CYA. Are you working for a start up with a vague idea, little cash, and no ROI model? Then you'll apply a lighter weight process with an iterative life cycle, rapid prototyping, with a concentration on vetting your ideas and getting to market as quickly as possible to grab market share. Working for an open source project with no revenue model? Then hack away!

Finally, are you somewhere in the middle, working for a medium size project for a dotcom, with a dozen developers, PMs, testers, designers, and business stakeholders with an estimated year over year revenue of 1 million dollars? Then you'll want: the accountability of a medium weight software specification; usability testing to validate initial business ideas; software estimation to determine milestones and deadlines for business commitments; software processes to have repeatable release practices; iterative development to verify that your estimates are correct; and automated regression testing and unit tests to reduce bugs in your delivered software.

Software engineering means different things to different developers. Many of the detractors of "software engineering" view it as unnecessary documentation in the way of getting to the fun part of coding. But you have to remember that software engineering is more than just generating paperwork. It's about knowing what SDLCs are available to the practitioner, such as RUP, Iterative, Scrum, or Waterfall. It's about knowing that estimation and setting milestones is a good thing when you have commitments that you need to adhere to (it's easier to steer a ship around an island if you know its location well in advance). Using project management software is appropriate if you need to make sure you're still on track for hitting your milestones. Requirements analysis and change control is good if you have a client who keeps trying to add features to the product that threaten to compromise your milestones (death by a thousand paper cuts). It's about knowing that commenting your code and documenting your interfaces and APIs is a good thing if someone else needs to know how to maintain your system when you're gone (see 'hit by a bus' above). It's about having repeatable successes by documenting the processes used to get the product to production. It's about using unit testing to prevent new bugs from cropping up when you make changes to your system, and verifying that old bugs stay fixed.

I don't believe that software engineering is dead; otherwise we would all give up on unit testing, setting milestones and deadlines, and writing comments in our code. Knowing that Coding Horror is an advocate of unit testing, I know this is not the case.

Part of engineering is about knowing your goals and constraints, the tools at your disposal, and how to navigate trade off space to achieve your goals. We all operate within a constraint space to do what we do best. I absolutely know that I can't let 4 developers do whatever they want for 4 months to deliver a million dollar ROI project, so I apply software engineering to achieve the business goals within this constraint space. I apply the right dose (this is where craft comes in) of software engineering practices to ensure that I can achieve the business goal, keep my career, and successfully deliver the product on time.

In this regard, software development as an engineering discipline is still very much alive and strong.
This really is a sales pitch for software engineering. It's unfortunate that software engineering is equated to the generation of reams of documentation. It's almost like saying that electrical engineering is about solving differential equations for potential energy fields to predict electron distributions. This misses the point that EE is about building computers and other electrical systems based on first principles and higher level abstractions (transistors, circuits, logic gates, ALUs, HDLs, and logic libraries). It's about designing an electrical system using first and second order principles within your constraint space (hardware, physical, budget, power, etc.)

Likewise, software engineering has a similar scent to it, where the tools and first order principles are different, but the approach of solving a problem within a constraint space is the same. If we think of software engineering as a set of tools and approaches to be used in the right situation, then it's not dead by any means.