Why is it not Engineering?

So why is Software Engineering still an art rather than science? Let me try to address this issue based on my experience and the knowledge, I don’t claim to be an authority. In my opinion, this is due to multiple factors.

1. In the early days of computers and software, the landscape was full of proprietary OS and DBMS systems. So a single common measure of productivity across platforms was not very easily available. So if a system was developed on CICS/VSAM platform or VMS/Rdb the productivity and effort numbers were quite different although the functionality remained the same. Very often the decision was based on the cost of the hardware and the reliability that it offered. Therefore IBM was miles ahead of the competition. Although their software stack was very complex to use and maintain, they had a rock solid hardware that could handle loads and could guarantee reliable up time.
2. The improvements in the hardware and software technologies have been at a very rapid pace. So the target solution and the benchmarks had to be continuously assessed on newer unknown platforms. There was always a ramp time to learn and assimilate the newer tools. The vendors claimed productivity gains during this time but largely these were not proven in the market. Hence the practice of estimating an effort on one technology and then extrapolating that with assumptions to newer technologies came into existence and with it also came in the ‘factor of safety’ added by the developers to protect their estimates.
3. Tools which increased the productivity such as the CASE tools started getting into mainstream development projects and that too impacted the productivity numbers. The evolution was from the flowcharts to DFDs and HIPOs through the ER diagrams. As the relational model of the database became the standard, the tools also started converging and the design artifacts started getting standarized. Again the landscape was changing rapidly on this front too.
4. The estimate, even today, is very often a fitment exercise to arrive at a number that either meets the budget number or a deadline or some similar objective. It is often padded up and down several times to account for the unknowns but as the negotiations start these buffers soon start flying out of the window.
5. Customer requirements are not very clear and precise, or there are no tools to capture them in a very clearly agreed and understood manner. The answer to whether a piece of code meets a requirement is never a Boolean yes or no. This coupled with the large lifecycles of the projects leads to a situation where the stakeholders change and the requirements keep changing. Project Management practices have evolved over time to manage this scope creep to an extent by agreeing to a set of deliverables and stalling the new requirements till then.
6. Finally, I think the technology community is obsessed with working on the “bleeding edge” tools and technologies and therefore they influence the decisions to go for the latest when it comes to deciding which platform to use. There often is not enough information available to estimate the productivity numbers nor do they have sufficient data from the field to check their estimates.

If one were to draw a parallel with the engineering world, say Civil engineering. There is plenty of data available from past projects in terms of material strength, labor productivity, curing times, material requirements and the works. This data is standardized and is available in the form of industry design handbooks. So for an engineer to estimate the time and cost required to construct a new bridge, it is almost a mathematical process that leads to an estimate of material, labor and cost. The type and size of material and its strength and price is available readily. I am trivializing this to a large degree but the point is that similar reliable standardized data (metrics) is not available in Software industry. This lack of metrics is possibly the single largest factor that makes the process of Software Engineering very person/organization dependent and therefore an art form.