Some initial thoughts to mull over for the weekend (from Fred Brooks);
The other chapters discuss why this is so and Brooks points out one of his mistakes in the first edition--"David Parnas Was Right, and I Was Wrong About Information Hiding."
Chemical engineers learned long ago that a process that works in the laboratory cannot be implemented in a factory in one step. An intermediate step called the pilot plant is necessary....In most [software] projects, the first system is barely usable. It may be too slow, too big, awkward to use, or all three. There is no alternative but to start again, smarting but smarter, and build a redesigned version in which these problems are solved.... Delivering the throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down. Hence, plan to throw one away; you will, anyhow.
My guess (I am fairly sure) that a large % of developers' Sapir-Whorp vocabulary does not contain the metaphors 'top-down', 'decomposition' and 'information hiding'. (this is part of your WHY remark.). There's a zillion ways to run a software project; not all are equal.
In part II we elaborate a bit more on loosely coupled systems and hiding nasty stuff behind black boxes (like legacy systems).This paper is the second in a series of two on the design of software systems in computational finance. Both papers are concerned with applications using a combinationof C++, C#, and C++/CLI in the Microsoft .NET Framework. We createa multilanguage application to value an equity-linked product. The actual pricingof this product will be performed by the QuantLib Monte Carlo framework. Weuse C++/CLI code as a wrapper class for native C++ code. Then, we use C# as afront end to the C++/CLI wrapper, after having constructed transaction-relatedparameters and market data. For flexible input data construction, a specificfactory mechanism will be implemented using Assembly, Reflection API, anddynamic data types. Finally, we interface C# client code to Excel. The source codefor the application in this paper can be requested by contacting the authors. Sincethis paper discusses how we implemented the application, we feel that compilingand running the corresponding code will be an aid in understanding the designrationale.Keywordsnative C++, C#, C++/CLI, interoperable and reusable code, multilanguage applications,factory design, equity-linked products, Excel-DNA//
'Software Engineering' is more of an art than science.