Thursday, 8 December 2016

What is architecture?

On a regular basis we hear people talk of good and bad architecture, but what is architecture?

Before I describe software architecture, let's see if we can come to an agreement of what architecture is.  What are the components of architecture, and what value does architecture have.


Architecture provides the structural and connective framework required for a system of components to function.  Architecture is specific to a context, good architecture for a software system is different from that of a car or a building.

Architecture in general is not visible, it is present in the system but under other visible design components.  Before a discussion of software architecture, it would be best to describe architecture using physical objects.

Architectural Elements

Architecture involves two elements:
  1. Structure
  2. connective elements.  
Let's look at these two elements with respect to a building.

Structure for a building involves the foundation and the pieces that provide support to the entire building; it is the skeleton of the building.  If a building has an interesting shape it is because underneath the framework of rebar and concrete support the shape.

Connective elements can be structural, but they provide a way of linking different structural components for the purposes of transporting something.  Connective elements in a building transport air, water, and electricity.

The structural capability of the framework will dictate how high a building can go; generally speaking architecture determines size.  If the building has a framework of rebar and concrete that will support a 10 story structure, it will be difficult to add additional floors over 10 easily. Adding additional floors will require effort expended to reinforce the existing structural strength of the building


If a connective element is missing then adding it will be expensive.  For example, old brick buildings often didn't plan for plumbing or electricity.  If this element is added afterwards, then it will be much more expensive to put in.

If plumbing is added afterwards then you will see the plumbing running outside the walls. This can lead to problems if the building experiences sub-zero weather. 

It is similarly inconvenient to add electricity or air-conditioning to a building which has not had these connective elements designed into the building when it was built.

For comparison purposes, let's look at architecture in a couple of  contexts:
Object Structural component Connective elements
Building
  • Steel frame
  • Concrete
  • Bricks
  • Plumbing
  • Electrical system
  • HVAC system
  • Elevator
  • Emergency stairwell 
Car
  • Chasis
  • Tires
  • Drive train
  • Steering column
  • Gas conduits
  • Electrical system
  • HVAC system

Architecture and Visibility

In the two examples above, several things about architecture stand out:
  1. Structural components are generally hidden unless they are functional
  2. Connective elements are covered up in the final object
So for a building the structural element of the steel frame is invisible hidden under finishing elements. But there are cases where we see concrete or brick walls exposed.  Structural elements are generally not attractive and so we put in extra effort (i.e. cost) to hide the structural elements. Sometimes in the case of concrete or brick walls we will leave them exposed because the visual need to hide these elements are not there, i.e. a warehouse.

Connective elements are almost always hidden.  The sight of electrical wires, plumbing, or HVAC tubes is not aesthetically pleasing and we generally hide these elements.  If we are hiding a connective element, they are cheapest to put in when an object is being created the first time.

Cost of Fixing Hidden Connective Elements

Repairing a hidden connective element is expensive.  For example, fixing plumbing and electrical wires in a house are expensive depending on how hard it is to access the connective element.

Adding a connective element after the fact is much more expensive and much less attractive.  There are brick buildings that were built prior to indoor plumbing and electricity being available.  A good example of these buildings are the residences at Harvard or old brick warehouses that are now office space.  In the case of the Harvard residences the plumbing runs outside the residences and due to the winter weather in Boston, is subject to freezing.  Adding electricity to a brick warehouse involves running metal conduits inside the walls; since they are exposed, they are subject to water accidents.

Visible Structural Elements

In the case of a car the tires are a structural element, but they are exposed to view.  This is why effort goes into making the tires as attractive as possible, i.e. white wall tires, decorative hub caps, etc.

Purpose of Architecture

Once the architecture is set it determines two things:
  1. The size of an object
  2. The functional capabilities of the object
As previously mentioned, the structural architecture of a building will dictate its maximum size and the connective elements will outline its capabilities.  Connective elements are always about functional capabilities.

The purpose of structural architecture is to partition an object into sub-components that are independent and can be designed separately.  For example, in a building, the structural architecture allows you to subsequently design each of the apartments separately without worrying about how the design of one room affects another.

For a car, the main structural element of the chassis allows you to design the following sub-components separately:
  • engine
  • doors
  • lights
  • seats
By allowing sub-components to be designed separately we subdivide a problem (i.e. divide-and-conquer), which reduces the complexity of overall design.  It allows separate teams to work on the sub-components.  Good architecture facilitates strong polymorphism in the sub-components.

In an apartment, each apartment can be designed differently and by different people.  In a car, the engine can be designed by one group of people different from that designing the doors, lights, or other part of the car.

The connective components in each object either: 1) provides a shared service that is accessible to multiple components, or 2) provides a coordination of sub-components to achieve a higher level function.

Electricity, plumbing, and HVAC are all examples of shared services that are available to an entire building.  The usage of electricity in one room does not dictate the usage of electricity in another room, however, aggregate usage of electricity is sized by the architecture.

Elevators for a building and steering columns and the gas system of a car are examples of components that are coordinated across multiple levels of the object.  They provide an overall functionality when all of the sub-components coordinated are functioning.

In general, if one of the sub-components is not functioning then the entire set of coordinating objects will fail to function.  For a car, if any component in the diagram fails then the engine itself will fail.

Summary

In any object composed of multiple sub-objects there is architecture.  Architecture provides two different but related functions that is:
  1. Structural components
  2. Connective elements
The architecture of an object dictates how large it can be and determines how separately each sub-component can be designed.

This article focused on physical objects, the next article will focus on software architecture.

Friday, 7 October 2016

Why Outsourcing Fails


Every year, we see corporations outsource operations only to pull back back some, if not all, of the operations. When this happens, the cost of pulling back operations or splitting things over multiple countries can leave you with significant operational issues and similar or worse costs.

Successful corporations understand the need to control costs in operations. The "spare no expense" philosophy of John Hammond in Jurassic Park is a one way ticket to disaster.

But, a focus on cost reduction is what leads corporations to face-fault when they outsource.

Corporations are seduced by the idea of potentially cutting costs dramatically; why pay expensive resources to do repetitive things when overseas people can do the same work for pennies?

The cardinal rule when outsourcing is:

Only operations that you completely understand and control can be outsourced 

To understand operations is to understand the costs of exceptions as well as how they are processed and escalated. Local resources deal efficiently with exceptions because they have informal relationships in the organization and use them to get things done efficiently; those relationships are not available to outsourced resources.

When operations gets outsourced, not only do foreign employees not understand how to deal with exceptions but also:

  • They are often in another time zone 
  • They don't have full access to local resources 
  • They don't know who to contact inside the corporation 
  • They don't understand local business rules 
  • The network configuration can make it impossible to communicate properly 

Understanding an operational function means that you capture all details, including exceptions, in your information systems. Often, legacy information systems capture exception information as unstructured fields (i.e. memo fields) and off shore resources will not understand this.

99% of your operations might be straight forward, but the 1% of exceptions can at best lead to longer implementation times.  Best case scenario is that you will look bad to your customers and at worst, it will kill your bottom line.

If your information systems contain exception information in a structured way then at least you can eventually train foreign resources to resolve them. But unless exception information is captured so you can control it (i.e. not in general text fields) then it is the same thing as not having it.

When outsourced resources are under strict performance guidelines, they will do anything to clear their work queues. This often involves kicking back work they perceive as an exception as improperly specified; they will claim 'garbage in, garbage out'.

This will result in miscommunications and longer lead times to implement your services. Not only will delivery times, and errors go up, and your customers will get mad.

By the time you have fired the local employees, all the knowledge of the function has left the building and you are left with a broken system.

Key questions to get answers to:

  • What does each exception in operations cost? 
  • How often do they happen? 
  • Can the information systems record exceptions in a way that you can track and control them? 
  • Are the mechanisms that local resources use to resolve exceptions be used by outsourced resources? 

If you don't have a clear answer to each of these questions then outsourcing is likely to be a disaster. When you have a clear answer to the 3 questions and understand how you will address each exception then you will have much more success outsourcing.

Wednesday, 10 August 2016

Project failure is due to bad requirements


Projects can fail for a number of reasons, but at the root of most failures is a failure to gather correct and consistent requirements.  We've all laughed at some variant of the above diagram, but these issues are all because:
  • We fail to capture correct and consistent requirements
  • We play "broken telephone" when we are communicating requirements
Let's take a concrete example.  Suppose we have an orienteering challenge where you need to go from the start point below to the finish point, this is the actual requirement.  


But, there is a gap between what the actual requirements are and what is actually written down.

Good Requirements

As long as the written requirements don't diverge too much from the actual requirements, you will have time to adjust the requirements during project execution and still get to the finish point.  

So as long as the initial written requirements are in the green zone, you can still complete the project on time because the requirements point you in the correct direction.



Mediocre Requirements

Now suppose one of the following happen:
  • You don't have all the core requirements because insufficient people were consulted
  • You have conflicting requirements from different sources
When you have less than accurate requirements you will be in trouble.  The initial code and architecture of the project will be created based on the quality of the requirements.  If those requirements are suspect, then there will be rework as code needs to be adjusted as the scope of the project seems to shift.

So now you will find yourself in one of the yellow zones below

Even with the best execution, the project will be challenged.  Deadlines will be missed as you attempt to bring the requirements back to what they need to be.  This is like trying to change a tire on a car and discovering the jack is missing.

It is important to realize that adjusting poor requirements is not Scope Creep.  Fixing incorrect and inconsistent requirements is necessary and it is pointless for a project manager or executive to disallow these changes.  It would be worthwhile for project postmortems if the project manager tracked whether a requirement was missing or inconsistent.

Challenged projects are typically declared as successes, but only after massive compromises, burned out resources, damage to reputations, and loss of revenue.  When all the damage is taken into account, this is hardly a victory.


Bad Requirements

The last situation occurs where you only have vague requirements before you start a project.  This situation happens where the executives need a project done quickly and over-estimate the teams familiarity with the subject domain.

These projects start with requirements in the red zone below.  You don't have a prayer of completing this project and it will turn into a Death March with all of its characteristics (see Death March Calculus).


Making core course corrections to bad requirements are like trying to change a tire on a car when you are going down the highway at 100 mph.  It won't happen.

Conclusion

It is human nature to assume that the sooner you start a project that the sooner you will finish.   That assumption is only correct if you have good requirements to point you in the correct direction.  Good requirements are consistent and correct and include at a minimum the core requirements for the primary users.

Starting a project with mediocre or poor requirements is simply a recipe for project failure.  Mediocre or poor requirements are incomplete and inconsistent.

If you have been part of a project failure then you will discover that despite the other factors that went wrong, requirements failure was at the root of it.

Monday, 8 August 2016

Specifically, what is complexity?

People involved in software projects would say that software development is about understanding complexity.

What is complexity?

What can we do about it?

Complexity is easy to define at a high level, but people get vague when pressed for a precise definition.  Let's see if we can:

  • quantify the problem
  • define it
  • suggest how to address it

2 Brooks, Frederick Phillips. “The Mythic Man Month”, 1995. ISBN 0-201-00650-2

Wednesday, 20 July 2016

Hidden Assumption of Agile

Agile cures common problems that we experience in software development, however, there are limitations to Agile.  It may seem like a silver bullet, but there are circumstances under which Agile is not the best choice for development, or at a minimum, not to develop the entire project.

The problems with the waterfall model are well known and understood by most by now.  The waterfall methodology at a high level appeals to executives because it provides a clean way of thinking about a problem.

After all:
  • you can't design something before you have the requirements
  • you can't code something before you have the design
  • you can't test something before you do the coding
These facts are true in Agile as well.  The only difference is that you don't try to do all of any phase as a big bang.  We understand that unless we use some kind of iterative process we can't return to a previous phase and fix anything.  So the Agile process looks as follows:

That is, instead of doing all of one phase as a big bang, we do a bit at a time.  Some methodologies call the code that results from each sprint as a 'brick', and through building bricks, we eventually complete the wall.  So Agile is essentially slicing up the waterfall methodology, but all of the work that needs to be done is still done.
So why on earth would we use a waterfall process for ANYTHING?

Well one of the hidden assumptions of Agile is that you can build all software projects one brick at a time.  But, when was the last time that you saw contractors show up and just start building a skyscraper without a plan?

I mean, isn't it just a matter of building the skyscraper one layer at a time, why bother with planning?

Even though a skyscraper will be built one floor at a time from the bottom up, there still needs to be some architectural planning that happens first.  The materials in the support structure of the building together with the common components that provide common services to each floor (plumbing, electricity, air conditioning) need to be planned out in advance. 


If insufficient planning is put into the architecture, then insufficient plumbing is put in place to drive water to the top of the building, insufficient breakers are purchased to support the electrical needs, insufficiently strong materials might be put in place to hold the weight of the building.

To some degree software is malleable and you can add things after the fact.  But there is often a cost to add things after the fact which is much higher than if the element was designed up front.  For example, you could add add a parking garage to the Empire State building, but it would be cost prohibitive. If they had wanted a garage, it should have been built when the building was constructed.

So any project that will require serious architecture should pause and plan for the structural and connective elements that will be required by the project.  The requirements that are tied to the architecture should be developed first so that the architecture of the project is developed first.  Of course, the architecture can be developed using Agile or the traditional waterfall approach.

Once the architecture is in place, then the rest of development can be done using Agile knowing that all the structures and connective elements are in place.  This is akin to building the skyscraper one floor at a time once the architecture is planned.

The alternative to building out the architecture first is that you find yourself in the position of having architected a building of 10 floors and then realize that you need to have 30 floors. The initial architecture which came together quickly will be snowed under by the work arounds that you need to as you get above 10 floors.  You might get to 15 or 20 floors, but you will never finish the project.

Things that would be expensive to add after you have built a few floors:

  • Needing an extra elevator
  • Needing an extra floor in the parking garage
  • Needing the entire building cabled with fiber optics
Most projects do not require advanced architectures, just like most 2 story houses don't need to have architects.  But if your project is large and has serious architectural considerations then you are best suited to plan for the architecture up front.

Make sure to plan the architecture before doing Agile development

Friday, 8 July 2016

User stories or use cases?

Alistair Cockburn says "A user story is to a use case as a gazelle is to a gazebo".

If you are extremely familiar with both techniques then this makes sense.  If you are not familiar with both then suffice it to say that you can put a gazelle in a gazebo but not vice versa.

A user story is of the form As a <type of user>, I want <some goal> so that <some reason> (see here).  

A use case for withdraw money has many more components to it, and a skeleton of such a use case can be found here.  I develop this skeleton use case in a four part tutorial:
  • Part 1: Use case as a dialog
  • Part 2: What is an actor?
  • Part 3: Adding interface details
  • Part 4: Adding application context
The above user story is only a part of the entire use case.  For an ATM, a user story might be As a user, I want to withdraw money so that I can have physical cash.  This user story is only the summary of the withdraw money use case without the details.

User stories come from the Extreme Programming methodology. The assumption was that there will be a high degree of interaction between the developers, and the end customer and that QA will largely be done through test driven development.  It is important to realize that Extreme Programming does not scale.

Once your development team gets large, i.e. you have 3+ agile teams and your code base gets large enough to warrant a formal testing environment, then you will outgrow user stories as your only method of capturing requirements.

You can still use user stories for new modules developed with an end customer to take advantage of the light weight and rapid nature of user stories, but at some point those user stories should transition to use cases.

Unfortunately, there are quite a few ways to build use cases and not all of them are effective.  Alistair Cockburn's method (found here) is an excellent way to capture use cases for more sophisticated systems.  There is no doubt that use cases are heavier weight than user stories, but consider this:
  • Use cases can more easily be turned into test cases by QA
  • With use cases you more easily prove that you have all the requirements
  • Use cases annotated with screens and reports can be used to collaborate with remote off site customers
  • Well written use cases can easily be broken up into a sprint back log
  • Use cases will work if you are not using Agile development methods
So don't forgo the speed and dynamic nature of user stories, just recognize that there are limits to user stories and that you will need to transition to use cases when your project team or application grows.

Friday, 17 June 2016

Ready, Fire, Aim: How most gather requirements

Connecting with your customers and delivering value depends on understanding your customer's requirements and selling the correct product or solution that solves your customer's problems.

Commonly, the requirements gathering process is done hastily or not at all in the rush to get the sale. After all, the faster you can make the sales, the faster the money is in the bank -- correct?

The more that product customization is required, the more it is important understand the customer's actual problem.  Long lead times to build and implement a solution means will not only lead to difficult implementations but also to a more disappointed customer and a loss of future revenue.

If you build software, which has long lead times, it is even more important to make sure that what you are building will satisfy the customer's requirements as changing the final software solution will be nearly impossible and very costly.

Why do we continually misunderstand and sell the wrong solutions and building the wrong software?

Human Nature

Time pressures to make a sale put us under pressure and this stress leads to making quick decisions about whether a product or solution can be sold to a customer.  We listen to the customer but interpret everything he says according to the products and solutions that we have.

Often the customer will use words that seem to match exactly the products we have.  But then after the sale once we have to implement we often find that either we deliver a poor solution to the customer or a solution is infeasible and we must refund the customers money.

However, behind every word that the customer is using there is an implied usage, and understanding that implied usage is where we fail to gather requirements. For example, suppose the customer says I need a car.

Suppose that you sell used cars:
  • the customer asked for a car
  • you sell cars
Ergo problem solved!
  • What if the customer needs an SUV but you don't have any?
  • What if the customer really needs a truck?
  • What if the customer needs a car with many modifications?
  • What if the car the customer needs has never been built?
We hear the word car and we think that we know what the customer means.  The order-taker sales person will spring into action and sell what he thinks the customer needs. Behind the word car is an implied usage and unless you can ferret out the meaning that the customer has in mind, you are unlikely to sell the correct solution.

If you don't understand how the customer will to use your solution then you don't understand the problem.

Comedy of Errors

For products that require customization, the sale will get transferred to professional services that will dig deeper into the customer's requirements.  At this point you discover that the needs of the customer cannot be met.  This leads to sales people putting pressure on professional services and product management to find a solution, after all, losing the sale is not an option.

Sometimes heroic actions by the product management, professional services, and software development teams lead to a successful implementation, but usually not until there has been severe pain at the customer and midnight oil burned in your company.  You can eventually be successful but that customer will never buy from you again.

Consequences

As WIlliam Ralph Inge said, There are no rewards or punishments -- only consequences. The consequence of selling the wrong solution to a customer is:
  • Whether you lose the sale or not, the customer loses faith in you
  • The sales person is perceived as incompetent in the rest of the organization especially by professional services and software devleopment
  • Your cost of sales and implementation is much higher than expected
  • Your reputation is damaged

Conclusion

Selling the correct solution to the customer requires that you understand the customer's problem before you sell the solution.

When customization is required, good sales people engage resources that can capture the customer's requirements accurately and assess that you can deliver a solution to the customer.

Slowing down to understand the customer requirements and how you will solve his problems is the key.  By understanding the customer's requirements and producing the correct solutions you become a trusted adviser to the customer.

See also

  • Shift Happens, effect of scope shift in a software development project.