Tools to support agile ways of working in large organizations

We are using JIRA to manage user stories and features in an agile setup. I was asked to make some tweaks to the JIRA setup to make the work more efficient.

As with many large organizations, the JIRA setup is managed by a central organization.  Their approach to their work is somewhat less than agile 🙂

 

 

Me JIRA admin
May I have a JIRA project to play around in and test some configurations?  I do not want a standard configuration as my changes will affect other projects in that case  
  What standard configuration do you want?
None, I want to be able to change configurations myself.  
  Do you want to be in test or production?
What’s the difference for me?  
  I think you should be in test
Okay  
  Here’s a project
But it has a standard configuration  
  Oops, here’s another project
Well, this project did not have a standard configuration, but a default-configuration.  But I still can’t change anything.  
  What, you want to change things?
Yes, that was sort of the whole point of this….  
  But then you need full JIRA admin access
Why full access, I only need to change in my project…  
   Then you need full JIRA admin access
Can I have JIRA access?  
  You already have JIRA access if you can log in
I mean JIRA admin access…  
  No you can’t
Can I have my own standard configuration to play with  
  No you can’t
Who manages the standard configuration that is in use here?  
  JIRA admin
Is that your team?  
  Yes
Who should I talk to  
  JIRA admin

I think I give up……

Difference between priorities and working order

I was recently in a prioritization meeting where the discussion got quite confused and it made me realize one thing…  There is a big difference between priorities and working order (I e in what order you tackle things in a project)

Priorities are about business need and how desirable a feature is.  A high priority means that the business really wants a specific feature and that they are (hopefully) willing to spend some effort and money on it.  A lower priority feature is something that the business can live without if they have to.

This has often been linked to working order, meaning that features should be worked in in priority order, from high priority to lower.  The reasoning behind this is often that if, for any reason, the project is stopped at least we have developed the highest priority features and therefore we can put those into use.

BUT (and this is where the confusion often occurs)

There could be other reasons for doing lower-priority features before higher-priority ones.

First, a higher-priority feature might actually depend on a lower-priority feature.  This means that the higher-priority feature cannot be developed or deployed before the lower-priority feature is in place.  In this case, it makes perfect sense to work on lower-priority features before higher-priority ones.

Second, resource constraints may push us to work on higher priority features before lower-priority ones.  Imagine we have 3 features, A, B and C which are prioritized in that order.  Furthermore we have 2 developers Albert and Cecilia.  Their respective competence areas mean that features A and B can only be developed by Albert whereas C can be developed by Cecilia.  In this case, it makes sense to do feature C before B, since Albert is busy developing A.

So, when we are talking about priorities, it would be beneficial for all parties to understand these complexities.  Otherwise we could end up PRIORITIZING features based on dependencies or resource constraints, which would be wrong.

Master data and local data

Lately I’ve been involved in a project for charging of banking fees at a major Swedish bank.  My role here has been architect and system analyst.  The project is about introducing a new charging solution in the system landscape as well as to sunset an old solution.

As with all large corporations, the system landscape is complex and contains lots of systems.  There is one system that deals with customer data and our system has to interface to that to get customer information.  The other system is the master for customer data across the whole business.

Now, there is a need to enrich the customer information that is available in the central system.  One example is that we need to store an invoicing address, and there are other examples as well.  The invoicing address is not available in the central system today.

So, now we have two options on how to do this:

First, we could augment the central system with the invoicing address information and second we could let our system keep the invoicing address as well as the other examples of information that is needed.

How would you do it?

There are, of course, pros and cons of either approach.  The first approach is beneficial since it makes the invoicing address easily available for other systems should they need it.  Also, it keeps the customer information together to provide a fuller picture of the customer information in one place.  The second approach, however, is more inline with the current solution and it requires much less analysis and development work.  It makes the solution for this project simpler and introduces fewer dependencies into our solution.

On a theoretical level, one could say that information that is clearly global across the business should be mastered in the global system and information that is clearly specific and local to one system should live in the local system.  Also, one needs a mechanism to link records in the global system with records in the local one.

But this case is not so clear….  Invoicing address, is that local to charging, or is it interesting from other points-of-view as well?  The answer is not so clear, more a matter of opinion….

Large products and the information available about them

Today I was in the situation where I needed to better understand a product from a large software vendor.  (Oh, by the way, I have a new assignment; I’m an architect at another major Swedish bank now).  I needed to understand what the product is, what it does and a little bit how it works.

I googled the product name and got lots of hits, the first ones were from the vendor.  Here I was overwhelmed by information about why the product is needed, how excellent it is, white papers on specific configurations people have used etc. etc. etc.  But nowhere did I find information that answered my simple question, what is the product, what does it do at its core and how does it work.

Do you, like me, find yourself in this situation a lot?  You search for information about basic questions about a product from a large company but find lots of information about details or specifics that you really don’t need.  The vendor overwhelms you with information but nowhere do they really answer what their product is and what it does.

Why is this? Do they sell more this way?  Maybe I think like an engineer, but I don’t think they sell more.  If they cannot answer these simple questions, a lot of people are turned away…

I also think that smaller vendors are much better at this.  Perhaps they have smaller products that are easier to define and document, but I also think that a lot of smaller vendors are much better at keeping at their core values of the product.  Take YouTube for example (arguably the vendor behind YouTube, which is Google is not a small vendor, but YouTube has its roots in the small-vendor category).  YouTube is fundamentally just a video sharing service.  That is very easy to define.  But they have also stayed that way for very, very long. I imagine that the technical solution behind YouTube is very complex but the set of functionality is still quite small.  As a user, you immediately understand what YouTube is.  Contrast this to something like IBM Websphere which is a huge product (in fact a family of products) that does lots of things and where the information available on the web is huge, but not really useful for most people, like in my example today.

Datensparsamkeit and quality of data

I just read Martin Fowlers blogpost on the concept of Datensparsamkeit that you can find here

My summary of his post is that he argues that organizations should only store the data they really need and not, in this day and age of big data, store all data they can get their hands on. His primary concern is that of privacy.

I fully support his concern of privacy, but reading the blogpost it got me to think about another reason for Datensparsamkeit.

In my current project we have been collecting data from various sources, internal to the organization as well as external. After all, the project is about a data warehouse and those are fundamentally about collecting data. In some situations the data was not needed right away, but for various reasons we built the interface to collect it anyway. After this, the interface was put into production and the data collection was started.

In all of the situations when we collected data that was not needed right away, we run into serious data quality issues. We had situations when the interface was broken for months, no data was collected nobody realized it. We also had a situation where a calculation to generate derived data was seriously flawed for months and nobody realized it.

Since the data was not needed right away, nobody was ensuring the quality of it and therefore the collection of the data was not only a big waste, it also led to serious rework to fix the problems.

So, in addition to the privacy concerns of Datensparsamkeit, I would also add serious data quality issues that leads to Datensparsamkeit.

Why is it so hard to change plans

Another story from my current assignment… (a bit vague, since the objective is not to point fingers at anyone, but instead to make reflections on the larger pattern of organizational behavior)

A neighboring project that is working on the same system with the same technology has developed a tool. A tool that solves a real problem that they have. Now they have some political traction so that this tool has been pushed out to everybody else, including us. But we don’t have their problem, we have another related one. Also, their tool actually makes dealing with our problem worse. We did, however, not really see this when it was decided that our project should use the tool, so the plans were made and now we have executed them.

I have really tried to make people aware of the fact that their tool does not help us, but instead makes things worse. So far, however, no one who makes the decisions has really listened to me. The tool has been pushed on us anyway. When the plans were made they were very hard to change, even if the execution of these specific plans were delayed for a significant amount of time (several months in our case).

Why are we in this situation? And why are such situations so common?

If plans are made they become very hard to change. Also if the desired change to the plan is to scrap the idea entirely it is even harder to do. The sunk cost of planning and preparing is holding everybody back from realizing that we should not go further, even though everybody is intelligent and intellectually knows that this is an economic fallacy and we should not throw good money after bad money.

The fact that the tool in question helps the other project but makes things harder for us is also a complicating factor that does not lend itself well to simple plans and one-dimensional views of the world. Therefore such complicating factors are often overlooked.

Finally there may be some prestige invested by various parties that makes it hard to move away from made plans.

Policy and reason don’t mix

A few days ago, my wife ran into something that made me think about policy and reason…

Where we live there is a bus that we usually ride to go to the subway station. Ever since a new contractor took over the bus-operation, the service of the busses has deteriorated sharply. A few days ago the bus was late, again. My wife called the bus-company to complain and learned that our local busstop does not have a fixed schedule, instead it is only the major busstops that do. The schedule for the intermediate ones are only there as a guide and the busses may be up to 10 minutes late depending on traffic.

Now, my wife tried to interject that there was no traffic at all on the roads. Still the same answer. Now she tried to say that there is no way the bus can be 10 minutes late when our local busstop is only 2 stops away from where the bus starts (which according to the explanation before should have been one with a fixed schedule). Still the same answer…

A related case was several years ago when my friend the pilot was going on a flight as a passenger. He got a bad seat and asked the flight attendant if he could move to another seat. The answer was that this was not possible, since it would screw up the pilot’s weight and balance calculations. My friends answer was: “Not weight, but possibly balance”

Both of these cases point me to something. In both cases there was some policy decided by the organization in question. Policys are there to guide decisions. But once the policy is made, it is very hard for somebody to try a logical argument against them. The people on the front line of the organizations with the policies can easily ‘hide’ behind the policies so that they do not have to think for themselves. Organizations with policies may feel that they do not have to deal with real-world issues and circumstances and do not have to empower/educate their front-line personnel. But also, they miss out on great opportunities for improvement and excellent customer service.

Fail-fast and garbage files

Today it happened again… Why don’t we have a better way of handling this?

My project (as I probably mentioned before) is a data warehouse solution to calculate market risk for a bank. As all data warehouse solutions much of what we do deals with data in files and data in tables. The tables themselves are represented as files also (we are using SAS solutions). Over time things change, names change, definitions change, machines change etc. But in many cases the old files that are not relevant anymore do not get deleted and they become garbage that lays around the various directories of our system. Everything works fine and nobody is worried.

Then one day we move to a new machine, or to a new environment, or to something else. All of a sudden, the garbage is not there anymore. And we get a crash. Turns out that everything was not fine, since the system did not read the files we thought it read but instead it read the garbage. With no garbage, nothing to read, and therefore a crash. This has happened to us at least four or five times over the past 3 weeks (we are in deployment mode right now).

The agile movement talks about failing fast. You should design your systems so that any bugs make the system crash and crash early. This way, the bugs are found early, and can be fixed quickly and cheaply.

Our data warehouse system, based on SAS has evidently not heard about failing fast yet. Also, I do not see that mindset in the community of data warehouse developers that I have met in this project. Could this be a big opportunity for improvement? Time for me to start some convincing…..

Why you need information modeling and/or simple ways to test and deploy

My current assignment deals with a data warehouse system where some calculations should be done based on the data in the data warehouse.  We also have some tables that control how the calculation should be performed (e.g. how data in separate data sets should be mapped to each other).

Over time, we have introduced more features to the calculation and most of these features need some control table structure so therefore the control tables have evolved into being more and more complex in their structure and in their interrelationships.

Now, with the great benefit of 20-20 hindsight, it is easy to see some areas of the structure of the control tables that are sub-optimal at best and downright ugly at worst.  There is one example where the same column of the same control table is used to map to two separate concepts in the model, depending on what exact data is used.

Why did this happen?  Why were we not able to control the structure of these important tables over time in the project.  I see a few different elements of an explanation:

  1. We did not have a good enough understanding of the total problem when we started.  (of course this is not what you typically have in an agile project, which leads us to the other explanations)
  2. The project, like all projects, was really pressed for time, so nobody wanted to redo work that was already invested in the early versions of the control tables
  3. The test and deployment processes of the project takes a long time, therefore rework was not really wanted.

So, what do I think we should have done instead

  1. Invested some time in some information modeling early on, so that we could have a better view of the full picture of these control tables early on
  2. Created reusable test cases and possibly some level of automated tests so that we could easily retest something that was reworked.
  3. Spent more time after the initial functionality was created to cleanup and fix these kinds of technical debt issues.

Project management and one-dimensional thinking

In my current assignment I was recently in a meeting where we were discussing a new technology that I have some previous experience with.  The purpose of the meeting was for the project to understand how we would go about learning more about this technology that could possibly be used by us.

I don’t want to bore you with the specifics of the technology, because that is not really the point of this post, but let’s just say that it is a standard for exchanging messages about financial transactions.  The world of financial trading is complex, multi-faceted and multi-dimensional.  On the one hand you could talk about separate classes of financial instruments (equity, bonds, fx etc), on the other hand you can talk about separate classes of transactions (spot, option, future etc) and on the third hand you could also talk about separate types of information being conveyed by these messages (transactions, positions, events etc).  I have found that navigating such a multi-dimensional world in various discussions is hard and on many occasions I have been in meetings where people did not understand each other because of confusions on where they were in this multi-dimensional world.  In fact, I once wrote an article for The Rational Edge on just that topic but for system models, let me see if I can dig it up again and post here….

Back to my meeting…

Early on we were discussing what asset classes that the technology in question supported.  The people in the meeting were mainly project managers and requirements people (and me…)  The project managers being who they are quickly made a list of asset classes and gave the task to the requirements people to investigate what parts of the list were supported and what parts were not.  Now I felt I had to say something… My understanding of the technology in question is that it mainly deals with transactions and events and not in positions.  For our purposes we were interested in positions.  This is a separate dimension so it doesn’t matter if the technology supports messages abouttransactions of various asset classes, since we were interested in the positions of those asset classes.  But somehow I was not really able to get this point across to the other people in the meeting.

Maybe I did a poor job of communicating, but I don’t think that is the whole story.  By making that first (one-dimensional) list, the project manager inadvertently locked peoples thinking into that the list was the only important thing.  If we could just cover the list, everything would be fine.  Furthermore, the list was easy to check off in the documentation of the technology so the job at hand would be easy to do…  My objection forced people to wake up from this illusion which is always hard to do and therefore usually meets some resistance.

My experience is that many project managers are so focused on creating a (one-dimensional) list of things to do and then to check off that list that it is hard to for somebody else to point out that the real world usually does not lend itself well to be described by one-dimensional lists.

 

<Update 140401: I have found the article. You can find it here>