I just read Martin Fowlers blogpost on the concept of Datensparsamkeit that you can find here
My summary of his post is that he argues that organizations should only store the data they really need and not, in this day and age of big data, store all data they can get their hands on. His primary concern is that of privacy.
I fully support his concern of privacy, but reading the blogpost it got me to think about another reason for Datensparsamkeit.
In my current project we have been collecting data from various sources, internal to the organization as well as external. After all, the project is about a data warehouse and those are fundamentally about collecting data. In some situations the data was not needed right away, but for various reasons we built the interface to collect it anyway. After this, the interface was put into production and the data collection was started.
In all of the situations when we collected data that was not needed right away, we run into serious data quality issues. We had situations when the interface was broken for months, no data was collected nobody realized it. We also had a situation where a calculation to generate derived data was seriously flawed for months and nobody realized it.
Since the data was not needed right away, nobody was ensuring the quality of it and therefore the collection of the data was not only a big waste, it also led to serious rework to fix the problems.
So, in addition to the privacy concerns of Datensparsamkeit, I would also add serious data quality issues that leads to Datensparsamkeit.
Let me tell you a little story from my current assignment. I have a gig as a developer in a project. A few days ago I was about to start on a new piece of functionality that nobody had thought much about beforehand. There were no requirements and only vague ideas of what the functionality was good for. Given that I have much experience in working with requirements I thought that I was certainly capable of finding out the requirements myself and this would be a much faster way to do things than to wait for an analyst to show up. So, I started asking questions, trying to get further down in the details of what this new functionality should do, why it should do it and how it should be done.
I have been in that situation many times before. This time, just as practically all other times, the response from the people around me was not positive. Instead it was along the lines of “what, you are a technical person; you should not do that work; that is for analysts to do…”
Why is that? Why have most organizations put in a very high wall between requirements people (analysts, business people) and technical people (architects, developers)? Why are people so afraid to “break” this boundary? My latest employer even had distinct lines of organization for analysts and architects and it was very odd to be somebody like me (who is somewhat of a mix between analyst and architect/developer). Also, since many organizations have this wall in place, there tends to be a focus on some kind of formal handoff from requirements to development.
To me it is obvious that there are great benefits of working on both sides of this distinction simultaneously. Requirements without the understanding of what really can be done and what is easy and hard in the current environment tend to focus on the wrong things and not reaping the benefits that the technical environment gives you. Architecture/development without the understanding of what is really needed and why tends to build great technical solutions that few people want.