In principle identical, fundamentally different

The data warehouse is a very robust piece of software, it got it's rough spots and edges, but it hardly ever fails. So it's not surprising Henrik the newest in the line of bright youngsters working in the Data Warehouse have not been exposed to bugs before. But now he found a gronk, a huge SAP data extraction fails silently leaving no traces and no data. I looked at the problem, it looked like a no-brainer, so I just added a few tracing statement in my development environment and everything worked well I got about seven million rows from SAP, so I assumed I in the past had fixed the bug but not moved the module to the production environment. I moved the development module into production but forgot to test, which of course had consequences, some hundred extraction jobs failed that night. We had to roll back the original version of the script and Henrik had to rerun the failing jobs. Later when Henrik discussed the bug with Camilla (the BI team lead) he said "It's a bit puzzling, it works in the development environment and Lars told me it is in principle identical to the production environment." Camilla burst into laughter "when Lars says 'in principle identical', he means fundamentally different".

I have had another look into this bug and it looks like we have a problem with the str_replace function of PHP. With a bit of luck I will hunt this bug down during the weekend. I have promised not to touch the production environment though.

Many years from now, some years after I left the company the first time I had a chat with the IT manager and he said "You are great and you have built us the best production environment for our ERP systems in the world, but the environment have stabilized a lot since you left".

It is very hard to combine innovative, stable and low cost. You can pick any two, but you cannot pick all three. When I started the Data Warehouse project I decided to go for innovative and low cost.The first because that’s what I am, the second out of necessity, I didn’t have any funds. But still the Data Warehouse is remarkable stable as long as I do not interfere.