Agile development 2

In my previous post on agile development, I wrote On top of my head I cannot find any special type of application not suited for agile development, having second thoughts about that I could come up with several software development projects not suitable for agile methods. Large IT projects; not only large as in many involved, but also large as in scope, e.g. create a new programming language. If you start to create a new language, you probably need a syntax and a grammar for the new language or a specification of the new language. Just to start develop the new language with two empty hands does not sound right to me. An extreme example of this is the Perl6 language, which has been under development since 2000, in all fairness there has been a working although not complete version since 2010 or so. Perl6 have a detailed pretty static specification, but no time plan ‘it is ready when it is ready’. just lacking a time plan does not make a project qualify as agile. I would not call Perl6 agile. The Perl6 project is as far as I can see well managed, and the most interesting ongoing IT development project I know of. Agile is not the answer to everything.

Detailed time plans are not a part of agile development, but sometimes or rather most of times you need a time plan. Try to tell the boss your latest project is agile and therefore you do not need a time plan. He will not be impressed, he will probably demand  a detailed time plan where one milestone with a deadline is detailed specifications. It will be even worse if you are a consultant trying to sell a development project to a customer. I do not have a good advice how to tackle this situation, personally I cheat a bit. If you read the previous post you know simple use cases are very central to ‘my’ agile model, I call these use cases ‘specifications’. I am a realistic time optimist in the sense I know I am a time optimist. I make up a time plan like this: I start with a rough optimistic time estimate, multiply this estimate with my optimistic factor (2.5) then add another 50% for the unexpected and finally round up to next unit of time, this formula is often pretty accurate. Then I break this down into milestones with desired granularity. Needless to say this is not very scientific, it is sort of case based planning, where I base my time estimates on experience, ‘I know this task takes so long time to complete’.  
While writing this I came across this article. If I got the point of points right, it is a more elaborate ‘time planning’ method than my own but the basic idea is the same.

Once again my intention with this post was to describe an example of agile Business Intelligence, the application and the organisation behind, but if you have followed me this far you can see I failed again. I will make a new attempt to write about that, until then you can read this post.


Agile development

In some posts I have written a little about Business Support organisations and developing methods. I am a strong believer in the agile method. First time I saw agile properly formulated was around 1982, then my now retired colleague/mentor Rolf Carlsson wrote a paper titled Experimental software development through iterations (which I unfortunately have lost), where he described at that time a new revolutionary development model, which today is called agile development. I had already worked according to Rolf’s development model for some year’s so it was nothing new to me, but most people I discussed this with didn’t get it at all. Central to Rolf’s agile model is simple use cases, users must not only understand the scenarios they should feel comfortable with them and recognize themselves in these use cases. A developer creates a basic prototype from a use cases and then gradually refine the prototype together with users. Program specifications in the classical sense is not used at all. Deadlines so important in other project models are insignificant. And that is what I think is the thing most people find the hardest to get a grip on, time based planning is not a part of the agile development processes.    

Today most systems claim to be ‘agile’ and everyone is for ‘agile development’ models, but still few seems to to get around their heads what agile is all about. Agile development has been eloquently and succinctly described in this manifesto and these principles. If you are not familiar with the agile manifesto please study these links carefully before proceed. Agile means you should not use detailed specification or time planning nor should you use deadlines. The world is changing and the only thing we can be sure of, forecasts and predictions of the future are wrong, time plans are forecasts and specifications or software blueprints are predictions of the finished application. Agile means a radical shift away from waterfall methods planning, if you do agile development with detailed specifications and deadlines or time based ‘toll gates’ the agile is just a false label. Agile means continuous real user interaction, users do not (need to) know all up front and they have the right to change their mind any time. Since agile also means refinement of the application through iterative development, the application can be taken into production or ‘system tested’ in an early stage, since you start build a simple working model gradually augmented with required functionality. This is in sharp contrast to waterfall methods where you have a functionality freeze and a development period where the parts are assembled late in the development process for system testing, where users for the first time see the application.

First application I developed according to agile principles was an ERP system including purchase, logistics and production control. That application was in production at different plants (here is one) for some 26 years before it was replaced by SAP. (Yes I’m immensely proud of that.) If you looked at the video you also saw a distribution center, the application controlling the distribution was an agile developed application built by Rolf and a colleague of him, that application was in service only for 25 years. There was a lot of users involved in these projects and I had a lot of help from Rolf with my project, and I assisted Rolf to a small extent with his.

On top of my head I cannot find any special type of application not suited for agile development. But there are IT projects not suitable for agile methods. I can imagine very large projects can be hard to manage without time planning. I believe  large scale (i.e. many IT developers involved) is a recipe for failure and should be avoided.  Implementing standard applications like SAP ERP, is not an agile process in my eyes. You need a meticulously worked out timeplan with deadlines to keep track of the project. If you lose control over the implementation with hordes of consultants and the entire company involved it will be a very costly project, it might even ruin the company. On top of every successful large scale turn key system implementation there is a stern policeman with a rigid timeplan and stopwatch in one hand, a whip in the other and a tight grip on the consultants with his/hers third hand. One can question if implementing a SAP system has anything to do with software development.

Business Intelligence in general is very agile. I argue you must follow agile principles if you want to create great BI applications. My intention with this post was to describe an example of agile BI the application and the organisation behind, but if you have followed me this far you can see I failed. I will make a new attempt to write about that, until then you can read this post.  


PDO connecting to MS SQL and prereqs

The other day I created an interesting job in the Data Warehouse. It’s not a Data Warehouse job as such, I read logs from another application PricePoint and mails the result to concerned. PricePoint is a Windows application and store it’s data in an MS SQL server database. Many here at the company thinks the Data Warehouse is an odd application Linux only, not able to communicate with ‘proper’ software at all (proper is MS). This is of course false, the DW is compatible with many things. The DW have no problem accessing MS SQL server databases or MS office document if required.
There was a little twist with the job I was asked to do. There is a table PendingMessages containing transactions not yet processed. ‘Can you send a mail with failed transactions attached when all transactions are processed, but not later than eight thirty in the morning?’
I could just schedule a job 08:30, but I decided to be a bit creative, so I created a job that checks when all transactions are processed and then sends the mail but no later than 08:30. First I created prereq that checked for this:

This is what this prereq does; It call the PricePoint MSSQL database via PDO and selects number of pending transactions. Since we want the prereq to be false as long as there are pending transactions and true when there is no transactions left we simply negate the result, (we could have created a correct result by writing a more complex SQL query, but just negate the prereq is simpler). We check the prereq every 300 seconds and we do this 20 times or until 08:30 whichever happens first. But there is a snag to this, when time is up we should proceed no matter what the prereq says but this prereq fails when time is up and that is what you want most of the time, wait for an event and if that event does not happen fail. There are some ways to circumvent this problem. You can put the prereq into an <init> section and then explicitly set the init return code to TRUE with a rectify statement like this:

But this not only looks ugly, the logic behind is not obvious and it is a misuse of  both the <init> section and the rectify statement. I do not like to trick my way around programming difficulties so I did not go for this solution. Instead I inserted a <prereqwait> which handles the timing functionalities of the prereq. When I implemented prereqwait I thought it might make sense to allow for a happy ending so I introduced timesup=’ok’ like this:

This is definitely an improvement, actually it’s quite neat, but having second thoughts about the entire job I decided to skip the prereq altogether and go for the simpler solution, just schedule the job at 08:30 like this:

Much better, now no one has  to figure out what the odd looking prereq is good for, but can appreciate the simple MS SQL connectivity via PDO. Take a look at the schedule, there is some nice functionality in there. And I also showed some nice timing functionality by <prereqwait> and <prereq>. I think my jobscheduler with the Integration Tag Language can give any job scheduler on the market a run for their money. (I would love to see someone prove me wrong:)

At last this is what the laconic result from a run looks like in MS Outlook:

Here you see not all transactions were processed at 08:30 and the attachment is a CSV file, it should have been an xmlx file but I missed that, however its just a matter of changing the sql converter in the getMsgs job above.