A happy new year again

At the end of this year I created a mail dispatch system that made me really aware we’re living on a spinning globe where some already are in 2018, I’m still counting down the last hours of 2017. A happy new year to all of you, my best wishes to all of you for the new year.


When I'm sixty five

In Sweden the age of retirement is 65 years. You can retire between 63 and 67, but 65 is the norm,
you are more or less expected to retire at that age. Next month I am sixty five,
Still I’m not sure what I will do, I have said I will work until I’m 70, but it’s not only up to me or rather
my will. If I want a paycheck someone must be willing to pay and to work I also have to be fit
for work healthy body and an alert mind.
Mens sana in corpore sano.

That is something to pray for Juvenalis wrote. You never know, I hope for the best though.
There is so much more I want to do in my professional life. I said to all and everyone after I’m 65
I will only do fun things. Last week I said “I will only do funny work”, but that was just my funny english,
I meant I will only do work I like, and that is still IT development work, After +40 years I still like invent
new ways of using computers, design and develop systems and programs. I find it extremely
rewarding to meet people in their work and help them convert boring tasks into computer programs.
To be on the shopfloor learning the production process and create better planning schemes it is
really really something I find amusing. When I was young a senior IT manager told me
When you automate a process you should know the process as good as the guys working with it.
'Process phantoms' can ruin any working process. Go out in the real world a practice first.
That advice I always have tried to follow. About 10 years ago I was asked to develop a support system
for some service engineers, I had some clever ideas how to use at the time novel notepads.But first
I followed two service guys on some service calls. The first customer we visited was an Ericson plant
assembling mobile phone base stations. We were told to switch off our mobile devices plus the plant
was radio signal  shielded  preventing outside signals interfere with the base station production and
testing. Next we started service the customer's tools. Take the tool apart, then grease everything
then reassemble the tool, after a days work my body was aching and my arms was covered in
grease up to the elbows. From that I learned paper and pen sometimes are perfect tools for taking
notes. After a thorough cleanup we committed our notes to regular laptops ordered spare parts,
upgraded tool software etc. Notepads of that time was not a clever idea, for different reasons I
never had the opportunity to create a better system for the service guys. A few years later I found
a new SAP module  which I thought could be the base for a service system. I talked about this
module (which I can’t remember the name of while writing) with colleagues. After a while the
company decided to use this SAP module as the base for a new sales system. When that project
started I was silently kicked out. I never knew why, but now knowing what the company created I’m
happy not being part or associated with that project. Today no one knows I was the original proponent
of the SAP module in the company,

I started writing this post with an idea to write about what I want to in the next coming years,
not what I’m likely to do, but what I want to do. The post became something else.
I will do another attempt before next year.


The irony of the singularity

I have since the seventies from time to time pondered upon the AI technological singularity. I do beleive we will eventually reach the singularity, but not anytime soon. This blog post sum up my ideas well. In terms of technological skills we are very far from creating something that can be smarter than us. In fact we still do not know if it is possible to be smarter than what we are. When we are discussing AI we have since Turing always talked about the electronic brain or computer singularity. I find it more likely we can enhance our brain by tweaking our own DNA, if that will make us smarter that remains to be seen.  
Anyway I was discussing the computer singularity  with one of my sons, and we both find it likely if the singularity happens that AI will end humanity, all historical evidence points in that direction and that is what we have to relate to. Then my son said:
“That is the end of it all, when humanity is annihilated, the AI will shut itself off. Who says the superior AI has any desire to ‘live’ or ‘exist’? We just cannot know.”

The singularity happens, humankind is annihilated, the AI shut itself off. That is the end, no harmageddon, no terminal nuclear winter. Just a “Thank you for the fish” a flip on the switch and nothing more. That would be the ultimate irony of humankind.

I find my son’s remark the most thought provoking idea about the singularity I have heard in a very long time.


Last week

The past week we the enterprise IT architects in the company met up for discussions, seminars workshops etc. First physical meeting in a long time, This was a good week, stimulating and inspiring not that often you have the chance to sit down and discuss IT architecture in details. However when IT architects meet we most discuss process management and IT technique per se, and there I have some problems. I’m very sceptical to work processes out of experience, very seldom processes developed for others are followed by the ‘others’. One wise V.P. of IT once told me ‘be aware ot the *process phantoms” (his words), they can screw up anything’. Processes or rules of thumb for yourself or your colleagues is a completely different matter though. Although I’m an IT architect I’m a bit out of my comfort zone when we go deep into architecture. I tend to focus on a bit more mundane matters or ‘Das problem an sich’, take a problem and solve it, then create a framework around successful solutions, work my way up instead of the more common IT architectural ‘top to down’ pattern. I do not think my reversed way is better or worse it’s just another way, unfortunately this thinking is not in line with my colleagues, which sometimes rejects my proposals with ‘Surely you're joking’, but no most of the time I’m serious. One such example I avoid delta loading, full load integration is much less error prone and less complex, anyone can set up self healing full loads in no time, the same cannot be said about delta loads. Yes full loads also complicates some architectural patterns like ‘pubsub’ but that is another topic. Most things we discussed is a bit ‘off topic’ for me, I’m more in the ‘business’ part than IT techniques we discussed.

Most of us architects are involved in an Active Directory split of grand proportions due to the split of the company, right now we are migrating 15.000 people around the globe to a new AD, and this is just the first part. According to Microsoft noone have done such an extensive AD split before. I have a peripheral role in this AD project, I created a mail dispatcher sending mail instructions to people before and after migration. I can see other uses of the mail dispatcher, so I will try to productify it. That is the way I operate.

Next year as an architect I hope to work on packaging development environments, IoT, Business Intelligence, manufacturing automation  and a lot of other cool things. But for some reason I most of time end up working on other things than I planned. And I have a meeting with a swedish master data management user group coming up later this month, something I look forward to, we use to have a good balance between practical matters and master data management per se.


Adaptive development

Recently I helped some colleagues to automate mail deliveries. When they first asked for help they did not really know what they wanted. They asked if I could help them create a schedule when to send mails to recipients. All they had was an Excel sheet with mail addresses of people who should be migrated from one Active Directory to another with date of migration and location.
Based on this Excel sheet I created a small database and a prototype of a  simple mail generator. When I showed my prototype it took a while before they understood the prototype was more than a schedule, their first impression was an overly complex schedule. But when they realised the prototype was actually emitting mails according to the schedule they were more than happy “we didn’t know how we should be able to deliver those 50,000 emails”.
When I asked for the contents of the mails they told me it was five emails in eight languages. That should not be a problem I only had to replace personalised items in the mails  with my symbolics. The mail templates was in MS outlook msg form. That was a problem, my mail-generator was in Linux and I did not have any tools to work with outlook msg files. After some experiments I concluded  best I could do was to convert the templates to htm in UTF16le encoding and transfer them binary to Linux. In Linux I first converted the templates to UTF-8 encoding and then manually edited the HTM files added images removed some distorting HTML code.
When I started to test I realized the dates were in local time and my server was on CET time.
I asked if it mattered if the  mails arrived one day early, “yes it does, the mails should arrive 05:00 in the morning local time”. Not only had I to calculate the day  to deliver the mails I also had send it the right time of the day. Back to the drawing board and after some tinkering I had modified the generator submitting mails at the right time. Now we were ready for production and a lot of mails were fired of each day at the right time.
Now my colleagues came back and told me “We have a problem, the ‘welcome emails’ sent after the migration should not be sent following the schedule as we said. We like these mails be sent when the migration is confirmed by the local IT support. They send us a report each morning at 09:00”. Tomorrow I will get this report, tweak the system yet a bit and fire off emails to successfully migrated users.

In total we have spent less than two hours discussing functionality of this rather complex mail automation. Not one line of specification is written, no project plans, no process descriptions, no sprints. Nothing is planned and nothing is written but a fully operational system. I consider this adaptive development style superior to any other development model I have seen. It is the dev part of the extreme devops style I always used which I modestly call ‘never on a Friday’. Extreme in extreme devops comes from the fact not only do the same person do development, put it into production and run the operations, I also design and build the hardware and install software from opsys to application code.


Email automation, the generator - 6

“Can you help us? We need to send  mails to users before (and after) they are migrated  migrated to a new Active Directory.”.
I had an idea in the back of my head to turn my Data Warehouse into a mail generator.

In some previous posts I described the setup for my mail generator. Create a database with recipients and rules for the mail distribution and transform the outlook mail templates so they can be used by the mail-generator which I describe in this mail. Without much further ado here is an overview of the ITL schedule:
First we remove recipients not eligible for these mails, then we create a list of all mails we will send this execution, then cleanup work tables, then log all mails, then send the mails and finally send an Excel with all all mails sent this time to the Project manager. Since mails should be delivered early morning local time, this schedule is fired off from Cron hourly as there is always early morning somewhere on the globe.
The first job ‘filterRecipients’ with global symbolics needed along the way is trivial:
First is the symbolics and then the job   ‘filterRecipients’ which creates a working copy of the recipients and then remove UK recipients as they will get their mails from elsewhere.
Next is the big thing, the ’createMailList’ job select the recipents  who should have a mail today and whose local time is 05:00:00. There is migration date  for each recipients, and the recipient should have 5 mails the first 2 weeks before and the last mail 2 weeks after the migration date. There are many ways to solve this, I choosed to create an iterator with dates spanning from 2 weeks before the first migation date until 2 weeks after the last migration date. Normally start and end date is current date, but varying these dates I can simulate parts or a whole mail campaign.
One further complication is the mail receiving date is a local date while the server is on CET, this means some mails must be sent the day before which is pretty hard to select, so I have introduced a ‘delivery date’ which is dependent on the recipient's time zone.
This first part of the job sets up start and end date and the forevery iterator function is setting up the date array which is consumed by the forevery iterator itself.
This iterator traverse through the subsequent SQL day by day producing a list of mails to be sent.

At the end you see 2 SQL-converters, the default one creates an iterator-array and the Excel one creates an Excel sheet used in subsequent jobs.
The next job logs the mails to be sent, inserting rows one by one from the iterator

After the mail list is created and logged it is time to send the mails:

And finally send the Excel with all mails to whom it may concern:  

The prereq in the beginning is a boolean gate, this one is is true if there is mails sent above.
This schedule is the run every hour, I use CRON for scheduling.

Here is the complete ITL code for the mail generator:

<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<schedule mustcomplete='yes' logresult='yes' period='day' notify='admfail.xml' logmsg='Send NGAD migration mails'>

 <tag name='C_DB' value='NGAD' cmt='The mail database'/>
 <tag name='STATUS' value='Planned' cmt='the status of eligible users'/>
 <tag name='TID'> <function>return microtime()</function></tag> <!-- a unique id -->

 <job name='filterRecipients'                    cmt='Remove recipients not eligible for mails'>
   <sql name='recipientsCopy'>
        use @C_DB;
drop table if exists `@RECIPIENTS_ACTIVE`;
   <sql name='brexit'>
 <job name='createMailList'                      cmt='Create an iterator with each mail we will send this time'>
    <tag name='STARTDATE'>
     <sql>SELECT CURDATE() as STARTDATE</sql>
   <tag name='ENDDATE'>
     <sql>SELECT CURDATE() as ENDDATE</sql>
   <tag name='TEMPTAB' value='mailusers'/>
     <function cmt='Generates an array of arrays with one date for each date between @STARTDATE and @ENDDATE'>
$dt1 = date_create('@STARTDATE');
$dt2 = date_create('@ENDDATE');
$days = date_diff($dt2, $dt1);
$days = $days->format('%a');
$days = $days + 1;
$rows = array();
$dt = date('Ymd', strtotime("@STARTDATE"));
for ($i = 0; $i &lt; $days; $i++) {
   $rows[] = array(MAILDATE=> date('Y-m-d', strtotime("$dt"." +$i day")));
return $rows;
   <sql cmt='get a row for each user that should receive a mail now'>
     use @C_DB;
     create temporary table `@TEMPTAB` (
     SELECT '@MAILDATE' as xQmailDate
     ,a.`OLD Mail` as xQoldMail, a.`New e-mail` as xQnewMail, a.`OLD UPN` as xQoldUPN, a.`New UPN` as xQnewUPN
     , a.`Time Zone` as xQtimeZone, a.`Location/City` as xQlocation
     ,a.`Short Name` as xQshortName
     ,a.`Migration date` as xQmigrationDate
     ,coalesce(b.`ITcontact`,d.`ITcontact`) xQITcontact
     ,c.`ISO6392` as xQlanguage
     ,coalesce(b.`migrationRoom`,d.`migrationRoom`) xQmigrationRoom
     ,coalesce(b.`ITcontactRoom`,d.`ITcontactRoom`) xQITroom
     ,coalesce(b.`replyto`,d.`replyto`) xQreplyTo
     ,e.`tz_offset` as xQtz_offset
     , case '@MAILDATE'
     when SUBDATE(a.`deliveryDate`, INTERVAL 2 week) then concat(c.`ISO6392`, '-2w.htm')
     when SUBDATE(a.`deliveryDate`, INTERVAL 1 week) then concat(c.`ISO6392`, '-1w.htm')
     when SUBDATE(a.`deliveryDate`, INTERVAL 1 day) then concat(c.`ISO6392`, '-1d.htm')
     when ADDDATE(a.`deliveryDate`, INTERVAL 1 day) then concat(c.`ISO6392`, '+1d.htm')
     when ADDDATE(a.`deliveryDate`, INTERVAL 2 week) then concat(c.`ISO6392`, '+2w.htm')
     else 'Not Found'
     END as mailForm    
left join LocationDate b on b.`Location/City` = a.`Location/City` and b.`Migration date` = a.`Migration date`
left join Language c on c.Language = a.Language
left join Location d on d.`Location/City` = a.`Location/City`
left join TimeZone e on e.`Time Zone` = a.`Time Zone`
where a.`Status` = '@STATUS' and
a.`OLD Mail` != '' and
time(convert_tz(UTC_TIMESTAMP(),'+00:00',e.tz_offset)) BETWEEN '04:45:00' AND '05:15:00' and
SUBDATE(a.`deliveryDate`, INTERVAL 2 week)
,SUBDATE(a.`deliveryDate`, INTERVAL 1 week)
,SUBDATE(a.`deliveryDate`, INTERVAL 1 day)
,ADDDATE(a.`deliveryDate`, INTERVAL 1 day)
,ADDDATE(a.`deliveryDate`, INTERVAL 2 week)
SUBDATE(a.`deliveryDate`, INTERVAL 1 week))
, coalesce(b.`subject`, c.`subject`) as xQsubject
, coalesce(b.`Mailform`, c.`Mailform`) as xQmailForm
left join mailTemplate b on  b.`ISO6392` = a.`xQlanguage` and b.`Mailform` = a.`mailForm`  
left join mailTemplate c on  'eng' = c.`ISO6392` and c.`Mailform` = concat('eng', substr(a.`mailForm`,4))
where coalesce(b.`Mailform`, c.`Mailform`) is not NULL
DROP table if exists `@TEMPTAB`;
   <!--time(convert_tz(UTC_TIMESTAMP(),'+00:00',e.tz_offset)) BETWEEN '04:45:00' AND '05:15:00' and -->
   <!-- time(convert_tz(UTC_TIMESTAMP(),'+00:00',e.tz_offset)) BETWEEN '04:45:00' AND '05:15:00' and -->
   <sqlconverter name='sqlconverter_default.php' target='driver0'/>
   <sqlconverter name='sqlconverter_PHPExcel04.php' target='driver0'/>
   <Zsqlconverter name='sqlconverter_CSV.php' target='driver0'/>
 <job name='cleanupTables'                       cmt='Remove work tables'>
        use @C_DB;
drop table if exists `@RECIPIENTS_ACTIVE`;

 <job name='logMails'                            cmt='Insert mails in the log table'>
   <forevery array='@J_createMailList/driver0'/>
   <Xsql name='createtable'>
     use @C_DB;
     CREATE TABLE if not exists `maillog` (
     `Email` varchar(80) DEFAULT NULL,
     `Timezone` varchar(48) DEFAULT NULL,
     `Location` varchar(64) DEFAULT NULL,
`Shortname` varchar(64) DEFAULT NULL,
     `Migrationdate` varchar(10) DEFAULT NULL,
     `Language` char(3) DEFAULT NULL,
     Mailform  varchar(10) DEFAULT NULL,
     cdttm timestamp DEFAULT CURRENT_TIMESTAMP
   <sql name='insertrows'>  
     use @C_DB;
     insert into maillog (Email,Timezone,Location,Shortname,Migrationdate,Language,Mailform)

 <job name='sendMail' type='sendmail' send='yes' cmt='Send all the mails'>
   <forevery array='@J_createMailList/driver0'/>
       <mailer info='mailerNGAD.xml' pgm='phpmailer'/>
   <tag name='THECSS' file='@template/css/mysqltable01.css'/>
   <subject>@xQlanguage - @xQsubject</subject>
   <replyto>@xQreplyTo,EpiRoc AD migration</replyto>
    <attachment file='/home/tooljn/NGAD/image/image001.jpg' name='image001.jpg'/>
   <attachment file='/home/tooljn/NGAD/image/image002.png' name='image002.png'/>
   <attachment file='/home/tooljn/NGAD/image/image003.gif' name='image003.gif'/>
    <filebody file='/home/tooljn/NGAD/mail/@xQmailForm'/>
 <job name='sendRecipentsList' type='sendmail'   cmt='Send an Excel with all mails sent to PM'>
   <prereq type='boolean'>(count(unserialize(file_get_contents('@J_createMailList/driver0'))))</prereq>
   <!--The prereq is true if at leat one mail sent -->
   <mailer pgm='phpmailer'/>
   <subject>Mails sent by @C_application mail generator</subject>
   <attachment file='@J_createMailList/driver0.xlsx' name='sentMails.xlsx'/>
Hi, in the attached Excel you find a list of mails sent.