12D

2017-11-26

The irony of the singularity

I have since the seventies from time to time pondered upon the AI technological singularity. I do beleive we will eventually reach the singularity, but not anytime soon. This blog post sum up my ideas well. In terms of technological skills we are very far from creating something that can be smarter than us. In fact we still do not know if it is possible to be smarter than what we are. When we are discussing AI we have since Turing always talked about the electronic brain or computer singularity. I find it more likely we can enhance our brain by tweaking our own DNA, if that will make us smarter that remains to be seen.

Anyway I was discussing the computer singularity with one of my sons, and we both find it likely if the singularity happens that AI will end humanity, all historical evidence points in that direction and that is what we have to relate to. Then my son said:

“That is the end of it all, when humanity is annihilated, the AI will shut itself off. Who says the superior AI has any desire to ‘live’ or ‘exist’? We just cannot know.”

The singularity happens, humankind is annihilated, the AI shut itself off. That is the end, no harmageddon, no terminal nuclear winter. Just a “Thank you for the fish” a flip on the switch and nothing more. That would be the ultimate irony of humankind.

I find my son’s remark the most thought provoking idea about the singularity I have heard in a very long time.

2017-11-19

Last week

The past week we the enterprise IT architects in the company met up for discussions, seminars workshops etc. First physical meeting in a long time, This was a good week, stimulating and inspiring not that often you have the chance to sit down and discuss IT architecture in details. However when IT architects meet we most discuss process management and IT technique per se, and there I have some problems. I’m very sceptical to work processes out of experience, very seldom processes developed for others are followed by the ‘others’. One wise V.P. of IT once told me ‘be aware ot the *process phantoms” (his words), they can screw up anything’. Processes or rules of thumb for yourself or your colleagues is a completely different matter though. Although I’m an IT architect I’m a bit out of my comfort zone when we go deep into architecture. I tend to focus on a bit more mundane matters or ‘Das problem an sich’, take a problem and solve it, then create a framework around successful solutions, work my way up instead of the more common IT architectural ‘top to down’ pattern. I do not think my reversed way is better or worse it’s just another way, unfortunately this thinking is not in line with my colleagues, which sometimes rejects my proposals with ‘Surely you're joking’, but no most of the time I’m serious. One such example I avoid delta loading, full load integration is much less error prone and less complex, anyone can set up self healing full loads in no time, the same cannot be said about delta loads. Yes full loads also complicates some architectural patterns like ‘pubsub’ but that is another topic. Most things we discussed is a bit ‘off topic’ for me, I’m more in the ‘business’ part than IT techniques we discussed.

Most of us architects are involved in an Active Directory split of grand proportions due to the split of the company, right now we are migrating 15.000 people around the globe to a new AD, and this is just the first part. According to Microsoft noone have done such an extensive AD split before. I have a peripheral role in this AD project, I created a mail dispatcher sending mail instructions to people before and after migration. I can see other uses of the mail dispatcher, so I will try to productify it. That is the way I operate.

Next year as an architect I hope to work on packaging development environments, IoT, Business Intelligence, manufacturing automation and a lot of other cool things. But for some reason I most of time end up working on other things than I planned. And I have a meeting with a swedish master data management user group coming up later this month, something I look forward to, we use to have a good balance between practical matters and master data management per se.

2017-11-05

Adaptive development

Recently I helped some colleagues to automate mail deliveries. When they first asked for help they did not really know what they wanted. They asked if I could help them create a schedule when to send mails to recipients. All they had was an Excel sheet with mail addresses of people who should be migrated from one Active Directory to another with date of migration and location.

Based on this Excel sheet I created a small database and a prototype of a simple mail generator. When I showed my prototype it took a while before they understood the prototype was more than a schedule, their first impression was an overly complex schedule. But when they realised the prototype was actually emitting mails according to the schedule they were more than happy “we didn’t know how we should be able to deliver those 50,000 emails”.

When I asked for the contents of the mails they told me it was five emails in eight languages. That should not be a problem I only had to replace personalised items in the mails with my symbolics. The mail templates was in MS outlook msg form. That was a problem, my mail-generator was in Linux and I did not have any tools to work with outlook msg files. After some experiments I concluded best I could do was to convert the templates to htm in UTF16le encoding and transfer them binary to Linux. In Linux I first converted the templates to UTF-8 encoding and then manually edited the HTM files added images removed some distorting HTML code.

When I started to test I realized the dates were in local time and my server was on CET time.

I asked if it mattered if the mails arrived one day early, “yes it does, the mails should arrive 05:00 in the morning local time”. Not only had I to calculate the day to deliver the mails I also had send it the right time of the day. Back to the drawing board and after some tinkering I had modified the generator submitting mails at the right time. Now we were ready for production and a lot of mails were fired of each day at the right time.

Now my colleagues came back and told me “We have a problem, the ‘welcome emails’ sent after the migration should not be sent following the schedule as we said. We like these mails be sent when the migration is confirmed by the local IT support. They send us a report each morning at 09:00”. Tomorrow I will get this report, tweak the system yet a bit and fire off emails to successfully migrated users.

In total we have spent less than two hours discussing functionality of this rather complex mail automation. Not one line of specification is written, no project plans, no process descriptions, no sprints. Nothing is planned and nothing is written but a fully operational system. I consider this adaptive development style superior to any other development model I have seen. It is the dev part of the extreme devops style I always used which I modestly call ‘never on a Friday’. Extreme in extreme devops comes from the fact not only do the same person do development, put it into production and run the operations, I also design and build the hardware and install software from opsys to application code.

2017-10-31

Email automation, the generator - 6

“Can you help us? We need to send mails to users before (and after) they are migrated to a new Active Directory.”.

I had an idea in the back of my head to turn my Data Warehouse into a mail generator.

In some previous posts I described the setup for my mail generator. Create a database with recipients and rules for the mail distribution and transform the outlook mail templates so they can be used by the mail-generator which I describe in this mail. Without much further ado here is an overview of the ITL schedule:

First we remove recipients not eligible for these mails, then we create a list of all mails we will send this execution, then cleanup work tables, then log all mails, then send the mails and finally send an Excel with all all mails sent this time to the Project manager. Since mails should be delivered early morning local time, this schedule is fired off from Cron hourly as there is always early morning somewhere on the globe.

The first job ‘filterRecipients’ with global symbolics needed along the way is trivial:

First is the symbolics and then the job ‘filterRecipients’ which creates a working copy of the recipients and then remove UK recipients as they will get their mails from elsewhere.

Next is the big thing, the ’createMailList’ job select the recipents who should have a mail today and whose local time is 05:00:00. There is migration date for each recipients, and the recipient should have 5 mails the first 2 weeks before and the last mail 2 weeks after the migration date. There are many ways to solve this, I choosed to create an iterator with dates spanning from 2 weeks before the first migation date until 2 weeks after the last migration date. Normally start and end date is current date, but varying these dates I can simulate parts or a whole mail campaign.

One further complication is the mail receiving date is a local date while the server is on CET, this means some mails must be sent the day before which is pretty hard to select, so I have introduced a ‘delivery date’ which is dependent on the recipient's time zone.

This first part of the job sets up start and end date and the forevery iterator function is setting up the date array which is consumed by the forevery iterator itself.

This iterator traverse through the subsequent SQL day by day producing a list of mails to be sent.

At the end you see 2 SQL-converters, the default one creates an iterator-array and the Excel one creates an Excel sheet used in subsequent jobs.

The next job logs the mails to be sent, inserting rows one by one from the iterator

After the mail list is created and logged it is time to send the mails:

And finally send the Excel with all mails to whom it may concern:

The prereq in the beginning is a boolean gate, this one is is true if there is mails sent above.

This schedule is the run every hour, I use CRON for scheduling.

Here is the complete ITL code for the mail generator:

<?xml version='1.0' encoding='UTF-8' standalone='yes'?>

<tag name='TID'> <function>return microtime()</function></tag>

use @C_DB;

drop table if exists `@RECIPIENTS_ACTIVE`;

create table `@RECIPIENTS_ACTIVE` (SELECT * from RECIPIENTS);

</sql>

delete FROM `@RECIPIENTS_ACTIVE` WHERE `NOIMP:Country Code` = 'UK';

</sql>

</job>

<sql>SELECT CURDATE() as STARTDATE</sql>

</tag>

<sql>SELECT CURDATE() as ENDDATE</sql>

</tag>

$dt1 = date_create('@STARTDATE');

$dt2 = date_create('@ENDDATE');

$days = date_diff($dt2, $dt1);

$days = $days->format('%a');

$days = $days + 1;

$rows = array();

$dt = date('Ymd', strtotime("@STARTDATE"));

for ($i = 0; $i < $days; $i++) {

$rows[] = array(MAILDATE=> date('Y-m-d', strtotime("$dt"." +$i day")));

}

return $rows;

</function><parm/>

</forevery>

use @C_DB;

create temporary table `@TEMPTAB` (

SELECT '@MAILDATE' as xQmailDate

,a.`OLD Mail` as xQoldMail, a.`New e-mail` as xQnewMail, a.`OLD UPN` as xQoldUPN, a.`New UPN` as xQnewUPN

, a.`Time Zone` as xQtimeZone, a.`Location/City` as xQlocation

,a.`Short Name` as xQshortName

,a.`Migration date` as xQmigrationDate

,coalesce(b.`ITcontact`,d.`ITcontact`) xQITcontact

,c.`ISO6392` as xQlanguage

,coalesce(b.`migrationRoom`,d.`migrationRoom`) xQmigrationRoom

,coalesce(b.`ITcontactRoom`,d.`ITcontactRoom`) xQITroom

,coalesce(b.`replyto`,d.`replyto`) xQreplyTo

,e.`tz_offset` as xQtz_offset

, case '@MAILDATE'

when SUBDATE(a.`deliveryDate`, INTERVAL 2 week) then concat(c.`ISO6392`, '-2w.htm')

when SUBDATE(a.`deliveryDate`, INTERVAL 1 week) then concat(c.`ISO6392`, '-1w.htm')

when SUBDATE(a.`deliveryDate`, INTERVAL 1 day) then concat(c.`ISO6392`, '-1d.htm')

when ADDDATE(a.`deliveryDate`, INTERVAL 1 day) then concat(c.`ISO6392`, '+1d.htm')

when ADDDATE(a.`deliveryDate`, INTERVAL 2 week) then concat(c.`ISO6392`, '+2w.htm')

else 'Not Found'

END as mailForm

from `@RECIPIENTS_ACTIVE` a

left join LocationDate b on b.`Location/City` = a.`Location/City` and b.`Migration date` = a.`Migration date`

left join Language c on c.Language = a.Language

left join Location d on d.`Location/City` = a.`Location/City`

left join TimeZone e on e.`Time Zone` = a.`Time Zone`

where a.`Status` = '@STATUS' and

a.`OLD Mail` != '' and

time(convert_tz(UTC_TIMESTAMP(),'+00:00',e.tz_offset)) BETWEEN '04:45:00' AND '05:15:00' and

ADDDATE('@MAILDATE', INTERVAL 0 day) in (

SUBDATE(a.`deliveryDate`, INTERVAL 2 week)

,SUBDATE(a.`deliveryDate`, INTERVAL 1 week)

,SUBDATE(a.`deliveryDate`, INTERVAL 1 day)

,ADDDATE(a.`deliveryDate`, INTERVAL 1 day)

,ADDDATE(a.`deliveryDate`, INTERVAL 2 week)

)

and ADDDATE('@MAILDATE', INTERVAL 0 day) in (

SUBDATE(a.`deliveryDate`, INTERVAL 1 week))

);

SELECT a.*

, coalesce(b.`subject`, c.`subject`) as xQsubject

, coalesce(b.`Mailform`, c.`Mailform`) as xQmailForm

FROM `@TEMPTAB` a

left join mailTemplate b on b.`ISO6392` = a.`xQlanguage` and b.`Mailform` = a.`mailForm`

left join mailTemplate c on 'eng' = c.`ISO6392` and c.`Mailform` = concat('eng', substr(a.`mailForm`,4))

where coalesce(b.`Mailform`, c.`Mailform`) is not NULL

;

DROP table if exists `@TEMPTAB`;

</sql>

</job>

<sql>

use @C_DB;

drop table if exists `@RECIPIENTS_ACTIVE`;

</sql>

</job>

use @C_DB;

CREATE TABLE if not exists `maillog` (

`Email` varchar(80) DEFAULT NULL,

`Timezone` varchar(48) DEFAULT NULL,

`Location` varchar(64) DEFAULT NULL,

`Shortname` varchar(64) DEFAULT NULL,

`Migrationdate` varchar(10) DEFAULT NULL,

`Language` char(3) DEFAULT NULL,

Mailform varchar(10) DEFAULT NULL,

cdttm timestamp DEFAULT CURRENT_TIMESTAMP

) ENGINE=MyISAM DEFAULT CHARSET=utf8

</Xsql>

use @C_DB;

insert into maillog (Email,Timezone,Location,Shortname,Migrationdate,Language,Mailform)

values('@xQoldMail','@xQtz_offset','@xQlocation','@xQshortName','@xQmigrationDate','@xQlanguage','@xQmailForm')

</sql>

</job>

<subject>@xQlanguage - @xQsubject</subject>

<replyto>@xQreplyTo,EpiRoc AD migration</replyto>

<recipient>@ZZxQoldMail</recipient>

</job>

<prereq type='boolean'>(count(unserialize(file_get_contents('@J_createMailList/driver0'))))</prereq>

<subject>Mails sent by @C_application mail generator</subject>

<recipient>lars.a.johansson@se.atlascopco.com</recipient>

<htmlbody><![CDATA[

Hi, in the attached Excel you find a list of mails sent.

</p></body></html>

]]></htmlbody>

</job>

</schedule>