Thank you Amazon

for the appreciation of my language skills.

Based on my recent visit Amazon recommends me to buy Complicated Stuff in Simple English.

 “I know words, I have the best words. I have the best, but there is no better word than stupid.”

Lasse Trump

Cogito, ergo sum

Some days ago I wrote a post on AlphaGo, DeepMind and the GO competition. In the post I urged DeepMind to go for something much harder StarCraft. This was not a random selection, my youngest son happened to be one of Europe's best StarCraft player some years ago. First I dismissed his obsession with the game as just unhealthy waste of time. But after watching him play a few times I changed my mind. I realised StarCraft is an extremely complex game. I still remember the son preparing for a game. ‘I must be in a good mood and have slept well, otherwise I will loose”, “I need to be alert”, he warmed up as a top athlete, with finger practices for the dexterity. StarCraft openings are slow you don’t see much activity it’s more strategy you position for the final kill in the end play where the speed of keystrokes and mouse moves often are insane. “One erroneous move and you’re lost”.

“The best players are not particularly fast, it’s a strategy game much more complex than Chess” my son said. He spend more time studying other players and reading game theory. He tried to study old games of upcoming opponents to understand how they play and finding ways to outsmart them.

I think StarCraft will be a hard nut to crack for DeepMind. They or some other will finally defeat humans in a fair game of StarCraft of that I’m am sure. But will that program contain true intelligence I’m not so sure. It will definitely not be ‘human intelligent’, we may eventually construct ‘human intelligence’ or even ‘super human intelligence’ but that will be 20 years from now for a long time to come, of that I’m pretty certain.

Lately I seen people claiming the Turing test is wrong, since a program can pass the test without having all human characteristics of intelligence. Turing himself was open to ‘alien’ artificial intelligence. One day in a distant future we may see AI smarter than human. But then I’m gone and you too. Actually all of mankind will be annihilated.  

Starting with D again

I started for the X time to study the D language, for some reason I like the language but cannot muster enough energy to do more than a few HW programs. This time I will devote some of the Easter to the language and see if I can connect to SAP just for the hell of it. But just install D caused me some problems Symantec does not like D, first installing the langauage gave me a severe warning:

  

Then installing the project build tool DUB made Symantec annoyed:

 

Symantec simply removed the DUB installer to protect me. However after some fiddling around with a zipped version and installed openRAR via Windows app store I finally got DUB installed.

Now I can start creating the HW program D version for the zillionth time, it remains to be seen if I can make it further this time.

DeepMind GO for StarCraft.

I just watched a fascinating game of GO where the world champion Lee Sedol defeated the Google DeepMind program for the first time.

DeepMind admitting the defeat.

After losing three games the self learning human Lee Sedol has finally beaten the DeepMind machine. It is really fascinating to see humans with self learning capabilities can by themselves learn to outsmart machines.

Not knowing the  game of GO, it seems to me it is an ideal board game for computers to play, the basic rules seems simple to turn into algorithms, it is the humongous possible moves that seems to be the challenge. To store data and calculate next move it should just be a matter of capacity. But it is probably trickier than I think. The DeepMind seems to be faster than Lee Sedol who needed one minute slots extra time for the entire end game. I like to see statistics on think time for the players as the game progressed. Actually I like to see stats on everything from these games. And it would be really interesting if DeepMind could explain the reasoning behind every move.

I have written some posts on my thoughts of AI. I do not believe much on the imminent machine takeover, the so called singularity. We have not figured out what human intelligence is, we have come a long way but we are nowhere near of a full description of our intelligence. It may not be necessary to understand our own brain to create a smarter machine brain, but I think it would help if we know ourselves first. The singularity is still 20 years in future, and I predict it will be so for a while.

Nevertheless it is not a small feat for a machine playing GO 3-1 against human. Now I like to see a machine take on something considerable harder, beat human in a game of StarCraft.

Update:
Oh fuck! DeepMind's AlphaGo program won the last match so 4-1 to the machine against the self learning human brain. I had hoped for a second human victory. Still AI is in it's infancy machines are still far from human intellectual capacity. GO is just a deterministic board game, with simple rules. It is the multitude of possible moves that makes it hard for human. And that also makes it hard for human to study and learn from AlphaGo how to beat the program. Which by the way is an interesting idea, self learning humans learn by study machines how to outsmart them. I still put my two cents on humans.

More parallel processes?

Some time ago I wrote a post about how we improved performance of a job a lot by parallelize the process and introduce a covering index. With the covering index and 10 parallel processes we cut down execution time from +10 hours to a few minutes. The job uses map and reduce design pattern, where 8000 top nodes with some 400000 lower level nodes are assembled into three structures.

Last weekend I decided to see how far I could take the parallelizing within the present infrastructure. How much could we improve the performance by increase the number of threads? Measure parallel processes in a live production environment is quite complex there are many variables to take into account like system load, caches etc. I decided to run my tests during low traffic hours many times with warm caches and buffers,  then take the average from many test runs. The figures were consistent with only small variations so I think result is relevant. As you see in the graph I started with 5 parallel processes and ended with 100 processes. The job scales reasonable well down to about 15 processes, while more than 80 process do not improve the performance much.

Running this job with 10 parallel processes gives a total execution time well below 3 minutes which is much lower than what is required. There is no need to add more parallel processes. But it’s nice to know we can go further.

Improving performance with covering index.

To appreciate this post you must read the post  when fast is not enough first.

I was asked some weeks ago, ‘can you explain your PHP structure assembling program?’.

Why? I asked.

‘It seems we have a problem, it takes about 10 hours to assemble the spare parts structure for the CPD factory!’

Ten hours is definitely too much, in this spare BOM tree there are some 8000 structures or bills of meterial and some +400000 tree nodes. I expected this to take some 15 minutes at the most.

Something must be awful wrong, so I decided to take a look myself first. Sure enough it was the tree assembling PHP program that took some 10 hours. It was setup to distribute the work over 7 parallel threads like this:

This was obviously just cut and pasted from a job with a much larger BOM where the chunks assigned to threads was optimized for that special BOM. As you can see from the forevery iterator, the first chunk takes the first 11000 bills or top nodes. In this example we only have 8000 BOMs to assemble, which means all go into the first chunk effectively single-threading the assembly job, in this case it would probably be better to split the BOMs evenly over 7 chunks like this:

  

Here we do not try to optimize each chunk, it will be good enough to split the BOMs evenly (on 7 chunks) and parallel process them all. Said and done, now we were down to about 1 hour 20 minutes, which was expected since we now run seven parallel threads, still this was not good enough, since I know the PHP program iterates the same SQL query for every node I took a look into PhpMyAdmin to see what’s going on:

  

We run this query over and over again, and a close examine of the table we found an index missing. In this case you really want a covering index as this will drastically improve performance.  This is easily fixed:

We just add an alter statement in the job creating the table.

(A covering index is an index that satisfies a query without accessing the table.)

And now we are down to about three minutes for the tree assembly job.

The moral of this post is: indexes are good, but optimizing too much is not necessarily good.

Good Luck Camilla

This post is the hardest I will probably write in this blog, normally I do not have problems finding words but today I have. There are many thoughts but few words...

One year ago I wrote a post on the progress of the Data Warehouse. It was not a happy post since a highly esteemed member of BI development team left the company.

Today I have to write yet another sad post - Camilla leaves the company after some five years of brilliant work. Apart from being a brilliant Business Intelligence developer she took the lead and added much needed structure, good work processes, workflows and good will to The Data Warehouse.

She will be sorly missed by us all.

But the Data Warehouse must go on. When someone leaves some other have to step in.  This time it is Henrik that will carry on the good work together with newcomer Lucas.

Henrik scores on the football pitch.

I’m confident Henrik and Lucas together with the users will run the Data Warehouse successfully, and carry Camillas legacy forward.