Email automation - 1

This is the second post (the first post, the next post you find here) in my quest for mail automation I need to grab some data from an Excel sheet. This is my first attempt. Exporting data from Excel the wrong way.

For my email generator I need the recipients to be stored in a MySQL table, today the recipients are maintained in an Excel sheet.  First I need to export the recipients from Excel. Below you see the Location/City Pará in my Excel sheet.

Now I save this  as  a UTF-8 CSV file, following this simple procedure:  

End of simple Excel  UTF-8 export  procedure.

And what do I get?

Do you see the dark area above? That is an ‘át’ as in  ‘Parát’. Of course it should be an á as in Pará, (‘át’ vas my first attempt to paste it in this text).

This do not look good, and even worse no matter how I try to load this csv file into MySQL I fail. Whatever the problem is this is clearly not the solution. And since the final goal is mail automation, this manual erroneous process this is a dead end.

Email automation - 0

My so called smartphone signals incoming email with a merry sound in a major tone. This weekend the company ERP system is down for service, that in itself is bad. In the 90ties I promised myself only creating “never down” IT systems. I had to cheat a bit neither the technology or the cost was on my side, so I learned when ‘my’ systems was not used and serviced them at those time slots, yes I burned the lamp between 02:00 to 05:00 Sunday nights a lot. When I created the Data Warehouse I kept my promise to myself, the Data Warehouse was up the twelve years I was in charge except three power outages, I think that still is damn good, certainly no other system in the company even come close to those twelve years..

So far this morning my telephone has signalled over 300 incoming mail warning alerts from the Data Warehouse of broken workflow schedules all depending on the company ERP system, you can read about the failed workflows here. (And warning alerts keep on coming.)

Email automation is something I started in the early 80ties using the Memo system of Volvo IT in the mainframe world. Memo was an early Email system not built for automation at all, but I with some ingenuity and assembler skills was able to send warnings from the company ERP system at the time, the ERP system I had built myself (yes I am immensely proud of that, not many can say they built an ERP system for a global enterprise, yes again I did it all myself, production, logistics & purchase).

Email automation is something I worked with the last two weeks and that is the reason for this post, hopefully the first in a serie about my new email generator.

Perl6 CRO - 2

In my first post about Perl6 and CRO I  mentioned I had problems take care of POST and GET messages at the same time.  The problem was a misspelled semi-colon

get -> 'greet', :%zname {   say %zname:  content 'text/plain',  'bye'; }

I wrote a colon instead of the correct semi-colon, you see it above say %zname: the result was

Cannot resolve caller say(Hash: DateTime); none of these signatures match:    (Mu $: *%_)  in block  at myfstserver.p6 line 26  in code  at /home/tooljn/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC86B9F9A31CA5BEE35765E177159BAEDB91AAA (Cro::HTTP::Router) line 79

It took me some four hours to figure this out. If the error message had been a more helpful

%zname: is uninitialized or %zname: does not exist ...

I had probably spotted the problem instantly.

Now  my CRO server works beautifully. CRO seems amazing so far, a hell lot of functionality stuffed in there. Now I only have to learn Perl6. The language is truly great and CRO looks even greater and it is only the first beta version, really looking forward to the next version.

Update:
I just learned the colon I tripped over is an alternative method invocation syntax, which makes my complaint invalid. I need to learn Perl6 before complaining, but thinking before talking is not one of my stong suits, (either😊)

Object Orientation is not such a good idea

 I found this on youtube https://www.youtube.com/watch?v=QM1iUe6IofM
The speaker expresses much the same ideas and opinions I have about Object Oriented programming. Actually on programming  at large, the speaker offers an alternative the good old procedural style as a better alternative.
I have many times found myself in the same situation the speaker describs when you need to change OO programs.You end up with many classes and interfaces with small code bits scattered all over.

Perl6 CRO and Data Warehouse streaming services

What do you do when you do not have anything to do on a Sunday afternoon?

Why not implement Streaming services for a Data Warehouse? The company’s tool fleet can send performance stats over the net. I get questions about and  requests for Internet of Things, if not daily at least on a monthly basis. Often from apprehensive or perplexed persons asking for bongoDB or Big Data Hadraloop solutions. For some reason they all seem to think they need some new more or less esoteric software to gather and store data from net-connected gadgets. Sunday after lunch I stumbled on this presentation. I liked the idea about building a streaming service of stacked micro services so I had a go with Perl6 Cro. I sat up a streaming HTTP server using Cro, this server receives GET and POST calls and write them to disk as text files. Then there is another service ( a PHP program) picking up these textfiles and inserts them in a MySQL database.

The Data Warehouse streaming services

This is not by far an optimal performant or final solution, it is just a PoC, still my first test indicated it can easily handle 10 transactions a second (nothing in the chain seemed saturated), and it is a persistent solution, all incoming transactions are written to disk and not erased until they are safely stored in the database.

Not bad for a Sunday afternoon. I had some help from Andreas with POST messages and Postman for testing.

This solution is not without problems, CRO is in Beta stage, and I have problems to take care of POST and GET messages at the same time. I do not know Perl6, so I do not know if my Perl6 knowledge is too low, the CRO documentation is not correct or there are bugs in CRO first Beta version. But I’m confident my issues will be ironed out in near future. There are other more mature frameworks, but CRO looks both cool and promising, so I’ll try to stick to it.

I have said many times SQL databases can be used for so called Big Data applications, and I always eat my own dog food. If for some reason it shows SQL databases is not up to the task I switch to something else. The same goes for CRO, but it’s not fair to ditch a Beta software just because you do not understand it, and so far it looks great. (Andreas thinks I should go for a mature production ready alternative though.)

Web server and new glasses

Some years ago I wrote my first Node.js Hapi.js program. Now it was time to write the second, or rather extend the first with a new function. First I upgraded  Node and Hapi from releases 0.x.x something to node vs  8.4.0 and Hapi vs 16.5.2, it looked like a giant leap and I expected to rewrite all from scratch, but only a few simple adjustment was needed.

The first version run a Data Warehouse job in the background and at the end responded if it was a success or not. That was fine until some long running made the HTTP interface time out. After many futile attempts to fix the timeout, I called Andreas he told ‘you don’t do it that way, create one function that starts the job and immediately return the pid of the submitted job, and another function that replies if the pid is active or not. I should have figured out that myself, but I didn’t.

Anyway that should be simple and it actually was except for one detail, I have new glasses, text is a bit blurred with the new lenses.

var foo = spawn("/bin/ps", [' -p ' + pid]. {shell: true});

The invalid dot magnified, should be a comma.

I had by mistake written a . (dot) instead of a , (comma), but I did not see the dot to me it looked the same as the comma it should be. Spending an entire day on google trying to figure out what the problem was, I finally rewrote the spawn statement and it worked just fine. Sometimes programming is so frustrating, simple things can take forever.

Now I will rewrite this simple web server into a proper REST server. This should be easy, my program is more or less a REST server as it is, I just have to tweak the code a bit.

But as usual I’m confused with the never ending list of ever growing Node js frameworks. I am thinking adding TrailsJs to Hapi.js just for the hell of it. I have absolutely no clue what the implications of that is.

The new “PID active” function:

It remains to be seen if I have to set the spawned process as detached or not. According to the documentation ‘detached’ is for Windows only.

The growth of

The Data Warehouse still surprises me. After 16 years it still grows in terms of users, jobs, sql queries, reports etc. I had a look the other day at a Twitter published graph and saw jobs/month has increased from 58000 to 71000 during the last year, a staggering 22% that’s a lot.

Number of jobs or resource consumption are not a very good measures of the useful production of a Data Warehouse or any other application. It is very hard to measure the useful production of most computer apps. Number of jobs may indicate inefficiency or a stupid divide of logical tasks into too many jobs. Anyway it is a measure, assuming the Data Warehouse is not deteriorating the increase of jobs is an indication of increased use. As far as I can see the Data Warehouse is not deteriorating, if anything it is getting better.

Never in my wildest dream did I anticipated the phenomenal progress of the Data Warehouse when we started 2001 by solving one problem for one purchaser. And never did I expect the Data Warehouse be alive and kicking 16 years later. I'm not surprised however, useful applications have a tendency to live long. After my rewrite of the first version 2006, the Data Warehouse was superior to anything there was on the market. Some claim it still is, I don't know about that but it is certainly different from anything on the market.