Showing posts with label Data Warehouse. Show all posts
Showing posts with label Data Warehouse. Show all posts

2016-08-28

The Data Warehouse, the first 15 years



It was 15 years ago today I started up the Data Warehouse, and it has been in style ever since. In its first years it was not much more than some PHP basic scripts  and a MySQL database. But the success of the Data Warehouse and the steep growth of data imports, made it necessary to restructure the data flow in the Data Warehouse.
2004 I decided a more formal approach was needed. I wanted to build a workflow engine (a job scheduler) and a supporting language. The new version of the Data Warehouse was ready for production 2006. The following years some features were added.
From 2009 very little if any development has been done. Still the Data Warehouse is alive and kicking remarkably well. There have been attempts to replace it with mainstream products on the market. These attempts have failed, mainly due to excessive complexity and slowness of those products.

When I started 2000 I could not get funds for Business Intelligence activities at all in the company. I had to use scrapped PCs for servers and do coding myself and use free software. Much have happened since then, today BI activities are allowed to cost, management have realised the value of good BI and today the big guys are prepared to spend millions on financial analytics.
Unfortunately you cannot say the same for what I call operational analytics, i.e. what goes on in the factories purchasing, logistics and production. But the workshop guys at the company do not care much, they already have my low cost Data Warehouse. I was told some months ago, in one of the major factories the ERP system is used for registering only, all planning etc are done in the Data Warehouse + some surrounding apps. The reason for the increased use of the Data Warehouse is speed and costs! I’m told development of extra features in standard ERP systems, take forever and the cost is ridicules high.
Fifteen years is not much for an IT system and I expect the Data Warehouse will celebrate 20 years five years from now. What’s more remarkable about the Data Warehouse, it is barely maintained at all the last five to seven years, still it cracks on, now with a new BI crew. Take any of the standard BI system on the market five to seven years ago. Would you like to run any of those with a new almost untrained crew? I wouldn’t.
Of course I’m not happy about all things in the Data Warehouse, but it is a well designed, stable and balanced Data Warehouse. ‘World class’ a senior consultant Business Analyst once told me, ‘it beats everything I worked with’. All in all I’m very happy with the Data Warehouse.

2014-03-16

Business Intelligence agility, system & organisation


If you follow the data warehouse  tweets, you may have noticed there are often failed jobs in the job statistics. It is actually worse than what you see, jobs which never start due to failing prerequisites is not registered, e.g. a necessary source system job ends unsuccessfully. Still very seldom this is a problem since we only have a production environment, most tests are also run in the production environment, and the majority of failed jobs are tests. You should turn off logging when you test, but who remembers that? And who cares?

In an ERP system landscape it makes sense to have separate testing and production environments for very obvious reasons.You never want to put a transaction or program of any kind into ERP production without proper testing, since ERP transactions often updates the database and it is very hard to test and debug ERP transactions in an everchanging ERP environment. In a test environment you can set up your own scenarios and test the transactions without the risk of other updating your test data. There is also a security aspects, e.g. no sane owner of an application register financial transactions would grant ‘free’ access to developers into the production environment. Thus it makes sense to set up a permanent test environment for ERP systems which also is a good a playground for user training.

For BI or Data Warehouse applications the situation is different, most transactions are read-only (most updates are done in controlled and relatively static batches) and debugging is very hard without current production data. Since the data is stable testing can preferably be done in the the production environment. More important agility is key in BI. Developing directly in production (together with the user) is much faster when you can see the result in real time, than developing in a test environment. I question if you can develop together with users in a test environment at all. Anyway agile it is not. When you need to test updating programs you create a personal mart or take a snapshot, that is one of the key features of a Data Warehouse. And then you test in your mart or snapshot. You need plenty of disk-space for this (and easily accessible backups, for fast restore when you screw up). What use are good data mart facilities if you cannot use them?

If you develop in a separate environment, do not develop together with users, if you as a developer cannot create data marts and/or snapshots on the fly or do not have simple access to backups; you do not have agile Business Intelligence, no matter what your BI vendor says.

There is yet another aspect that can aggravate the agility of the BI, the development methodology. In ERP requirements and program specifications are good things, but in BI specifications other than very high level are evil.
If you develop according to the ERP specification model it is very likely you do not have Business Intelligence at all, it is more likely you are doing standard reporting.


‘When I want a new analysis of stock movements since 2006 I want it in hours not tomorrow. I do not necessarily care about 100% veracity, I care about velocity!’
A Data Warehouse user.


It’s not only the BI system, it is very much your BI organization that determines to what degree you are agile.


The loophole:

The observant reader may have seen a flagrant loophole in this post,  ‘the security aspect of BI development‘. Shouldn’t the BI developer be kept out of the production environment?
I argue you cannot keep the BI developer away from the production data, since debugging and testing often is very hard and sometimes almost impossible without access to production data. There are situations where you cannot allow the developers access to the production data, but then you will not have a very agile BI system. You have to trust the BI developer if you want speedy agile BI systems, and accept the fact the BI developer knows more about the business than they maybe should. 
Business Intelligence developers should sign an agreement of non disclosure.

2014-01-25

The Data Warehouse goes YouTube



For some time now (6 years) I been pondering upon how to visualize the Data Warehouse.  Lately I been inspired by Derick Rethan’s fantastic OpenStreetMap year of edits.

I started by download zillions of software to be able to create ‘activity images’ of the Data Warehouse. I actually managed to create some png frames based on job logs but to do what I wanted I realised it would take just to much of my time  so I stopped. Then I remembered I had seen an animation of web logs some years ago I googled but didn’t find anything. Then I asked  Andreas if he knew about this animation. ‘Yes you sent me a link about this some years ago check out Logstalgia, I send you the link’.

After downloading another shitload of software I had Logstalgia up and running. It didn’t do what I had in mind, but the visualization Logstalgia does is cool, and t is so much easier to animate the logs this way. First I created a pong-event for each entry in the Data Warehouse job logs. I started at CET 18:00 20140115 and ended the next day at 18:00. One day in the Data Warehouse starts at 18:00 by some house cleaning after the backup snapshot is taken. At about 19:00 data loading for our Japanese plant starts. At midnight loading of the other Data Warehouses starts and job activity peaks around 04:00 - 05:00.
At about 08:00 in morning of 2014-01-15, the Data Warehouse and Qlikview users start coming in and the pattern changes. I think this video is worth viewing in it’s entirety.

I had to ‘time compress’ the jobs otherwise the video had been all too long. All jobs started is seen as a pong-ball in the video, the job bounces as a green success or a red failed.
After the jobs I inserted MySQL slow-query log. These entries are marked slow-query and these pong-balls  bounces with the query time in red like 10.05. Slow queries in a Data Warehouse are likely to happen. If you allow users in creating their own jobs and adhoc reporting joining large tables via Excel and Query you have lot’s of slow queries. As long as they do not affect other too much we do not mind.
Next I inserted MySQL general log. This was more problematic. I could not show all queries there were some six million queries that day, I picked all connects but they were too many, I removed all connects from jobs, leaving connects from Excel users, each connect pong-ball bounces with a blue connect.
Finally I inserted Qlikview session start from Qlikview log. This was a horror, the qlikview log is not meant to be parsed by a bash-script, I hand edited this log with the Kate-editor. The Qlikview pong-balls bounces with a green session.

Then I created the video by piping the ‘log-pong’ to ffmpeg:
logstalgia -s 8 --hide-paddle -g "DATAWAREHUSE,URI=mysql?$,1" -800x480 --output-ppm-stream - ~/logstalgia15.txt | ffmpeg -y -r 25 -f image2pipe -vcodec ppm -i -   -pix_fmt yuv420p -crf 1 -threads 8 -bf 0 logstalgia.mp4
Once again the result can be seen at http://youtu.be/bTY9IgJd8kA.
I did some mistakes along the way. First was to start a 18:00 this caused lots of extra work most notably with Qlikview since these logs are organised in dates. Second I used Bash-scripts instead of PHP or any other decent programming language.

This might look like it was easy, but for me as a complete ‘video newbie’, it was not easy, just to make this log-pong flow look reasonably well took a long long time. I’m very happy with the result.

At last the Music; at first I had Verdi’s Thriumpal March in mind but it was to short, so I picked Beethoven’s Final of Eroica instead, not a bad substitute.

2013-12-21

Business Intelligence, the future and beyond

Just a few days after I wrote this post I received a mail promoting a new BI system from one of the big guys.

Excerpt from the admail.

The first two items are almost identical with two sentence I wrote in a document describing my Data Warehouse some ten years ago. I didn’t wrote anything about harnessed though, and  I used the phrase system centric as opposed to my user centric Data Warehouse. And for the last sentence I more than once been baffled by design patterns disrupting the user experience when loading data.
I think this new modern Business Intelligence approach is the way to go, it’s the future and it’s my old Data Warehouse design.

2013-12-15

Some Data Warehouse

I am writing a serie of posts about parallel processing of computer background workflows  in general, and more specific how I  parallel process workflows with my Integration Tag Language ITL . ITL is not a real computer language, it doesn’t generate executing code, it only process  a parse tree, nonetheless it passes for a language in a duck test. I can use ITL  to describe and execute processes and I’m happy with that. Some computer systems I created have been labeled ‘not real’ by others, I don’t mind. Once I created the fastest search engine there was in the IBM Z-server environment, it was labeled (by a competitor) not a real search engine. It was a column based database with bitmap indexes and massively parallel search, it kicked ass with contenders, and I was happy with that. I have created a Data Warehouse, it has also been labeled not a real Data Warehouse, it beats the crap out of competitors though, and I’m happy with that. With my XML based Integration Tag Language I can describe and execute complex parallel workflows easier than anything else I have seen, and I’m happy with that too. Interestingly the host language I use PHP, is often labeled not a real language. PHP a perfect match for ITL, it’s so not real.

Not a real Data Warehouse

My not real Data warehouse has a rather long history, it started around 1995 when I was working as a Business Intelligence Analyst. I got hold of a 4Gb harddisk for my PC, I realised I could fit a BI storage on this huge  disk. At that time I had read a lot about the spintronic revolution that was about to come with gigantic disks, RAM  and super processors to ridiculously low prices. I started to play with the idea of building a future BI system, where you do not have squeeze data onto the disks, where large portions of data could reside in RAM and be parallel processed by many processors.

Simple tables, no stupefying multi dimensional extended bogus schema

The first thing I did was to get rid of the traditional data models , the normalized OLTP, and the denormalized OLAP models, I was thinking denormalize on a grand scale, each report should have it’s own table from which users could slice, dice and pivot as much they liked in their own spreadsheet (Excel) applications. I called these tables Business Query Sets, since in future we would have oceans of disk space , we could afford to build databases as the normal user perceived them, as tables not multi dimensional extended snowflakes or star or whatever the traditional BI storages are called. Have you ever heard a user ask for reports in extended cube format?

Simple extraction, table based full loads

In future data access will be so fast you can extract entire tables, no more delta loads, I thought. No special extractor code in the source system, just simple SQL on tables. Delta loads are hell, the only thing you can be sure of delta loads break, no matter what you been told delta loads fails. In rare situations you need delta loads, you should have good procedures in place to mend broken delta loads, otherwise you will have a corrupt system. As for having special extractor code in each source system, I would probably not be allowed to put in any extractor code in the source system and you lose control spreading out code all over. Special extractor code was never a clever idea anyway.

Tired of waiting for reports?

It’s fast as hell to select * from table . ‘In future I could trade disk space for speed’ I reasoned. And lots of indexes, it’s just disk space. All frequently used data will reside in huge RAM caches. I also envisioned a Data Warehouse in every LAN, since hardware will be cheap you can have BI servers in every LAN . A LAN database  server is faster than a WAN application  server.  

A user Business Intelligence App, not a Business Intelligence System

Not only did I skip the traditional database models, I also scrapped the System , I wanted to build my Data Warehouse around the users, not build a Data Warehouse System. I wanted to invite users to explore the data with tools of their own choice. I didn’t want to build a System BI the user had to log in to and have funny ‘data explorer’ tools straightjacketed onto them.

2001 a start

Year 2001 I could start build my futuristic Data Warehouse. It wasn’t a start I had wished for. Actually no one in the company believed it was possible to build a Data Warehouse my way. I could not get a sponsor, it was only the fact I was the CIO, with my own (small) budget I could start build my Data Warehouse together with one member in my group. I had to start on a shoestring budget. We used scrapped desktops for servers, with a  new large 100Gb disk, 1Gb RAM and an extra network card. I only used free software. Payback time for the first version was one month. I soon began to design and build my own hardware  and since I used low cost components I could afford large RAM pools and keep much of the frequent data in memory. From a humble start with one user the, Data Warehouse  now produces six to twelve million queries (including ETL) a day, it has about 500 users and feed other applications with data, this includes the BI tool Qlikview . From start 2001 until May 2013 the system only has been down at three power outages. When I moved to corporate IT, the managers of the the product company migrated the Data Warehouse from my two of everything hardware design to single hardware , so now we have to take down the Data Warehouse once a year for service. Twelve years of continuous operation, not many system started 2001 have a track record like that.
For not being a real Data Warehouse, it’s quite some Data Warehouse.
Today I’m very happy for the lack of funds at the start. It forced me to think in directions I probably would not have done otherwise. Another piece of happiness; My colleague I started with was completely ignorant (and unimpressed) of Business Intelligence database theories. The few times I complexed the database design he said ‘I will not do that, I create simple tables it is faster and the users want  tables’. Then I measured the different approaches and his simpler models were always better.

Been there, done that

These days Business Intelligence vendors talk a lot about Big Data , hardware accelerators, in-memory databases, nearly online reporting  etc, I can say with a real pride ‘been there, done that’. I do not say other BI apps are not real, they are real, some really good.
I actually have heard a BI sales representative refer to my Data Warehouse as not real. My Data Warehouse has been called a simple Excel app. I have put in a lot of hard work into my Data Warehouse. Countless nights and weekends of coding, testing and measuring. I’m not happy when the appreciation of all that hard work is ‘a simple Excel app’. On the other hand the feeling of knowing ‘not many people know what I know of Business Intelligence applications’ makes me happy. And now seeing the big guys catching up on ideas I conceived some twenty years ago, makes me real happy.

2013-10-07

Twitter from the MySQL Data Warehouse

Some weeks ago I created a tweeting PHP script . Now I use this script for posting job activity from the Data Warehouse on Twitter. Since the  Data Warehouse jobs are registered in MySQL databases, we had to sum up the figures and feed them into the PHP script twitter02.php.  It turned out to an easy task, using the Integration Tag Language . I decided to test this with a  status message showing how many jobs have run the last 24 hours. Here is the schedule:
The first job ‘crtSQLMsg’ sums up all job activity and pass the result to the sqlconverter_CSV.php  script which converts the result table to a file:
Which looks like:
Now it’s only to post this file with the job ‘twittaMsg’, if you study the job you see how the job status message is prefixed with #DataWarehouse.
If you follow @tooljn at Twitter you have seen this tweet as:
 
I’m very happy with the SQL converter  functionality, which out of the box converted the result table into a readable message, and the  @tag GETSQLMSG  which slurps up the message in the subsequent twittaMsg job.
I end this post with the sqlconverter_CSV.php script:
<?php
/**
* SQL result converter - dynamically included in function {@link execSql()}
*
* This converter converts a SQL select result to a CSV file.
*
* This converter also accepts:
*
* Syntax:  <sqlconverter name='sqlconverter_CSV.php' target='report0' headers='no' delim='space' enclosed=''/>
* 1 delim                field delimiter                 default ';' semicolon
* 2 header        field headers                default TRUE/yes
* 3 enclosed        field enclosed by                default NULL
*
* Note delim ' ' doesn't work for unknown reason, so use 'space' instead. Bug?
*
* @see sqlconverter_default.php
* @author Lasse Johansson <lars.a.johansson@se.atlascopco.com>
* @version  1.0.0
* @package adac
* @subpackage sqlconverter
*/
$metafile = $sqltarget.'meta_';
$metasfx = '.TXT';
$targetsfx = 'CSV';
$fieldDelimiter = ';';        // default
$fieldEnclosed = "'";        // default
$headers = TRUE;        // default
if(array_key_exists('delim',$xmlconverter))
  $fieldDelimiter = is_string($xmlconverter['delim']) ? $xmlconverter['delim'] : $xmlconverter['delim'][0]['value'];
if(array_key_exists('enclosed',$xmlconverter))
  $fieldEnclosed = is_string($xmlconverter['enclosed']) ? $xmlconverter['enclosed'] : $xmlconverter['enclosed'][0]['value'];
if(array_key_exists('headers',$xmlconverter))
  $t_headers = is_string($xmlconverter['headers']) ? $xmlconverter['headers'] : $xmlconverter['headers'][0]['value'];
if ($t_headers == 'no') $headers = FALSE;
if ("$fieldDelimiter" == 'space') $fieldDelimiter = ' ';
if ("$$fieldEnclosed" == 'space') $fieldEnclosed = ' ';
$sqllog->logit('Note',"Enter sqlconverter_CVS.php using target=$sqltarget");
if(is_numeric(substr($sqltarget, -1,1))) {
        $metafile = "$metafile$metasfx";
        $sqltarget = "$sqltarget.$targetsfx";
} else {
        clearstatcache();
        for ($x=0; 1==1; $x++){
                if (!file_exists("$metafile$x$metasfx")){
                        $metafile = "$metafile$x$metasfx";
                        $sqltarget = "$sqltarget$x.$targetsfx";
                        break;
                }
        }
}
if (file_exists($sqltarget)) $fpc_flag = 'FILE_APPEND';
else $fpc_flag = NULL;
$report = '';
if ($fpc_flag == NULL and $headers){;
        $meta = '';
        while ($finfo = $result->fetch_field()) {
          $report .= $finfo->name."$fieldDelimiter";
          $meta .= sprintf("Name:    %s;", $finfo->name);
          $meta .= sprintf("OrgName:    %s;", $finfo->orgname);
          $meta .= sprintf("Table:    %s;", $finfo->table);
          $meta .= sprintf("OrgTable:    %s;", $finfo->orgtable);
          $meta .= sprintf("Default:    %s;", $finfo->def);
          $meta .= sprintf("MaxLen:    %d;", $finfo->max_length);
          $meta .= sprintf("Len:    %d;", $finfo->length);
          $meta .= sprintf("Charsetnr:    %d;", $finfo->charsetnr);
          $meta .= sprintf("Flags:    %d;", $finfo->flags);
          $meta .= sprintf("Type:    %d;", $finfo->type);
          $meta .= sprintf("Decimals:    %d;", $finfo->decimals);
          $meta .= "\n";
        }  
        file_put_contents($metafile,$meta);
        unset($meta);
        $report .= "\n";
}
//  Here comes the working code
while ($row = $result->fetch_row()) {
  foreach($row as &$fld) {$fld = "$fieldEnclosed"."$fld"."$fieldEnclosed";}
  $rowstr = implode("$fieldDelimiter", $row);
  $log->logit('Note',"$rowstr");
  $report .= $rowstr."\n";
}
if($fpc_flag == 'FILE_APPEND') file_put_contents($sqltarget,$report,FILE_APPEND);
else file_put_contents($sqltarget,$report);
unset($report);
$sqllog->logit('Note',"Exit sqlconverter_CVS.php");
   

2013-09-08

Moving a Data Warehouse - 3

In some posts I have written about the migration of my Data Warehouse  from my own hardware to Dell servers. This move was partly initiated by my transition to a new job in the company, the management didn’t dare to run the Data Warehouse on the hardware I built .

When I originally designed the Data Warehouse infrastructure two key principles were low cost  and simplicity . I needed a database so I created a database server. I needed an ETL engine so I created an ETL server. Then I needed PhpMyAdmin so I created a PhpMyAdmin server. One function one physical server.  The only extra in my Irons was an extra Network Interface for an internal server network. And in the beginning servers were scrapped IBM desktops. I maximized RAM, replaced the hard disk and added a network interface, dirt cheap. Then I installed a Linux and one application, fired up the new server and forgot about it for two to five years. My servers were mostly replaced when I needed more capacity not due to hardware failures. One of very few hardware problems I have had is described here .

I avoid software tweaking and optimization, I try do do standard installs right from the distro, e.g. I choose ‘big’ for Mysql config file that’s is about how much tuning I do. I have all databases and indexes on the same disk! About half a terabyte database with about eight million queries a day ( I have seen peaks over 15 million queries a day), this is on a custom made server with 16GB RAM.

Now this has changed with the migration to ‘real’ servers. The one server one function  approach would have been all too expensive, so I had a choice either pack more functions in one server or go virtual. I have for some years wanted to test a virtual solution, so I decided to go virtual without testing. I decided I go for two servers one physical database server and virtual host for all other servers. I do not believe for a second you can have a low cost simple virtual high performance database server. But the rest of my Data Warehouse servers could well be virtual, this way I could keep my one function one server  philosophy and still be reasonable cost efficient. I was right and I was wrong.

The new environment is much more complex, virtual servers add a software abstraction layer between the iron and Linux, and by going virtual you also need a software layer between your hard disks and the virtual servers for practical space management. For all this to work you need an expert to manage this environment. And an expert costs and the expert has his own preferences and experiences, e.g. a Linux professional does not necessarily know Mageia Linux. Since we didn’t have in house Linux operations expertise we hired a consultant. A mistake was not to listen to the consultants recommendation of virtualization software and Linux distro.  Not that it’s difficult for a Linux professional to learn another distro it just takes some time, but more important, the support of my now non standard infrastructure it will always be exotic for the consultants operations team. I should have spent more time with the consultants upfront going thru the server setup. We would probably have had a better server setup still adapted to my Data Warehouse.

This is complication I would not have had if we had used Windows Server instead of Linux distros, since Windows is a singular opsys. I do not know if this is good or bad.

I end this post with a humble statement. Still few people seem to have my insights in hardware  and infrastructure for Business Intelligence systems. Actually very few I talk to make any distinctions between any type of applications in this respect, the same hardware fits all give or take some RAM and CPU; that is the adaptation to applications you see. I believe most hardware infrastructure is grossly overpowered/priced and designed for ERP transactional applications.    

2013-08-21

Twitter from the coolest Data Warehouse

Last week a colleague said - Why don’t we Twitter the status from our Data Warehouse servers?

My reply was I tried to set it up a year ago or so but I failed. I used the Zend Framework one dot something. But of course I had to try again. So I started by downloading the Zend Framework 2 version 2.2.2. According to the documentation this was a simple task. After failing a number of times I realised the Twitter class was not in the the vs 2.2.2, after some googling around I found the code and installed it. But still my code did not find the ZF2 twitter code. I could not get the ZF2 autoloader to work so I had to require the ZF2 twitter code manually. Now there were some Oauth classes missing, after some googling I found the code and installed it. I could not get the autoloader to include the Oauth code so I had to require the code manually. And now I descended  into class bonanza hell, the Twitter and Oauth classes referred to other classes and interfaces and abstractions and God knows what. Since the autoloader is too hard for me I had to load all manually, trial and error until I had included all code needed to set up my classes to call Twitter. I was able to verify my credentials and fetch basic information about my Twitter account. But when I tried to post ‘Hello World!’ I got error Http return code 400. After trying a good number of times and googling a lot without any result, I had to give up.

One aspect I do not like with Object Orientation is the extreme complexity. All OO projects starts with a few simple classes with logical methods. But as the system grows (and changes) new classes are introduced which inherits from the original simple classes with almost identical methods. And then hell break loose, now come interfaces, roles, traits, mixins, abstractions, adapters and God knows what. And all these entities should be coupled and attached to each other in ways only logical to the developer. It is no simple task to dig into lots of alien errors like:  

PHP Fatal error:  Uncaught exception 'Zend\Http\Client\Adapter\Exception\RuntimeException' with message 'Error in cURL

request: Received HTTP code 400 from proxy after CONNECT' in /home/tooljn/PHPZF/ZendFramework-2.2.2/library/Zend/Http/Client/Adapter/Curl.php:422

Even if you have stack traces (which always are very long) you wonder what code is executed where and why?

There is seldom intuitivity in OO system, i.e. easy to understand and learn how connect OO object, you very often have to learn case by case.

This is more or less how my ZF2 code looked like when I gave up:

require_once 'Zend/Loader/StandardAutoloader.php';

$loader = new Zend\Loader\StandardAutoloader(array('autoregister_zf' => true));

// Register with spl_autoload:

$loader->register();

require_once 'Zend/ZendService/Twitter/Twitter.php';

require_once 'Zend/ZendService/Twitter/Response.php';

require_once 'Zend/ZendOAuth/OAuth.php';

require_once 'Zend/ZendOAuth/Config/ConfigInterface.php';

require_once 'Zend/ZendOAuth/Config/StandardConfig.php';

require_once 'Zend/ZendOAuth/Client.php';

require_once 'Zend/ZendOAuth/Http/Utility.php';

require_once 'Zend/ZendOAuth/Token/TokenInterface.php';

require_once 'Zend/ZendOAuth/Token/AbstractToken.php';

require_once 'Zend/ZendOAuth/Token/Access.php';

require_once 'Zend/ZendOAuth/Signature/SignatureInterface.php';

require_once 'Zend/ZendOAuth/Signature/AbstractSignature.php';

require_once 'Zend/ZendOAuth/Signature/Hmac.php';

array_key_exists('host',$job) ? $hostInfo = $xmlconverter['host'][0]['value'] : $hostInfo = 'twitter';

array_key_exists('app',$job) ? $tapp = $xmlconverter['app'][0]['value'] : $tapp = 'DW status';

$txml = getHostSysDsn($context, $hostInfo, "$tapp");

if (!$txml) return FALSE;

$curl_config = array(

        'adapter' => 'Zend\Http\Client\Adapter\Curl',

        'curloptions' => array(

            CURLOPT_SSL_VERIFYHOST => false,

            CURLOPT_SSL_VERIFYPEER => false));

           

$twitter_config = array(

    'access_token' => array(

        'token'  => $txml['Accesstoken'],

        'secret' => $txml['Accesstokensecret'],

    ),

    'oauth_options' => array(

        'consumerKey' => $txml['Consumerkey'],

        'consumerSecret' => $txml['Consumersecret'],

    ),

    'http_client_options' =>$curl_config

);

//var_dump($twitter_config);

$twitter = new ZendService\Twitter\Twitter($twitter_config);

$twitter->statuses->update('Hello World!');

The last line blows up with an error code 400 from Twitter, I have no idea why.

(In all fairness I should link to a post I wrote a long time ago with a cool example using  Zend Framwork1 at the end  .)

I do not think ZF2 is bad , it’s just too complex for me, and this often the case with OO systems. OO invites you to go complex over time. But this does not have to be, in this particular case I found an OO gem , A very simple class for posting on Twitter (the author  David Grudl  claims you can do any Twitter task from this class). Using David’s OO class I only had to write this code:

$message = trim(substr($job['message'][0]['value'],0,140));

$log->logit('Info',"Enter script=$scriptName message=$message");

require_once '/home/tooljn/TWITTER/twitter-php-master/src/twitter.class.php';

array_key_exists('host',$job) ? $hostInfo = $xmlconverter['host'][0]['value'] : $hostInfo = 'twitter';

array_key_exists('app',$job) ? $tapp = $xmlconverter['app'][0]['value'] : $tapp = 'DW status';

$txml = getHostSysDsn($context, $hostInfo, "$tapp");

if (!$txml) return FALSE;

$twitter = new Twitter($txml['Consumerkey'], $txml['Consumersecret'], $txml['Accesstoken'], $txml['Accesstokensecret']);

try {

        $tweet = $twitter->send("$message");

        $_RESULT = TRUE;

} catch (TwitterException $e) {

        $log->logit('Errror', $e->getMessage());

}

return $_RESULT;

And the code posted on Twitter without any fuzz on first attempt. The code was run from this job :

<?xml version='1.0' encoding='UTF-8' standalone='yes'?>

<schedule mustcomplete='yes' logmsg='This is a Twitter test'>

   <job name='twittaMsg' type='script' pgm='twitter01.php'>

      <iconv input='UTF-8' output='UTF-8' internal='UTF-8'/>

      <message>@TODAY-@NOW Hello Twitter</message>

  </job>

</schedule>