12D: 11/1/13

2013-11-30

PHP pcntl threads, Intel E5 VS2 & check-MK

During the writing of a series of posts on parallel scheduling of workflows in PHP, we have migrated an ETL application from my old physical server with an Intel I7 8 core processor to a virtual server in a Dell R720XD with a XEON E5-2609 4 core processor, and the ETL virtual server is assigned 2 CPUs. If you look at the CPU consumption within the ETL server it does not look very alarming.

You can see the server is working hard, but not that hard. I knew the E5-2609 was a weak processor for the load, but I figured it wouldn’t matter since the hard work is background processes run at night. If the jobs takes a bit longer no one would ever notice. But I was wrong, and I’m a bit embarrassed I should have known better. When you stress any kind of system things starts to happen, bad things. Computers, operating systems and programs are no exceptions.

I fork threads by PHP pcntl function and they failed random and silently in the new server. It is hell to debug these kind of failures where and why does failures occur, the root cause might not even be in the server since most threads communicate with remote ERP systems the prolonged execution time might trigger erratic behavior in those ERP systems or the communications layer or in the operating system or in … there are many many reasons parallel processes can go wrong, many more than single threaded processes. And I have to admit I run this in in PHP 5.6 RC2, yes I know you should not run production on RC versions, but that's just the way I am, early adopter and a bit lazy. During the Christmas holiday I will upgrade to a better PHP version, it’s not an easy task since I have to compile and link in some plugins, and do a lot of testing before I can switch. As a temporary circumvention we have moved the ‘biggest’ jobs back to the old server. Next week I will buy a new processor to the server. The good thing is I can have a much better processor than I could when I bought the server, now I choose between Intel XEON E5-2660 v2 and XEON E5-2670 v2, both of them are monster performers compared with the original E5-2609. The most important thing is the whopping 20 CPU’s these new processors can muster with hyper-threading. If I assign 10 processors to my ETL server, it will be more than sufficient. Then there is an interesting question, can I do the operation myself? I’m pretty experienced as a computer engineer, but I never tinkered with a rack server before. I’m not even sure I know how to get the server out of the rack. Anyway I do hope the thread failures will go away with the new processor. Debugging parallel processing is hard, swapping a processor is hopefully a clean simple operation.

The pictures above are from Mathias Kettner's check-MK built on top of Nagios. Really neat.

2013-11-29

PHP parallel job scheduling - 5

In the previous post on Parallel processing of workflows I wrote about nested jobs and how they can be used to express complex parallel job patterns. Nested jobs are created on the fly while the schedule is executing, making it possible to create jobs based on conditions not existing when the schedule starts. Job templates creates or casts jobs when the schedule start executing. Nested jobs and job templates are the two sides of the same coin, dynamic job creation. Like nested jobs, job templates can be used to express complex parallel job structures.

A job template can be driven by a template iterator, the same way a job can be driven by a job iterator.

Let’s say you have a number of IBM BPCS ERP systems spread around the globe, and you want to extract data from them at different times during the day, with different intervals etc. And you have set up a framework of rules for this in a database. Then you can set up a job template and generate jobs from it with the help of a template iterator like this:

This is a complete ETL application, extracting information from a number of BPCS systems spread across the world. The template consists of an iterator and template jobs. The template iterator <forevery> is created from MySQL database framework of rules and for each row a complete set of jobs is created with all @tags substituted. To speed up the extraction we run all extractions in parallel ( see the top job in the template ).

I hope you can see this is elegant, succinct and parallel.

You can find the last post in this serie here .

This is the PHP code that connects to IBM ISeries DB2 and extracts data and saves it in a Mysql database. I run this from a an instance of my PHP job scheduler in MS Windows server since I (and many with me) find it impossible to connect to DB2 Iseries from a Linux environment.

<?php

/**

* A PDO executor - dynamically included in {@link execScript()}

* This code is intended for communication with external database managers via PHP Database Object layer.

* The primary purpose is to copy data from the external DBMS into our own DBMS (Mysql).

* Example, Copy 'schedule' table/wiew from external DBMS 'DB2', to our database 'test':

* <job name='ajob' type='script' data='pdo.php'> <br>

* <pdo>  <br>

* <sql>select * from DB.schedule;</sql> <br>

* <sqlconverter> <br>

* <name>pdoconverter_loadfile01.php</name> <br>

* <target>copy0</target> <br>

* <autoload>replace</autoload><database>test</database><truncate>yes</truncate><table>schedule</table> <br>

</sqlconverter> <br>

* </pdo> <br>

* <script> <br>

* <sql><file>copy0.SQL</file></sql> <br>

* </script> <br>

* </job> <br>

* The <dsn> xml tag, points to a named entry in custom/pdo.xml. You can specify a default dsn in

* the context.

* You specify a <pdo></pdo> block for communication with the external DBMS, and an

* optional <script></script> block for communication with the native mysql DBMS.

* You are not restricted to this example, You can do about anything inside the <pdo> tag you can do

* inside a sql <script> tag, given the restrictions the pdo DBMS driver implies.

* Note! you can write your own import sql inside the <script>, you are not restricted to the

* {@link pdoconverter_loadfile01.php autoload} facility.

* Processing:

* 1 connect to the external database manager by sending the <pdo> block to {@link execPdo()}

* 2 connect to internal dbms by sending the <script> block to {@link execSql()}

* @author Lasse Johansson <lars.a.johansson@se.atlascopco.com>

* @package adac

* @subpackage utillity

* @return bool $_RETURN This is what the calling execSchedule checks at return not the returned boolean

file_put_contents('mydriver',

serialize(array(0 =>array(

'scheduleid'=>$schedule['_scheduleid'],

'jobid' =>$job['_jobid'],

'odbcmsg' =>'A serious error has occured, contact your baker'))));

$rows = array();

$zlogsql = $job['loginit'][0]['value'];

$logsqls = explode("\n",$zlogsql);

foreach($logsqls as &$logsql){$logsql = trim($logsql," ;\t\n\r");}

$logsql = implode(' ',$logsqls);

// print_r($logsql);

$mysql_cake = connect2mysql($context);

$OK = execQuery ($mysql_cake, $logsql, $rows);

if(!$OK) return FALSE;

$mysql_cake->close();

$stmts = FALSE;

$queries = array();

// if (array_key_exists('_sqlarray', $job)) $stmts = $job['_sqlarray'];

if (!$stmts and array_key_exists('script', $job)) {

if (array_key_exists('sql', $job['script'])) $stmts=$job['script']['sql'];

}

if(!$stmts and array_key_exists('sql', $job)) $stmts = $job['sql'];

if ($stmts == FALSE) return FALSE;

print_r($queries);

foreach($stmts as $key => $stmt){

$queries = array_merge($queries, parseSSV($stmt['value']));

}

foreach($queries as &$query){$query = trim($query," ;\t\n\r");}

print_r($queries);

$sqllog = new log("logfile=ODBClog.txt onscreen=no mode=all");

$log->logit('Note',"Environment and input directory are OK, commencing SQL process, details see ODBClog.txt");

$sqllog->logit('Note',"Here we go...");

// return $queries;

$hostSys = 'winodbc';

$odbcParm = getSysContactInfo($context,$schedule,$job,$hostSys);

print_r($odbcParm);

$odbcName = $odbcParm['name'];

if ($odbc = odbc_connect( $odbcName,$odbcParm['user'],$odbcParm['passwd'])) {

$log->logit('Info',"Could connect to odbc driver $odbcName");

}

else {

$odbcmsg = odbc_error();

$odbcmsg = "ODBC Driver Connection login timed out., SQL state " . $odbcmsg;

$log->logit('Failed',"Could not connect to odbc driver " . $odbcName);

$log->logit('Failed', $odbcmsg);

$sqllog->logit('Note',$odbcmsg);

file_put_contents('mydriver',

serialize(array(0 =>array(

'scheduleid'=>$schedule['_scheduleid'],

'jobid' =>$job['_jobid'],

'odbcmsg' =>$odbcmsg))));

return FALSE;

}

$returncodes = array();

for($i= 0; $i < count($queries); $i++) {

$Xquery = $queries[$i];

// print "$i $Xquery\n";

$odbcmsg = 'OK, no problems';

if (!$result = odbc_exec($odbc, $Xquery, 1)){

if(odbc_error()== 'S1090'){

$odbcmsg = 'OK, No data to FETCH, S1090';

$sqllog->logit('Note',"$odbcmsg ");

$log->logit('Note',"$odbcmsg ");

}else {

$odbcmsg = odbc_errormsg($odbc);

$sqllog->logit('Failed',"$odbcmsg ");

$log->logit('Failed',"$odbcmsg ");

return FALSE;

}

if($result){

if(! odbc_num_fields($result)) continue;

$sqlconverterDir = $context['sqlconverter'];

$sqlconverter = $job['sqlconverter'][0]['name'][0]['value'];

if($sqlconverter == '') $sqlconverter= 'sqlconverter_odbcdefault.php';

$sqltarget = $job['sqlconverter'][0]['target'][0]['value'];

if ($sqltarget == '') $sqltarget='driver0';

$sqllog->logit('Note',"Loading sqlconverter=$dir$sqlconverter");

/**

* This includes an SQL result set converter. <br>

* The converter is specified in the job and included from the SQLconverter library.<br>

* The converter uses $odbc handle and stores the result in $sqltarget.<br>

* See {@link sqlconverter_default.php The default SQL result converter} for details.

include("$sqlconverterDir"."$sqlconverter");

// $result= close();

}

$odbc = odbc_close($odbc);

file_put_contents('mydriver',

serialize(array(0 =>array(

'scheduleid'=>$schedule['_scheduleid'],

'jobid' =>$job['_jobid'],

'odbcmsg' =>$odbcmsg))));

$zlogsql = $job['logfetch'][0]['value'];

$logsqls = explode("\n",$zlogsql);

foreach($logsqls as &$logsql){$logsql = trim($logsql," ;\t\n\r");}

$logsql = implode(' ',$logsqls);

// print_r($logsql);

$rows = array();

$mysql_cake = connect2mysql($context);

$OK = execQuery ($mysql_cake, $logsql, $rows);

if(!$OK) return FALSE;

$mysql_cake->close();

$_RESULT = TRUE;

return TRUE;

2013-11-23

Bad English and dyslectic fingers.

Documentation is one purpose of this blog. I use some posts myself and then the bad/funny English and my dyslectic fingers become evident. Most of times I do not care since it's just bad English and I do understand the meaning of the posts.

I'm a terrible typist. I write excruciatingly slow with my left index finger often hitting the wrong key, leading to many errors. I often miss out words since my typing is slower than my thoughts.

This morning I found this:

XML is a markup language that defines a set of rules for encoding documents, it is not a Turing-complete language per se and it is very hard to express and interpret normal programing logic in xml if you do normal xml parsing and that is why I use xml.

This is completely up the wall. I use XML since it is very easy to parse with computer programs (in this case PHP has an easy to use XML parser). I do not use XML because it's hard to express normal programming logic, and the fact it is not a Turing complete language. I use XML despite it's deficiencies as a computer programming language. I would not missed this if I had written this in Swedish.

And I think XML is not Turing complete, I do not know. I haven't really checked.

I have read most of Turing's published writings on computers and it's fascinating to realize most of it, is so obvious today. I constantly had to remind myself he was breaking new grounds just sixty years ago, when he is nagging page up and down about coding 'gotos' on paper tape. Turing was a master mind, one of the great spirits of the twentieth century. If he qualify into the same league as Aristotle, Newton, Darwin, Einstein I'm not to judge, but the importance of his work cannot be overestimated. And he was a brilliant programmer. It's almost scary to read about his insights in computing and predictions about the future of computers.

I end this post with another blooper I found this morning:

NO the job is not long, it is a long running job.
NO the schedule is not spinning, the exit action is spinning off a schedule.
You can probably spot odd language constructs in this post too.

2013-11-18

PHP parallel job scheduling - 4

In the previous post on Parallel processing of Workflows , I introduced job iterators as a tool for splitting up a job (workflow step) into smaller pieces and parallel process them. In this post I describe iterator chunking and piggyback iterators.

If a job consists of repetitive task like producing a mail to all Vendors or extracting information for all Parts from an external ERP system you can create a job iterator and run a job once for each row in the job iterator like the example in the previous post. By cutting up a job into smaller pieces and parallel process them you may cut down wall clock time. But when the setup time for such a ‘job-step’ is significant, it may take longer time to process these steps individually even if you parallel process them. We need to balance setup time loss and job-step cycle time to achieve minimum wall clock time for the job. Such production control in the real world can be very complex with loads of parameters to consider, here most parameters are fixed or given, and the target minimum wall clock time is simple to monitor. Better still we don’t have to find the very optimum either, just about is more than good enough. It’s cheaper to buy more memory, bandwidth etc, than to squeeze out the last microsecond of your computers, just about will do fine.

An example; to extract data for all materials in a workshop from an ERP system you need to log in to the ERP system, run extraction transaction(s) for each material and lastly log out from the ERP system. Most likely log in and out of the ERP system will take longer time than extract the data for one material. You also have to consider how many tasks you can throw at the source system at one time. And how long will the source system let a single extraction session run, there is always a transaction time limit, when the limit expires you are kicked out.

These are the parameters we have to consider:

Setup time - log in and log log out
Process time - extract information for one part
Number of simultaneous tasks - sessions you’re allowed to run in parallel
Session time limit - how long one session can live

The very reason for cutting up the job is to minimize wall clock time, so optimal batch size(s) is the size that allow you to process your workload in shortest time. In my example we have 3600 materials. Let’s say it takes 1 second to log in and out of the ERP system, and 1 second to run the extraction, so run this sequentially in one session it will take about 3601 seconds. We like to shrink this to a minimum and we know the ERP administrator will be upset if we run more than 10 parallel sessions. If we run our 3600 materials in 10 streams it will take us some 720 seconds to complete the task. If we instead cut the 3600 material into 10 batches of 360 material in each it will only take us 361 seconds to process all materials. That’s some improvement! And some considerable less strain on your IT infrastructure (including the ERP system).

Let’s suppose the ERP session time limit is 240 seconds, to keep it simple and being conservative we then create 20 batches with 180 materials in each, then it will takes us some 362 seconds to complete the task. This is an increase we can live with. Real life experience has shown me it is almost this simple to rightsize job-steps. As long as you do not saturate any part of the infrastructure, the parallel execution will be stable and predictable. Often you can rightsize by looking at the watch and run some test shots.

Now to some real testing, it’s always a challenge to measure parallel processes, since the wall clock time depends on so many things, my testing here is done in a production environment and my figures are ‘smoothed’ mean times from many tests.

If we beef up the example from the previous post and run 100 rows instead of 4 and add 1 second of work for each row and 1 setup second, it takes 104 (100 * 1 + 1 + 3) seconds to run the job sequentially:

100 - number of sequentially processed job-steps
1 - second for each work-step
1 - second setup time
3 - overhead for the job scheduler, database manager, network etc.

If we instead of 1 second setup time for the job have a 1 second setup time for each row the job duration time will increase to 204 seconds. This should come as no surprise, 100 * (1 + 1) + 4. This is to prepare for parallel execution of all job-steps.

The modified example from the previous post with setup time for each job, so we can run all in parallel.

When I run all 100 rows in parallel it finish in 4 seconds! 1 * (1+1) + 2.

However if we are splitting up a job that start sessions against an external ERP system, you will not be permitted to start 100 heavy duty parallel sessions you may get away with 5 sessions though.

Running the same job with 5 parallel workers it finish in 42 seconds 20 * (1 + 1) + 2.

So far minimum allowed wall clock time is 42 seconds. But there a problem with this setup. First we run the setup for every row or job-step, this is like log 'in' and 'out' of an external system once for each job-step, we want to chop up the 100 rows into 5 chunks log in and out once for each chunk and the sequential process each row within a chunk. That is more elegant and should be faster, and more important that would give a natural point in the the process to express log in and out logic to the external ERP system (at the beginning and end of each chunk). And this is what iterator chunking and piggyback iterators allow us to do. In this first example:

the job A <forevery> iterator is cut up in 5 chunks, which runs the nested job B 5 times in parallel. In job B we have a piggyback iterator that is given one chunk each from the job A iterator. The piggyback iterator has an <init> to simulate the setup work. With <init> and <exit> you declare setup and cleanup logic for each chunk. This modified example runs in 23 seconds (20 * 1 + 1 + 2). This example uses a nested job, the ‘real’ purpose of nested jobs is to generate jobs on the fly, the example itself is a bit more complicated than it has to be, but it shows how you can build very complex logic for each chunk. A more succinct way to utilize a piggyback operator is to declare a <mypiggyback> iterator directly in the same job as chunked iterator like this:

Now we only have one job but there is a lot under the hood:

the <forevery> creates an iterator with chunks of 20 rows each. and starts 5 parallel workers
the <piggyback> resolves it’s given chunk into rows and then runs the <init>
the job <sql> is executed once for each row in the piggyback iterator
the piggyback <exit> is executed
when all chunks have finished the forevery <exit> is executed

This example runs in 22 seconds (20 * 1 + 1 + 2) slightly less overhead without nested jobs. We started with 104 seconds and ended up with 22 seconds in 5 parallel batches with setup logic for each batch, pretty neat I would say.

And now we are almost at the end of this post. If you study the last example you find it packed with complex logic expressed in relatively simple and succinct XML code. If you can express this more generalized, logical and succinct in XML please tell me.

There is one thing still itching, the final reduction step. Why do a reduction step when we have a potent database manager at hand? Why not throw the reduction onto MySQL:

This example still takes 22 seconds to run, but we do not have to take care of the result, we got it automagically in a nice database table.

These corny examples illustrates the capability of the piggyback operator well and gives some insights what can be achieved by connecting iterators, but they are far from real word examples, this example and this however is real world. Here you have some more cool examples.

In the next thriller post I discuss templates and how to use them for parallel processing.

2013-11-08

Class and Type

I use to say you should learn something new every day. The other day I learned about Type in Object Oriented theory. Type is the template for the interfaces of an object as the Class is the template of an Object. The Class is a mold where you cast Objects, Type is a mold where you cast ?
On the downside of this new knowledge; I'm not sure how I should use this enlightenment but time will tell I suppose. On the upside Erlang doesn't seem so complicated anymore, and I appreciate JSP structured Cobol more now.

2013-11-03

PHP parallel job scheduling - 3

In the last two posts on parallel processing of workflows , PHP parallel job scheduling part 1 and part 2 I have described how to parallel process workflow steps to gain wall clock time. But if an individual step takes to long time the step must be cut up into smaller pieces and processed in parallel. Here I describe how I cut up a too big job into right sized pieces and parallel process those pieces together in my PHP job scheduler. I call workflows schedule and workflow steps job , both schedules and jobs are defined in xml scripts . In this post I introduce iterators which allows to express complex (programming) logic within xml. XML is a markup language that defines a set of rules for encoding documents, it is not a Turing-complete language per se and it is very hard to express and interpret normal programing logic in xml if you do 'standard' xml parsing. (I use xml because it's flexible and there are an abundance of xml parsers available.) One use of the job iterators is to cut up a job into smaller pieces to save time when the job takes too long time to run.

What is too long time? When should we cut a job into smaller pieces? ETL processes that takes longer time than the time period of source data, e.g. if it takes 25 hours to load 24 hours of source data, that it is definitely too long time. But when you hear ‘it is not reasonable to load warehouse transactions more that twice a day, the load time is too long’ and you want those transactions loaded three times a day, you got a case of too long time. Another case of too long time is, when the source system has a shorter time limit of transactions than the extraction time, e.g. it takes 25 minutes to extract parts data from a source system but there is time limit on the extraction transaction of 20 minutes.

In my job scheduler an iterator is a PHP array construct used for iterating a schedule or a job (or a template), the iterator is declared with a <forevery> xml tag. Let’s look at some simple examples:

The job A will run twice since the iterator contains 2 rows, we will execute the SQL statements twice. This is absolutely meaningless but consider this schedule:

The <forevery> iterator contains two rows with one column ‘TABLE’.

The @TABLE carries the TABLE value from the iterator to the sql script. Now we see some functionality in the iterator but it’s still pretty lame, but the next one starts to be useful:

Now we can get a report for all the tables in the test database, it’s a clunky example but it illustrates one function of the <forevery> iterator. The <forevery> iterator contains the result table of the sql query SELECT ‘TABLE_NAME’ …, then we run the job sql query - select ‘@TABLE’... once for each row in the <forevery> iterator. When we have traversed the iterator we will have the report with table-name and rows for all tables in the test database.

So far in this post all examples are sequential, suppose the job sql query in the example above where a long running query then it would nice if we could parallel execute these queries like:

And this is what this example does. This is a very advanced example which is worth study in detail, this parallel example follows the map and reduce pattern . First we map the work to be done by execute the <forevery> sql, here we limit the work to four rows which is spread onto two workers. The parallel=’2’ declarative on <forevery> means release the work onto two workers and queue up the rest and release them one by one when a worker is free. The result of each ‘row-query’ is converted into a file by the <sqlconverter> , the target file is named ‘report’ + the row number of the iterator + ‘.CSV’.

When the iterator is traversed we have 4 result files:

report 0 .CSV
report 1 .CSV
report 2 .CSV
report 3 .CSV

These files are then reduced into the ‘report.CSV’ file by the <exit pgm=reduceDriver_default.php…/> .

This is an extract from the log, it’s less morbid than it looks, workers are only computer processes.

17660 - a <forevery> iterator with 4 rows is created
17660 - submits two iterator rows and states all workers busy, waiting for someone to die
17661 - writes name of home directory name - (1st iterator row starts)
17662 - writes name of home directory name - (2nd iterator row starts)
17663 - writes name of home directory name - (3d iterator row starts!)
17660 - submits the 3d row and states all workers busy, waiting for someone to die
17664 - writes name of home directory name - (4th iterator row starts)

Note since this is parallel execution there is no defined order of the messages; here msg 5 and 6 come in reversed order. If you have read this post so far I really think you should study this example carefully. There are some complex things under the hood. Event driven parallel execution with queue handling expressed in succinct XML parsed and executed by PHP.

And here is the reduced result, one line from each iterator row.

But this is not all, there is more to come . What do you do when the setup time is large? e.g. You have 10000 iterator rows, one row takes 30 seconds to process of those seconds 20 is setup time, then you like to create batches of rows and submit those batches for execution and that is what iterator chunking and piggyback iterators are for . This I describe in the next post.

In part 1 I wrote you should look at this code instead of my code below which shows how I deal with (parallel queues for) job iterators.

<?php

The awap program product is free software;

you can redistribute it and/or modify it under the terms of the

GNU General Public License as published by the Free Software Foundation;

either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but

WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY

or FITNESS FOR A PARTICULAR PURPOSE. */

/ extract of code for parallel execute iterator rows

foreach($driver['rows'] as $driverIndex => $driverRow) { //job iterator

$driverRow['driverIndex'] = $driverIndex;

if (array_key_exists('--showdriver',$context['_startup'])) $log->dumpit('Note','driver row', $driverRow);

if(array_key_exists('wait',$driver) and $driverIndex){

$iterSleep = $driver['wait'];

$log->logit('Note',"Iterator sleeping $iterSleep seconds.");

sleep($driver['wait']);

}

$_RESULT = TRUE; // Posit job goes well

$xlatearray = array_merge($driverRow,$xlateJSC); //array_merge($driverRow,$xlateJSC,$driverRow);

if ($rowDirectory) {

$drvDir = $jobDir.'/drv' . $driverIndex;

mkdir($drvDir);

copyr($jobDir, $drvDir, 'drv');

$jobDir = $drvDir;

$job['_directory'] = $drvDir;

file_put_contents("$drvDir".'/JOB',serialize($job));

$drvDirArray[] = $drvDir;

file_put_contents("$jobDir".'/xlatearray',serialize($xlatearray));

/Problem? In scriptEx1 we translate the job array $ok = array_walk_recursive($job, 'xlateTagsInArray', $xlate);

//Here we proceed with the untranslated $job this may cause challenges in at least driverInitTerm()

//We should probably do the translation here

}

if ($fork) $drvPid = pcntl_fork(); // Fork if possible

if($drvPid == -1) { // who am I?

$log->logit('Error', " I’m forked up! pcntl_fork pid=$drvPid. Abending...");

$_RESULT = FALSE;

return FALSE;

} elseif($drvPid) { // I’m the parent process, control max forks

if (count($pidar) >= $forkmax) {

$opt = 0;

$log->logit('Note',"Max workers=$forkmax, waiting for someone to die");

}

else $opt = WNOHANG;

while (($deadPid = pcntl_wait($status, $opt)) > 0) {

$deadI = $pidar[$deadPid];

$_drvResult = 0; // Posit job ends unsuccessfully.

if(pcntl_wifexited($status)) $_drvResult = pcntl_wexitstatus($status);

if ($_drvResult) $drvsOK++;

else $drvsNotOK++;

unset($pidar[$deadPid]);

$opt = WNOHANG;

}

} elseif($drvPid == 0) { // I’m a newly spawn kid or no fork

if ($fork) $log->setPid(posix_getpid());

$log->logit('Info',"Driver Index=$driverIndex");

//$log->dumpit('Note','XLATE',$xlatearray);

//$log->dumpit('Note','JOB',$job);

switch ("$jobType") {

case 'ftpinput':

case 'sql':

$_RESULT = execSql($jobDir, $xlatearray);

break;

case 'pdo':

$_RESULT = execPdo($jobDir, $xlatearray);

break;

case 'script':

$_RESULT = execScript($jobDir, $xlatearray);

break;

case 'sendmail':

if (array_key_exists('_sqlarray',$job)) $_RESULT = execSql($jobDir, $xlatearray);

if ($_RESULT) $_RESULT = execMail($jobDir, $xlatearray);

break;

case 'function':

$log->deprecated('Note',"use inherent_function instead!");

case 'inherent_function':

$_RESULT = execInherentFunction($jobDir, $xlatearray);

// this must be in sync with scriptEx

break;

case 'dummy':

case '':

case NULL:

break;

default:

$log->logit('Error',"Unknown job type '$jobType' found in schedule $scheduleName");

$_RESULT = FALSE;

}

if ($_RESULT) $_RESULT = execChildSchedule($context,$schedule,$job,$xlatearray);

if ($fork) exit((int) $_RESULT);

} // who am I?

$pidar[$drvPid] = $driverIndex;

$log->reset('mode');

$log->trace('off');

$jobDir = $jobDirX;

if (array_key_exists('until',$driver)){

$exp = $driver['until'];

eval("\$bool = $exp;");

if ($bool){

$log->logit('Note',"Iterator 'until' condition is true intercepting execution");

break;

}

$ok = newFiles($context,$job,$jobDir,$driver,$driverIndex,$jobdirfiles);

} // job iterator