Electric power

Yesterday the office had a planned power outage for twelve hours. Of course things went wrong.
A decision was made to run the servers on the UPS battery and a diesel generator.
I advised the Data Warehouse guys to shutdown the Data Warehouse servers anyhow. So far
so good, the servers were stopped and when the power was back one guy went to the server room
to start them up. The servers were already running but refused to communicate, the guy called me.
After some thinking I realised there must have been a power failure despite the UPS/Diesel backup
and when the power was back the servers started automatically before network or anything else were
up and running. We powered off and on the servers and Hallelujah! The entire Data Warehouse
started up just as it should except…
Last month I was forced to create a new ETL server extracting data from Bloomberg. That virtual server did not start automatically but that was cool I started that server manually. This morning I did not get the Bloomberg reports, so something must have gone wrong. I tried to Putty into the server it did not work, ping did not work, the f-ng  server refused to communicate, I could enter it via the KVM’s master console, most thing looked normal except I could not find any recent logs from Bloomberg extractions. After some hours of desperate looking around I saw that the IP address to the server was wrong, then I remembered I had cloned the server and created a new production server, I had started the test server. Attention to details have never been my thing. I just had to start the right server and we were back on track again. I suppose this is a price I have to pay for “moving fast”. I usually develop at a high velocity, sometimes more frequently the last years I forget small details as this new server, it’s just temporary until the entire Data Warehouse is moved to the new OpenSuse platform. But of course they had to cut the power in between.  

No comments:

Post a Comment