2021-07-16

IDRAC to the rescue

 Yesterday was abit hectic, while I was working on some Data Warehouse virtual servers, the database server went bananas, I had lots of server terminals up on my laptop and I shutdown one virtual server to change it's configuration, or so I thought I shutdown the physical host server, all console frames went black and I could not connect to the physical host. I then  to launch the virtual console of the IDRAC, but the connection was dropped probably due to some security policy😠 So there I was at my home without my car (a son had borrowed it for the day), 45 minutes of public transport the hottest day in the year was not very tempting. while pondring on my alternatives I saw the power ON/OFF toggle button on the IDRAC menu. What happens if I press the toggle twice? First power OFF the server and then power ON the server. That should in theory start the server, so I pushed the toggle twice and waited for 15 minutes.

Lo and Behold, the physical host were up and running and so where all virtual servers. Sometimes, very seldom I do admit, things work out exactly as they should. This was one of these rare occations,

Database server running wild

 Yesterday the hottest day so far this summer 32C, the Data Warehouse MySQL server went haywire. Conections where dropped and database tables where partially unavailable this is a very heavy used server no time for deep debugging, so I restarted MySQL and all was normal again, for a few hours, then it really went crazy, now the entire server was affected not only MySQL was uncontactable but the network as well and I could see how new MySQL daemons tried to start up every ten seconds or so. this time I noticed Bacula, ClamAV and Symantec rtvscand was very active at the same time on production prime time, this is probably not a good idea. I never seen anything like this in the twenty years The Data Warehouse been up and running. This time I restarted the database server and now it seems to be OK again.

I have asked the guys responsible for the Data Warehouse to revise the schedules for ClamAV and rtvscan. Why both these daemons are used I do not know. A poolicy of supporters and belts?