Topic: E-Smith Monitor mod's...

[Gary Parker]

Hi all,
   well, after getting my SMP box up and running I find myself wanting to know what it's doing. In the past I've used the rather groovy E-Smith Monitor package from Juan Jesús Prieto Tapia and have been well impressed with it (it runs just fine on 5.1.2 although their website [http://modesmith.dogbox.homeip.net/] doesn't seem to have been updated since 4.1.2).

What I'd like to do is monitor both CPUs in the one graph in a similar manner to the way that it plots In and Out on the same graph for the network stats. I know it uses rrdtool and so shouldn't be (too) difficult to mod but I have no experience with rrdtool and so would appreciate some pointers as to where to look for the config files etc.

Gary

[Jaime Nebrera]

Hi Gari,

  Well, http://modesmith.dogbox.homeip.net/ is not our web page )) They have just kindly left us some space to host the files.

  I don't understand well what you want to achieve. A separate line for each CPU?

  I don't know if this is possible to do, but surely a look into sysmonitor (the data gathering script) and monitor (the web panel) will help. Sadly, we don't have a dual motherboard to test / develop this

  See you

[Richard Bruce]

I'd like this too, but I'm in a similar position re development, I have bags of enthusiasm, but exactly zero experience! I do however have access to a development twin processor Dell Poweredge 2400 (currently running 5.1.2 to see if I can break it ) and am willing to test anything anyone can develop. Not much help, but it may be useful to someone.

Thanks again to Jaime for a fantastic add-on!

Richard Bruce
rbuce@cfpress.co.uk

[Gary Parker]

Thanks Richard and Jaime!
   I've had a look at the sysmonitor script and have figured out what I need to do to record two seperate usage values, one for each processor, and also how to draw the grpah from that data. One small problem, however, Jaime and Juan have used vmstat to get the cpu usage value, more specifically they take the system idle value and subtract it from 100 to get an artificial value for user + system processor usage. Unfortunately vmstat doesn't (can't) list this information for disrete cpu's. Does anyone know of a program that will do this, in a manner similar to top? If need be I'll use the batch output from top and mong it to get the values I need but there must be an easier way than that...

As soon as there's something to test, Richard, I'll let you know...

Gary

[Gary Parker]

Well...after burying my head in the perl cookbook and rrdtool manual for the last few days I've got something to show for it

grab

  http://www.witewulf.ath.cx/e-smith-monitor/monitor

and

  http://www.witewulf.ath.cx/e-smith-monitor/sysmonitor

then do 'service sysmonitor stop' and copy them to

  /etc/e-smith/web/functions/

and

  /usr/sbin/

respectively, followed by 'service sysmonitor start' and you should shortly find yourself with a CPU graph looking similar to the network ones, except the blue line is for CPU0 and the red line for CPU1.

The mods are by no means elegant, the worst being a nasty hack to get the idle values off the individual processors using top and a regexp:-

        {
        $_ = top b n 2 d 5 | grep CPU[0,1];
        ($getidle0) = /CPU0.*\n.*\nCPU0.*nice,\s+(\d+)/;
        ($getidle1) = /CPU1.*\n.*\nCPU1.*nice,\s+(\d+)/;
        $res0 = 100-$getidle0;
        $res1 = 100-$getidle1;
        /usr/bin/rrdtool update /var/lib/rrd/usedcpusmp.rrd N:$res0:$res1;
        exit ;
        }

I'm running top for two iterations as it incurs an 8% or so processor load when it itself actually loads, so the two iterations, 5 seconds apart, should let things settle down, we only record the second set of values. For my mind, top puts an unacceptable load on the system for use as a monitoring tool, especially as it's being run once every minute. If anyone knows of another method of finding the idle values for individual CPUs please let me know!

Be aware that this mod will only work if your bios/mobo numbers your CPUs 0 and 1. From reading the linux SMP FAQ I believe some mobos use weird numbering schemes for their CPUs.

Also note that this will only work on dual SMP boxen, *not* uni-processor systems, next step is to try and modify it further to autodetect the number of CPUs (both uni and/or SMP) present in a system and draw the graphs with respect to that number.

Please let me know what you think, if you have any suggestions and if it works for you!

Gary

[Jaime Nebrera]

Hi Gari,

  I'm one of the developers of e-smith monitor. This seems as a great work !! Well done. OK, once this gets a little stablished we could add it to the official monitor release.

  If you recall we used sysstat to gather system data. Maybe (just maybe) you could use it to collect the CPU (SMP system) data. It woul be great if the script detected if the system was dual or monoprocessor and the used the standard way or the sysstat way.

  Good job again. Keep us informed if you have progress with this.

  Regards.

[Gary Parker]

Mmm....just found the perfect peice of kit, mpstat, vmstat modified to report on SMP machines.

Check it out here - http://www.mindspring.com/~joeja/programs.html#C-Gtk

Will try it out tomorrow.....my bed is calling

Gary

[Gary Parker]

Well, I've posted a new version of the sysmonitor script that uses mpstat and it uses a lot less cpu than top, which is a result...

Get the script from http://www.witewulf.ath.cx/e-smith-monitor/

You'll also need the mpstat program in your path somewhere, get the whole distro here http://www.mindspring.com/~joeja/programs.html#C-Gtk or just grab the executable from my site (same address as above).

I'll start looking at the auto-detection soon....

You give this a try yet Richard?

Gary

[Gary Parker]

Right...
   more new versions!

Grab the monitor and sysmonitor scripts from the address in my previous post.

Make sure mpstat is in your path somewhere

'service sysmonitor stop'

cp the new sysmonitor script into /usr/sbin/

rm -rf /var/lib/rrd/usedcpu*

'service sysmonitor start'

cp the new monitor script into /etc/e-smith/web/functions/

Check it out!

sysmonitor should now autodetect how many CPUs you have (1,2 or 4) and create and start populating an rrdDB accordingly, this is why you need to delete any existing rrdDBs.

Again, the monitor script should also detect how many CPUs you have and plot the data accordingly.

Please note, this has only been tested on a single and dual processor system, I've no idea at all if the 4-way logging works as I've no access to that sort of hardware.

Jaime, as far as I'm concerned this is final now so please feel free to review it with regards to adding it to your official release if you want to.

Gary

[Richard Bruce]

Hi,
I'll be trying this out over the weekend. Sorry for the delay but I've been ill for a couple of weeks. Will let you know how it works out. thanks for all the work

Richard

[Richard Bruce]

Hmmm,

Just tried to grab the packages from the URL's you gave, but they seem to be unavailable. Any chance you could email it to me?

Ta.

Richard

[Gary Parker]

Hey Richard...bad karma reigns all around today

For reasons too long-whinded to go into here I lost *all* the data on my e-smith box last night, including the mod'd monitor scripts. There's a shiny new ATA100 RAID card sat on my desk waiting to go into the damn machine tonight.

Backups? Nah.....don't believe in 'em....

Although I had it working in the end, I didn't honestly find it to be all that useful because of the way Linux handles job scheduling on SMP systems.

Usage stats were being accurately plotted on a single graph and when running two seperate instances of setiathome, or some similar cpu-hogging process, it would draw two crinkly lines across the top of the graph quite happily. The problem is that Linux has no concept of processor affinity in a smp kernel and it has a habit of moving processes backwards and forwards between cpus if one of them is idling. Consequently, in a multi-processor system with only one high-processor-usage process running it will flit backwards and forwards every few cycles. This leads to the lines being plotted on the graph see-sawing wildly between ~100% and ~0% usage constantly, with the two averaging at the system's overall usage. On systems where there is a constant 'medium' usage of the available processors this should lead to the two lines being plotted roughly the same. Consequently, the graph as drawn by the stock e-smith-monitor is a good representation of what's actually going on.

This is also a pain for processes that make heavy use of L2 cache (like setiathome) as when the process gets moved to a new cpu it loses all the data it had in the L2 and has to get it back from main memory again. Someone was working on a patch and some utils to add processor affinity to the kernel but last time I looked it was dirty to say the least...

I wonder if anyone else downloaded the scripts and kept copies of them that they could mail to you?

Gary

[Gary Parker]

Well Richard...there's been a guy working on processor affinity for a while as far as I can see but it involves patching the kernel and recompiling and all sorts of nasty things like that...

The project's called PSET, check it out here (http://www.hockin.org/~thockin/pset/).

It doesn't seem to have been worked on for quite a while and I've no idea if the guy ever made any attempt to get it into the kernel.

Gary