Main Page   Namespace List   Class Hierarchy   Alphabetical List   Compound List   File List   Namespace Members   Compound Members   File Members  

Resource Monitoring Facility Documentation

INTRODUCTION

The Linux Resource Monitor Facility provides a consistent programmatic access to statistics and create threshold, watermark, and leaky bucket monitors for any instrumented subsystem in the Linux operating system.

The architecture of the Resource Monitor defines five primary components:

  1. Consumers - Linux applications that use the programmatic access.
  2. Resource Monitor Library - the interfaces used by Consumers for monitoring access.
  3. Resource Monitor daemon - the runtime process that manages consumer monitoring requests and provides continuous monitoring of statistics.
  4. Subsystem Monitor libraries - the subsystem-specific access methods used by the daemon to query statistic values.
  5. Data Capture libraries - the data capture methods used by the daemon to store statistic values.
The Resource Monitoring Facility is designed to be used by the following types of Consumer Applications:
  1. Autonomous network management agents
    Instrumentation of watermarks and thresholds can use/leverage the RM facility for operating system, driver and application monitors.
  2. Graphical displays
    Data can be obtained from the RM and presented in graphical form.
  3. Availability managers
    Monitors can be established for critical system resources.

DEFINITIONS

COMPONENTS

The Resource Monitoring Facility is delivered in multiple components. The components and their functionality are as follows.
  1. Consumer application headers
  2. Subsystem development headers
  3. Data Capture development headers
  4. Resource Monitor Library
    A library used by C/C++ client applications to access resource statistics and set monitors for those statistics. Library API's are in the ResourceMonitor namespace. The Resource Monitor Library is not thread-safe since the connection to the daemon is shared by all threads. The application is responsible for synchronizing access to this library. C++ applications can use ResourceMonitor::rmCreateConsumerSession and ResourceMonitor::rmCloseConsumerSession to create a different connection for each thread when using ResourceMonitor::RMsubsystem, ResourceMonitor::RMstatistic, and ResourceMonitor::RMmonitor.
  5. Resource Monitor Daemon
    A Linux daemon process that manages monitors on behalf of client (consumer) applications. For the daemon's manual page see resourcemonitord.
  6. Subsystem Libraries
    Dynamically loadable libraries implemented in C++ that provide interfaces for specific subsystems (drivers, kernel resources, applications) with for methods reading, resetting, and for some subsystems monitoring of subsystem statistics. A subsystem library registers its subsystem using SubsystemMonitor::IRMRegistration class methods and is required to provide methods for creating SubsystemMonitor::ISubsystemMonitor and SubsystemMonitor::Statistic classes. When the subsystem supports inline monitors, it is required to provide a method for creating the SubsystemMonitor::Monitor class. Subsystem API's are in the SubsystemMonitor namespace.
  7. Data Capture Libraries
    Dynamically loadable libraries implemented in C++ that provide interfaces for capturing statistical as a statistic is being monitored by the daemon. A data capture library registers its library using StatisticalData::IDataCaptureRegistration class methods and is required to provide methods for creating StatisticalData::IDataCapture classes. Data Capture API's are in the StatisticalData namespace.
  8. Documentation HTML documentation can be read starting with /opt/resourcemon/doc/html/en/index.html. Three man pages are installed in section 3, ResourceMonitorTypes, ResourceMonitor, and SubsystemMonitor.

DEPENDENCIES

QUESTIONS & ANSWERS

EXAMPLE MONITORING CODE

PROGRAMMING STYLE There are two styles of programming with the Resource Monitor facility, an object-oriented style and a function-oriented style. In the object-oriented style you create one or more RM objects and then use object methods to create the various elements of the page. This is a C/C++ example of supporting both object-oriented style and a function-oriented style interface:
  1. For example, using the object oriented style, here is how you create a simple monitor:
    // ***************************************************************************
    //                          CplusplusExample.cpp  -  description
    // This example shows a C++ style setup of a bidirectional threshold monitor
    // required compile options: -I/usr/include/ResourceMonitor
    // required link options: -lResourceMonitor -luuid
    // ***************************************************************************
    #include <ResourceMonitor.h>
    #include <RM_ProcessMonitor.h>
    ResourceMonitor::RMmonitor *CreateVirtualMemoryMonitor(rmValue threshold)
    {
        RMuid ssid = RMuid(RM_PROCESS_UUID_STRING);
        rmMonitorControl                moncontrol;
        rmMonitorConfiguration  monconfig;
    
        monconfig.statisticKey.SubsystemId = ssid.id;
        monconfig.statisticKey.ResourceId = getpid();
        monconfig.statisticKey.StatisticId = STAT_vsize_ID;
        monconfig.monitorType = rmThresholding;
        monconfig.statisticTransform = rmNone;
        monconfig.RMThresholdType = rmThreshold;
        monconfig.RMThresholdValue.rmValue32 = threshold.rmValue32;
        monconfig.RMThresholdCondition = rmValueIsAtOrAbove;
        monconfig.RMThresholdPrecondition = rmNoPrecondition;
        monconfig.RMThresholdEventSeverity = LOG_WARNING;
        monconfig.RMThresholdCancelEventSeverity = LOG_NOTICE;
    
        moncontrol.monitorType = rmThresholding;
        clearMonitorUID(moncontrol);
        clearDataCapture(moncontrol);
        moncontrol.location = rmDaemon;
        moncontrol.monitorId = 0;   // unused for daemon monitors
        moncontrol.monitoringInterval = 0; // monitor until stopped
        moncontrol.monitoringRate = 5;  // check every 5 seconds
        moncontrol.microMonitoringRate = 0; // check every 5 seconds
        moncontrol.RMThresholdTolerance.rmValueU32 = 0; // qualifies what "At" or "equal to" is
        moncontrol.RMThresholdSamples = 2;  // log event after 2 consecutive readings above the threshold
        moncontrol.RMThresholdLoggingRate = 60; // log events no more than once a minute
    
        try {
            ResourceMonitor::RMmonitor *mp = new ResourceMonitor::RMmonitor(monconfig, &moncontrol);
            mp->StartMonitor();
            return mp;
        }
        catch(...)
        {
            cerr << "failed to create CreateVirtualMemoryMonitor" << endl;
        return 0;
        }
    }
  2. For example, using the function oriented style, here is how you create a simple monitor:
    // ***************************************************************************
    //                          Cexample.c  -  description
    // This example shows a C-style setup of a bidirectional threshold monitor
    // required compile options: -I/usr/include/ResourceMonitor
    // required link options: -lResourceMonitor -luuid
    // ***************************************************************************
    #include <ResourceMonitor.h>
    #include <RM_ProcessMonitor.h>
    #include <stdio.h>
    rmHandle CreateVirtualMemoryMonitor(rmValue threshold)
    {
        rmHandle                                handle;
        rmMonitorControl                moncontrol;
        rmMonitorConfiguration  monconfig;
    
        uuid_parse(RM_PROCESS_UUID_STRING, monconfig.statisticKey.SubsystemId);
        monconfig.statisticKey.ResourceId = getpid();
        monconfig.statisticKey.StatisticId = STAT_vsize_ID;
        monconfig.monitorType = rmThresholding;
        monconfig.statisticTransform = rmNone;
        monconfig.RMThresholdType = rmThreshold;
        monconfig.RMThresholdValue.rmValue32 = threshold.rmValueU32;
        monconfig.RMThresholdCondition = rmValueIsAtOrAbove;
        monconfig.RMThresholdPrecondition = rmNoPrecondition;
        monconfig.RMThresholdEventSeverity = LOG_WARNING;
        monconfig.RMThresholdCancelEventSeverity = LOG_NOTICE;
    
        moncontrol.monitorType = rmThresholding;
        clearMonitorUID(moncontrol);
        clearDataCapture(moncontrol);
        moncontrol.location = rmDaemon;
        moncontrol.monitoringInterval = 0; // monitor until stopped
        moncontrol.monitoringRate = 5;  // check every 5 seconds
        moncontrol.microMonitoringRate = 0; // check every 5 seconds
        moncontrol.RMThresholdTolerance.rmValue32 = 0;  // qualifies what "At" or "NotAt" is
        moncontrol.RMThresholdSamples = 2;  // log event after 2 consecutive readings above the threshold
        moncontrol.RMThresholdLoggingRate = 60; // log events no more than once a minute
    
        handle = rmCreateMonitor( &monconfig, &moncontrol);
        if( handle == 0 )
            perror("rmCreateMonitor");
        else
            rmStartMonitor(handle);
        return handle;
    }
  3. Creating a monitor and setting up its event handler. See MonitoringEventSchema.h for the format of event data.
    // ***************************************************************************
    //                          EventingExample.c  -  description
    // required compile options: -I/usr/include/ResourceMonitor
    // required link options: -lResourceMonitor -luuid -levl -lnsl -lpthread
    // ***************************************************************************
    
    #include <ResourceMonitor.h>
    #include <RM_ProcessMonitor.h>
    #include <stdio.h>
    #include <stdlib.h>
    #define SMALLBUFSIZE    1024
    #include <string.h>
    int eventReceived = 0;
    
    rmHandle CreateVirtualMemoryMonitor(rmValue threshold)
    {
        rmHandle                                handle;
        rmMonitorControl                moncontrol;
        rmMonitorConfiguration  monconfig;
    
        uuid_parse(RM_PROCESS_UUID_STRING, monconfig.statisticKey.SubsystemId);
        monconfig.statisticKey.ResourceId = getpid();
        monconfig.statisticKey.StatisticId = STAT_vsize_ID;
        monconfig.monitorType = rmThresholding;
        monconfig.statisticTransform = rmNone;
        monconfig.RMThresholdType = rmThreshold;
        monconfig.RMThresholdValue.rmValueU32 = threshold.rmValueU32;
        monconfig.RMThresholdCondition = rmValueIsAtOrAbove;
        monconfig.RMThresholdPrecondition = rmNoPrecondition;
        monconfig.RMThresholdEventSeverity = LOG_WARNING;
        monconfig.RMThresholdCancelEventSeverity = LOG_NOTICE;
    
        moncontrol.monitorType = rmThresholding;
        clearMonitorUID(moncontrol);
        clearDataCapture(moncontrol);
        moncontrol.location = rmDaemon;
        moncontrol.monitoringInterval = 0; // monitor until stopped
        moncontrol.monitoringRate = 5;  // check every 5 seconds
        moncontrol.microMonitoringRate = 0; // check every 5 seconds
        moncontrol.RMThresholdTolerance.rmValueU32 = 0; // qualifies what "At" or "NotAt" is
        moncontrol.RMThresholdSamples = 2;  // log event after 2 consecutive readings above the threshold
        moncontrol.RMThresholdLoggingRate = 60; // log events no more than once a minute
    
        handle = rmCreateMonitor( &monconfig, &moncontrol);
        if( handle == 0 )
            perror("rmCreateMonitor");
        else
            rmStartMonitor(handle);
        return handle;
    }
    
    //
    // The call to posix_log_notify_add registered a query
    // string and this function as the callback. If the query
    // evaluates to true for any incoming event, then this
    // function will be called. Normal application execution
    // continues after this function returns.
    //
    
    void catchNotificationEvent(int signo, siginfo_t *info, void *ignored)
    {
        posix_log_recid_t recid;
        rmHandle handle = (rmHandle)info->si_int;
        rmMonitorState state;
    
        rmGetMonitorState( handle, &state);
        // Get the recid of the event that caused this function call.
        // IBM : we change our posix API - see update
        posix_log_siginfo_getrecid(info, ignored, &recid);
        printf(" signal=%d from event: handle=%d, Record ID=%d\n",
                                         info->si_signo, info->si_int, recid);
        rmStopMonitor(handle);
        rmDeleteMonitor(handle);
        eventReceived = 1;
    }
    
    int TestVirtualMemoryEvent()
    {
        char query_string[SMALLBUFSIZE], error_string[SMALLBUFSIZE];
        int once = 0;
        posix_log_notify_t notifyThresholdHandle;    //  Remember the monitor's query_handle and its query
        posix_log_query_t  thresholdQuery;    //  so we can stuff them into the Monitorset to use later.
        struct sigaction SigRTAction;          //  IBM's way of defining the signal handler...
        struct sigevent mynotification;
        char monitor_string[MAX_GUID_STRLEN];
        rmHandle handle = CreateVirtualMemoryMonitor( (rmValue)(u_int32_t)1, 0);
        rmMonitorControl control;
    
        if( handle == 0 ) return 1;
    
      //  Initialize thresholdNotification with the local function name which is to be the callback function
      //  and set the desired POSIX signal value. If your application only needs one callback registration
      //  then use SIGRTMIN+1. For each additional callback function, you should use different signal values
      //  between SIGRTMIN to SIGRTMAX (a range of 32 values).
    
        mynotification.sigev_value.sival_int = (int)handle;   // pass monitor handle to event handler
        mynotification.sigev_notify = SIGEV_SIGNAL;
        mynotification.sigev_signo = SIGRTMIN + 1;  // needs to match the sigaction number below
    
        (void) memset(&SigRTAction, 0, sizeof(SigRTAction));
        SigRTAction.sa_flags = SA_SIGINFO | SA_RESTART;
        SigRTAction.sa_handler = (void (*) (int)) catchNotificationEvent;  // to use catchNotificationEvent() for this Monitor's events.
    
        if (sigaction( SIGRTMIN + 1,                                                        //  signo_offset can typically default to 0, if
                    &SigRTAction, NULL) < 0) {                //  you are only using a single listener.
          perror("sigaction failed for new SIGRTMIN.");
            return 1;
        }
        // Initialize m_notification with the local function name which is
        // to be the callback function and set the desired POSIX signal
        // value. If your application only needs one callback registration
        // then use SIGRTMIN+1. For each additional callback function, you
        // should use different signal values between SIGRTMIN to SIGRTMAX
        // (a range of 32 values).
    
        // A query string will typically consist of a monitor's RID
        // e.g. "data contains \"Monitor_ID=cc63f3bb-d74b-41a0-ab46-5d81579c9b11\""
        rmGetMonitorControl(handle, &control);
        uuid_unparse(control.uid, monitor_string);
        snprintf(query_string,SMALLBUFSIZE,"data contains \"Monitor_ID=%s\" && severity=WARNING",monitor_string);
        // Register the query string and check it for basic syntax req.
        if(posix_log_query_create( query_string, POSIX_LOG_PRPS_NOTIFY,
                                 &thresholdQuery, error_string, SMALLBUFSIZE) != 0) {
            perror("could not create threshold query!");
            return 1;
        }
    
        if(posix_log_notify_add(&thresholdQuery, &mynotification, once, &notifyThresholdHandle)!=0) {
            perror("Threshold posix_log_notify_add() failed");
            posix_log_query_destroy(&thresholdQuery);
            return 1;
        }
    
        printf("Registered Threshold Notification with POSIX Event logger (handle=%d)\n",
                notifyThresholdHandle);
    
        while(eventReceived==0) {
            sleep(1);
            if( ++once > 10 )
                return 1;
        }
        return 0;
    }
    
    int main(int argc, char *argv[])
    {
        if( TestVirtualMemoryEvent() == 0 )
            exit( EXIT_SUCCESS );
        exit(1);
    }

NAME

resourcemonitord - Linux system resource monitoring facility daemon.

SYNOPSIS

resourcemonitord [ -t rmtab file ] [ -p persist file ] [ -s socket ] [ -r Registry Refresh Rate ] [ -m Self Monitoring Rate ] [ -c daemon CPU limit ] [ -l daemon resident memory limit ] [ -d ]

DESCRIPTION

resourcemonitord provides resource monitoring capability for system statistics made available through subsystem libraries. Use of Unix domain sockets limits monitoring to local resources, i.e. the same node.

OPTIONS

Use /etc/sysconfig/resourcemonitor to set resourcemonitord options.

-t resource subsystem file

Subsystem plug-ins are registered with the daemon in this file. Each entry in the file contains the subsystem's universally unique identifier(uid) and the path to the shared library of the plug-in. The default location of this file is /etc/opt/resourcemon/rmtab.

-p persistant monitor file

Upon restart, the daemon allows monitors to be recreated automatically from a persistent store. The file rmMonitors contains information necessary to recreate monitors. The default location of this file is /var/opt/resourcemon.

-s socket

This option is unsupported on this release. The default socket is /var/run/.RM-socket.

-r Registry Refresh Rate

The daemon periodically checks for registration changes in the subsystems and for changes in the rmtab file which contains the library locations for each subsystem. The daemon dynamically loads and unloads subsystem libraries as changes are made to this file. The default location of rmtab is /etc/opt/resourcemon. This refresh rate is frequency in seconds that the daemon checks the file for changes. The default refresh rate is 60 seconds.

-m Self Monitoring Rate

The daemon throttles itself when its CPU utilization exceeds the configured limit. When in the throttled state,EBUSY is returned for requests to create new monitors or to start existing monitors. The Self Monitoring Rate is the frequency the CPU and memory utilization is checked. The default value is 15 seconds.

-c daemon CPU limit

This is the daemon's percent CPU utilization limit. 74,000 daemon based monitor on 27 statistics use about 8% on a dual CPU 933 MZ Pentium III system. The default value is 5 percent.

-l daemon resident memory limit

This is the daemon's resident memory limit in megabytes. 74,000 daemon based monitors use about 50 MB. The malloc/free library tends to fragment memory, so deleteing monitors may not return memory to the system. Therfore, the daemon may remain throttled once this limit is reached. The default value is 25 MB.

-d

This option enables debugging of the daemon. With this option, resourcemonitord does not make itself a daemon or close its standard out and error file descriptors.

SIGNALS

resourcemonitord reacts to a set of signals. You may easily send a signal to resourcemonitord using the following:
 kill -SIGNAL `cat /var/run/resourcemonitord.pid`

SIGHUP

SIGHUP causes resourcemonitord to exit normally.

SIGINT

SIGINT causes resourcemonitord to exit normally.

SIGTTIN

SIGTTIN causes the resource subsytem file to be read immediately. This facilitates the installation of new subsystem plug-ins.

SIGTTOU

SIGTTOU generates a file detailing some of the internal state of the daemon. The location of this file is /var/opt/resourcemon/rmState.

DEBUGGING

When debugging is turned on using -d option then resourcemonitord will be very verbose, writing much of what it does on stdout.

COLLABORATORS

andrew.r.cress@intel.com, todd.c.davis@intel.com, bo.z.li@intel.com, perry.g.vincent@intel.com, and sonic.zhang@intel.com


Generated at Sat Sep 7 21:21:07 2002 for Resource Monitoring Facility by doxygen1.2.8.1 written by Dimitri van Heesch, © 1997-2001