Date State Author Note Description
31.07.2004 DEVELOPMENT mb created This document is currently growing together with the project.
03.01.2004 DEVELOPMENT mb created This document is currently growing together with the project.

Abstract

TOC

Intend

The intend is to create a useful, but not too simple tool, to accumulate from a large number of machines into a central store to allow easier management.

Scope

The scope of this application is to provide a central database as a basis to make detailed analyzes about server-health, hardware malfunctions and possibly even security analyzis. This means, the application provides the most services in collecting and distributing the events to viewers and the database. Deeper analyzis of the results inside the database is out of the scope of this application but could be build on database reports, triggers and stored procedures. In this arena, this application provides only basics.

Overview

The service application, implemented as a , collects via WMI asynchron queries. For each server to have the events collected, one EventCollector instance is running. The machines to monitor are defined in a xml-based configuration-file. The collected events are redirected to the CollectorService [this is one of the classes in use], which stores the events into different MSMQ queues. One of this queues is to allow fast-forward notification via Remoting, one to store the events into a SQL Server Database and another one is to optionally forward the events to another site. It is configurable, which of these queues will be used.

Another service, also a , is responsible to read the message queue and store the events into a . To be able to store a huge load of events into the , the operation is delayed a bit until a small timespan has gone or a number of events are collected. This amount of events and the timespan are defined in a configuration-file and additionally, is automatically adjusted by watching the performance of MSMQ.

The third reads the queue destined for fast admin notification and distributes the events via Remoting to all clients, which are online. These clients are WinForms controls hosted into .

GOALs

The goals are more-fold:

  • provide a centralized eventstore
  • fast immidiate administrator notification
  • allow cascading architecture to span multiple sites, but have one centralized store
  • high performance
  • possibly no event-loss
  • using only build-in technology
  • based on
  • online management
  • easy configuration through text-files[xml]
  • allowing db to go offline for maintenance without event-loss
  • implemented in true Windows NT Services
  • web-based management of the CollectorService
  • web-based event-view
  • source filtering capability
  • view filtering capability
  • allow to run in console-mode to allow easier debugging
  • allow multiple instances for debugging
  • run from NT4 machines up to Windows 2003 Servers or workstations [XP]
  • online switching to allow extended diagnostic messages to the debugger-interface

Technologies used

  • MS SQL Server
  • MSMQ
  • Remoting
  • client side script
  • ASP
  • XML-serialization
  • Interop
  • Windows NT Eventlogging

Requirements

  • Windows NT4, Windows 2000 or Windows 2003 Server
  • runtime, Version 1.1
  • privileged accounts for all server services
  • manually created configuration files

Setup

The setup is a simple commandline program [setup.cmd], which itself uses only xcopy-deployment. There is a documentation, which explains, how to create the web, which is necessary for the viewers. Also, the accounts for the services must be created manually.

Restrictions

There is currently no security for the queues and the remoting interface for the EventlogCollectorService. The remoting service is not published and only used via a web-interface, which could and should be hosted in a protected website. In this case, the remoting service can be restricted to be accessible only local, if the service runs on the same machine as the webserver. In this situation, security can easisly maintained by the web-application.

Names used inside the application

Regard:The table below contains filenames also, but is not a filelist. A filelist is given later.

This is more or less the synonym or the trade-mark of this whole application.
This is a shorthand of the name above and is used nearly everywhere in other names, like in the applications filenames as a prefix. The ServiceApplication for example, is named ECS.ApplicationService.cs/.exe.
ApplicationService/ServiceApplication[=now]
/CollectorServiceApplication??
/EventlogCollectorApplicationService=ECSA??
/EventlogCollectorServerice???
/EventlogEventCollectorServerice???
This is the Windows NT Service for the EventCollectorService. It's filename is ECS.ApplicationService[.cs/.exe]
CollectorService This is one of the most importent classes inside this application and the filename is ECS.CollectorService[.cs/.dll]
Collector This is the worker class inside this application and queries and receives the events from a watched machine. The filename is ECS.Collector[.cs/.dll]
EventlogCollectorNotificationService
/EventlogCollectorEventNotificationService
This reads the messages from the queue and distributes them via net remoting to viewers. This is also titled fast event notification.

Implementation

This is first more or less a more detailed overview, or a deeper insight. The following explanations contain links to documents, which explain the real details for a given part of the project. Beneth the Application structure details, this document conatins also the more general concepts and implementation details.

The Applications Structure

The EventCollectorApplication

The main structure of the application is build on three parts:

  • The ApplicationService/ServiceApplication[=now]

    Is resposible for tasks like:

    • Interacting with the NT Service Database and Manager
    • Initializing the application
      • Reading the main-configuration-file
      • determine the location, existence and validity of the sub-configuration-files
      • de-serialize the sub-configuration-files into runtime classes and provide some start-modification
      • intialize and possibly create MSMQ queues
      • starting and stopping the ServerService

  • The CollectorService

    is responsible for loading the runtime-classes from it's [sub]configuration-file and create the configuration-classes for the EventCollector's:

    • determine the machine to watch
    • apply global and local filtering into the local filters for each collector
    • create run-time statistics
    • allow start, stop of collectors via the remoting-interface
    • allow add of new collectors at run-time
    • save changes of the runtime-configuration to [sub]configuration-files
    • determine hangs and missing events and handle re-animation
    • stores the received eventlog messages into the MSMQ queues
    • provides the remoting-interface for management
    See class-view:CollectorService

  • The EventCollector

    Is responsible to handle most detail work:

    • connect to the [remote] machine via WMI
    • watch events received
    • filter events
    • notify the CollectorService to store events
    • notify the CollectorService about lost connections
    • release resources

The EventViewers

Are responsible to display events. There are two different types of eventviewers:

  1. a more dynamic version
  2. a more static version
  1. The more dynamic version is build on a remoting technology. The events received are stored into two different MSMQ queues, one for this fast-notification and one, to store the collected events into the database. This version is also based on a web-page, but with an embedded WinForms control, which uses remoting to connect to the distribution-service. The events cannot be filtered and only the last half hour is visible.
  2. The more static version is completly web-based. The display makes requests via an ASPX page to the database. It contains a client-side GUI to modify and filter the request and allow automatic refresh of the display.

The DBWriter Service Application

The Site Replication Service Application

The Application Details

The following table lists all application-parts and it's related detailed documents and one may go directly there.

part link note
The Service-Application
The Server-Service
The EventCollector
The Eventviewer
The DBWriter Service Application
The Site Replication Service Application

Application namespaces

The following table list's the public enums, delegates and classes, contained in each separate namespace.

namespacename contains
The top namespace for the application-specific classes.
ECS.ServiceApplication.Config
  • [Serializable]

    [XmlRoot("ECS.ServiceApplication.Config")]

    public class ApplicationConfig

  • [Serializable]

    public class StaticConfig

ECS.ServiceApplication
  • public class Application
System.Management.Tools

Extensions to the System.Management namespace. Mainly a class to convert WMI datetime values to DateTime. This class is more or less extracted from the output generated by the VS strict-type class-converter.

  • public class Library
ECS.ApplicationUtilities
  • public class SerializationHelper
  • public enum LogCategories:short
  • public class EventLog:System.Diagnostics.EventLog
  • public class Error
  • public class ErrorQueue
ECS.Structures
  • public class EventlogMessageRecord:EventArgs
  • public class QueueConfig
  • public class EventlogFilter:IComparable
  • public class LogfileFilter:EventlogFilter, IComparable
  • public class SourcenameFilter:EventlogFilter, IComparable
  • public class EventcodeFilter:EventlogFilter, IComparable
  • public class FilterHelper
  • public class EventFilters
ECS.Debugging
  • public class DebugHelper
ECS.ServerService.Config
  • [Serializable]

    [XmlRoot("ECS.ServerService.Config")]

    public class ServiceConfig

  • [Serializable]

    public class StaticConfig

  • public class DynamicConfig
ECS.ServerService
  • [Serializable]

    [XmlRoot("ECS.ServerService.CollectorStatistics")]

    [System.Xml.Serialization.XmlInclude(typeof(CollectorStatisticsItem))]

    public class CollectorStatisticsItems

  • [Serializable]

    public class CollectorStatisticsItem

  • public class CollectorStateTransition
  • public class CollectorComposita
  • public class CollectorCompositum
  • public class CollectorNotificationStatus
  • public delegate void TerminationEventHandler(string reason)
  • public class CollectorService:MarshalByRefObject, IDisposable
ECS.Collector.Config
  • [Flags]

    public enum ComputerStatus:int

  • [Serializable]

    [XmlRoot("ECS.Collector.StaticConfig")]

    public class StaticConfigItems

  • [Serializable]

    public class StaticConfigItem

  • [Serializable]

    public class DynamicConfigItem

ECS.Collector
  • public delegate void EventlogMessageEventHandler(string computername, EventlogMessageRecord msg)
  • public delegate void FirstEventlogMessageEventHandler(string computername)
  • public delegate void ConnectionLostEventHandler(string computername)
  • public delegate void StateChangedEventHandler(string computername, CollectorStateName oldState, CollectorStateName newState)
  • public enum CollectorStateName:int
  • public class CollectorStateManager
  • public class CollectorState
  • public class Created:CollectorState
  • public class Running:CollectorState
  • public class Stopped:CollectorState
  • public class Reachable:CollectorState
  • public class Unreachable:CollectorState
  • public class Suspect:CollectorState
  • public class Waitconfirm:CollectorState
  • public class Misconfigured:CollectorState
  • [Synchronization()]

    public class EventCollector:ContextBoundObject, IDisposable

ECS.DBWriter
  • class DBWriter:IDisposable

Application class details

This part of the document focues on details of classes which are globally used in the EventCollectorService application and the other, related applications. The details on each main classes is found inside the details documents, see table above.

Global classes

The Service-Application

The CollectorService

This class is responsible to control the Collector instances and more [see overview]

public class CollectorService:MarshalByRefObject, IDisposable
public CollectorService()
To use this class in the remoting scenario, this empty constructor is necessary.
public CollectorService(ServiceConfig servConf)
The ServiceConfig is passed in by the creator. This is mainly to take over the classes runtime configuration from the application-service and to initialize the message queues, which itself might be created by the calling ServiceApplication.
~CollectorService()
Calls simply Dispose()
public void Dispose()
If called, stops all running collector instances and realeases all resources.
public void Init()
  • loads the configuration-file for the collectors
  • prepares the StartupSummary
  • initializes all collectors
  • calls BuildEventCollector for each Collector-instance.
private void BuildEventCollector(StaticConfigItem statCfg, bool delayedAutostart)
  • Creates the dynamic configuration for each Collector-instance
  • merges global filter-settings with collector-specific filter-settings
  • assigns the event-handlers
  • if this instance of a collector has autoStart and autoStart is enabled, the QueuedStart is called over the thread-pool.
public void AddEventCollector(Collector.Config.StaticConfigItem statCfg)
This method add's a new EventCollector instance to the existing instances. The StaticConfigItem must be provided. The danymic configuration is build on the usual rules and then the collector might be started, depending on it's settings.
public void DelEventCollector(string key)
The given instance is simply disposed. Dispose() properly handles the situation, if the instance is currently running. The Collector class. This instance is only added to the runtime configuration not to the static configuration.
public void InsertEventCollector(string key)
Takes an existing instance from the runtime configuration and add it to the static configuration, which then will be saved.
public void RemoveEventCollector(string key)
Removes an Collector instance from the runtime configuration and from the static configuration, which will be saved. If this instance is currently running, it will be stopped.
public StaticConfigItem GetEventCollectorTemplate(string key)
Creates a static config item in memory from a template in the configuration file. This is useful in adding new collectors and building their configuration.
public EventCollector this[int index]
This indexer is used to have direct read-only access to each collector instance by index.
public EventCollector this[string index]
This indexer is used to have direct read-only access to each collector instance by name.
public int Count
This is just to get the number of created collector instances.
public void StopEventCollector(string key)
This method stops a collector, given it's name. This method exposes a collectors method so it is accessible via remoting, without having to create remoting services for each instance.
public void StartEventCollector(string key)
This method starts a collector given by name. The instance must naturally be existent. This method exposes a collectors method so it is accessible via remoting, without having to create remoting services for each instance.

The EventCollector

The Eventviewer

The DBWriter Service Application

The Site Replication Service Application

Copyright, Distribution, Availability and Support

Setup

Installation checklist

  • CLR installed

Installation, step-by-step

Copy the files

Configure the application

Configure the collectors

Create the webs

Appendixes

Useful articles

title issuer link themes
C#: A Message Queuing Service Application MS:MSDN goThere MSMQ, configuration via xml

Notes on using the DB

This is just a hint-list, of what can be done with DB triggers, if this application is in use.

Logfile EventSource EventId Action
System SaveDump nn something .....
Security AccountManagement 713[fictive!!!!!] A workstation has just changed it's account's password. This is a moment were you know exactly, this machine is online. Can be used to trigger any action, like an inventory scan.

Table of files

filename location
ECS.Application.exe %AppRoot%

Table of project-relevant directories

path what
D:\Develop\Dev\Experimente\cs\Experimente\Remoting\EventsEnhanced\V01 The root of the remoting part
D:\Develop\Dev\Experimente\cs\Experimente\Remoting\EventsEnhanced\V01\MSMQBased_EventInitiator The special event-provider, reading the MSMQ m1.eventcollectormonitor queue.
D:\Develop\Dev\Experimente\cs\Projetcs\EventlogCollectorService The root of the server application development tree
D:\InetPub\VServer\www.mbg.local\Cont\DevNet\WebClientControls\UserControlWithRemotingDemo The root of the IE client control using the remoting service
D:\InetPub\VRoots\www_mbg_local\EventlogCollectorService The root of all web-relevant parts [Regard:The dir Docs is a hardlink to the application-development tree!!]

Annotations

Document specials

This document uses HTML Components to simplify things. For example, there is a project-wide dictionary, which allows shorthands for keywords and later, these keywords appear inside the document as usual, but always with the same formatting. Additionally, links in this document carry a superscript formatted indicator, to which location it points to. For example, the # [number sign] is used to indicate, that a link is inside this document. Links annotated with a small w, points to a document in the internet and such a link opens in a new window.


mb, jan 2004, manfred.braun@manfbraun.de