Netmagis

logo

Netmagis 2.3.x installation

Note: this installation documentation is for Netmagis version 2.3.x (x >= 0) If you are installing from a previous version of Netmagis, you should rather read upgrade instructions.

Program and file installation

Determining which packages are needed

The netmagis is divided into packages in order to install only what you need:

PackageDescription
commonThis package is mandatory for all installations. It includes common files and a configuration example.
databaseDatabase server (maintenance and installation scripts, as well as scripts to upgrade database from previous versions).
wwwWeb application (CGI scripts, HTML templates, Apache configuration files, etc.).
serversScripts running on various servers (DNS, DHCP, mail router).
utilsScripts to ease batch modifications of hosts in the database.
detecteqSet of files or programs to detect configuration modifications on network equipments. This package must be installed on your central syslog server (for Cisco and Juniper equipments) and your central Freeradius server (for HP equipments). This package is needed if you plan to run the topo package.
topoBackend used to fetch configuration data from network equipments, modelize topology in a graph, and optionnaly inject new ports configuration to equipements.
In order to run properly, this package needs to know when a modification of configuration is done on any network equipment. This is the role of the detecteq package.
metroBackend used to poll equipments for metrology information.

Packages are designed such as they may be installed on one unique server as well as each on its dedicated server, depending on your security requirements, performance considerations or site specificities.

  • a large site may wish to separate all components:
    • the www package is installed on a central web hosting platform
    • the database package is installed on a central database cluster
    • the servers package is installed on the existing DNS and DHCP server, as well as on mail relay cluster
    • the detecteq package is installed on both the central syslog server and the central Radius server
    • the topo package is installed on a dedicated virtual server, for security reasons (access to the network equipements via the administration VLAN)
    • the metro package is installed on a dedicated physical server for performance and security reasons
  • on the opposite, a small site may wish to run all packages on a unique virtual server.

The www and the topo packages must be installed on the same architecture (CPU and OS), even if the servers are physically distinct.

Installing from FreeBSD ports

On FreeBSD, Netmagis is integrated in the ports tree, meaning that installation may be performed directly from binary packages or by compilation from the ports tree.

Installing from Debian/Ubuntu packages

Linux packages are provided for installing on:

  • Ubuntu (minimum distribution is 18.04)
  • Debian (minimum distribution is stretch - stable)

Append the following lines to your /etc/apt/sources.list:

deb http://netmagis.org/files/repo stable main
deb-src http://netmagis.org/files/repo stable main

Warning: starting from Ubuntu 18.04/Debian stretch, packages must be signed. You need to run the following command to add the Netmagis signature:

# wget -O - http://netmagis.org/files/netmagis.key | apt-key add  -

If you are using an Ubuntu distribution, check that the "universe" section is enabled in /etc/apt/sources.list.

Now, update your package database:

# apt-get update

The next step depends upon your architecture:

  • If your system is based on an amd64 architecture (x86_64 in "uname -i" output), you can install from binary packages (example for the database package):

    # apt-get install netmagis-database

  • If your system is based on another architecture (including i386 for old 32 bit hardware), you have to rebuild from source package:

    # apt-get install dpkg-dev
    # apt-get build-dep netmagis
    # apt-get source --build netmagis

    Once the packages are built, install them with gdebi (example for the database package and the pdp11 architecture):

    # apt-get install gdebi
    # gdebi netmagis-database_2.3-1_pdp11.deb

Installing from Netmagis source distribution

  1. Download the Netmagis distribution file and unpack the archive:

    $ tar xzvf netmagis-2.3.0.tar.gz

    Alternatively, if you are adventurous, you may also fetch the current Netmagis sources directly from the gitlab repository. Provided that git is installed on your system, use this command:

    $ git clone git@gitlab.com/netmagis/netmagis.git
  2. Install required dependencies. Please note that Tcl is a required dependency for all packages.

    • Linux (Ubuntu/Debian) note:
      # common and database
      aptitude -y install make git \
              postgresql-9.3 postgresql-pltcl-9.3 libpgtcl1.5 \
              tcl8.5 tcl8.5-dev tcllib tcltls tdom pwgen

      # www
      aptitude -y install
              apache2 libaprutil1-dbd-pgsql \
              netpbm texlive-latex-base graphviz transfig

      # topo
      aptitude -y install \
              gcc fping rancid default-mta

      # metro
      aptitude -y install \
              libsnmp-session-perl libsnmp-perl libnet-snmp-perl librrdtool-oo-perl \
              libdbd-pg-perl rrdtool

      # ldap authentication
      aptitude -y install openldap-utils slapd

    • Linux (CentOS) note:

      Warning: these notes for CentOS are obsolete and have not been tested for the current release of Netmagis. Use them at your own risk.

      Help wanted: if you can, please provide update installation notes for CentOS as an issue in the Netmagis development site.

      If you are installing for a CentOS distribution, you need to install the Epel repository or to download each package not in CentOS repository (such as pwgen, tcllib, etc.) from http://dl.fedoraproject.org/pub/epel/6.

      Then, you can use yum to install dependancies:

      # common and database
      yum install tcl tcllib pwgen tcl-devel \
              tcl-pgtcl postgresql-server gcc \
              make postgresql-pltcl

      # www
      yum install httpd apr-util-pgsql mod_ssl

      # topo
      yum install rancid fping

      # metro
      yum install rrdtool-perl net-snmp-libs net-snmp-perl \
              perl-Net-SNMP net-snmp-libs perl-Digest-HMAC \
              perl-IO-Socket-INET6 perl-Crypt-DES \
              perl-Net-SNMP perl-DBD-Pg

      Then, change the top-level Makefile to suit your environnement:

      TCLSH = /usr/bin/tclsh
      TCLCONF = /usr/lib64/tcl8.5/tclConfig.sh
  3. Choose an installation directory. You may use /local/netmagis/ for example. Netmagis's Makefile will install a hierarchy such as:

    .../etc/various configuration files
    .../bin/programs which should be called as an ordinary user
    .../sbin/programs which should be called as the root user
    .../lib/libraries, messages, and programs which are not meant to be called by humans
    .../share/examples, default configuration files, database upgrade scripts, etc. NB: on Debian/Ubuntu, examples files are in /usr/share/doc/netmagis/examples/
    .../var/files generated by topo or metro packages
    .../www/Web application (installed by www package)

  4. Modify the top-level Makefile variables to suit your choices:
    DESTDIRInstallation directory
    TCLSHFull path name of tclsh interpreter
    TCLCONFTcl configuration script (example: /usr/lib/tcl8.5/tclConfig.sh on Linux or /usr/local/lib/tcl8.5/tclConfig.sh on FreeBSD)

  5. Install the package:
    # make install-common
  6. Install the desired package(s). For example, if you want to install the database, www and the utils packages, type:
    # make install-database
    # make install-www
    # make install-utils

That's it! You may now proceed to initial configuration.

Initial PostgreSQL configuration

On the host where you installed the database package, you should have a PostgreSQL server. The recommanded version is at least 9.1.

You have to check that your PostgreSQL installation allow connections (and authenticate users with password) from hosts which act as clients of the Netmagis database (all hosts where you installed a netmagis package). This is usually configured in file pg_hba.conf which is located in:

FreeBSD˜pgsql/data/pg_hba.conf (where pgsql is the name of the PostgreSQL administror)
Linux (Debian/Ubuntu) /etc/postgresql/9.1/main/pg_hba.conf

Netmagis relies on password authentication, even if the database is on localhost, so the "ident" method won't work. You should change the configuration in pg_hba.conf to look like this:

host all all 127.0.0.1/32 md5
host all all ::1/128 md5

Netmagis uses only one PostgreSQL user, without special privilege (except database creation privilege). You must create it (as the PostgreSQL administor user: pgsql on FreeBSD, postgres on Linux) with the PostgreSQL createuser program:

createuser --no-superuser --no-createrole --createdb --pwprompt nm

Even if you want to use a different database for the mac package, you can use the same user.

The Netmagis configuration file

The Netmagis configuration file is located in .../etc/netmagis.conf on each host where a Netmagis package is installed. The common package does not install this file (in order to leave your file intact if you previousely installed Netmagis), but it installs a copy in .../etc/netmagis.conf.sample. Copy it manually to .../etc/netmagis.conf and edit it.

The following table list only directives you would want to modify for a fresh installation.

DirectiveDescription
dnsdb...Information needed to connect to the PostgreSQL server, with an unprivileged user (see below).
macdb...Information needed to connect to the PostgreSQL server for the mac package, with an unprivileged user. If you don't want to use a separate database, or if you don't want to use the mac package, use exactly the same values as for the the dnsdb... parameters.
ouiurlURL to the OUI database (a default value is provided in the sample configuration file). This is necessary to populate the mac.oui table with Ethernet vendor prefixes when the mac database is created.
cafilePath to the file containing the root certificate authorities. Usually /usr/local/share/certs/ca-root-nss.crt on FreeBSD and /etc/ssl/certs/ca-certificates.crt on Linux (Debian/Ubuntu).
whoamiCommand used to get login name of the user who is running the current program. It should display exclusively the login name. Default value is suitable for FreeBSD and Ubuntu.
rootusersLogin names of Netmagis users allowed to access the application even when it is in "maintenance mode".

Use the script netmagis-config (check your PATH) to test that Netmagis understands your modifications. The command:

$ netmagis-config dnsdbhost
should display a string such as:
dnsdbhost="myhost.example.com"

You can now proceed to database installation.

Database installation

PostgreSQL server should be initialized as described above. Netmagis should then be installed by providing .../etc/netmagis.conf directives:

DirectiveDescription
dbcopyName of a copy of the Netmagis database, generated every night (see the netmagis-dbmaint program in the database package). This copy lets you experiment and work on data without fear.
If this directive is empty, the maintenance command will not create a copy of the database.
dumpdirDirectory where the netmagis-dbmaint program stores periodic backups of the database.
pwgenCommand line to generate a random password

  • If you are upgrading from a previous WebDNS or Netmagis installation, please see the upgrade instructions.
  • If you are installing Netmagis for the first time, after you configured the netmagis.conf file, run the command
    $ .../sbin/netmagis-dbcreate
    This command:
    • creates the Netmagis database (it fails if the database is already created)
    • creates the MAC database if it is different from the main Netmagis database (it fails if the database is already created)
    • creates the SQL schemas, tables, functions and triggers
    • provides default values for Netmagis directives
    • creates Netmagis users listed in rootusers directive (in .../etc/netmagis.conf file)
    • creates a Netmagis user called nobody
    • assigns all Netmagis users a random password, and displays them on standard output
    Users are created in the Netmagis database, including the internal authentication schema.
    • if you use an LDAP or CAS authentication, rootusers users should exist in your LDAP directory.
    • if you plan to use the internal authentication database, note the generated passwords as they will be needed in the Web application installation
  • Unless you are upgrading from previous versions, you should import your data into the database. Doing so is described in the section below.

    Since this is a long task, if you want to test your installation, you may try to import example data. You may do so by issuing the command to load the multi-views example (you can load the no-view example instead):

    $ dropdb --username=nm netmagis
    $ cd .../share/examples/netmagis/with-views
    $ sh ./run-all.sh

    You can later destroy and re-create the database with your own data.

  • Install the sample crontab (.../share/examples/netmagis/crontab.database) in your default cron system. This crontab can be run by any unprivileged Unix user, provided that this user has write access to the dump directory (see the dumpdir directive above).

    FreeBSD users: if you install this crontab in the global /etc/crontab, please provide a user name just before the command name.

    The .../sbin/netmagis-dbmaint command performs some maintenance tasks:

    • make a hourly backup of database in the directory named in the dumpdir directive from .../etc/netmagis.conf
    • copy the current database in a copy named after the dbcopy directive
    • performs a PostgreSQL vacuum operation
    • remove some old entries in Netmagis internal spools

Netmagis database is now installed. You may now install the www package.

Web application installation

To store user accounts, the Web application can use either an existing external LDAP directory or the internal PostgreSQL database.

netmagis.conf directives
DirectiveDescription
passwdurlURL which should be displayed to let users change their password, if you use an external LDAP authentication. Leave empty if you use internal PostgreSQL authentication.
hostcmdPath to the host command
pwgenCommand line to generate a random password (not used for LDAP and CAS authentication)
randomDevice name of the non-blocking pseudo-random generator

  • A default Apache configuration file is provided in ..../share/examples/netmagis/httpd.conf. Adapt this file to your site and include it in your Apache configuration.
  • Now, direct your Web browser to the http://yourhost/netmagis/index, and connect with your login name, using the password generated by the netmagis-dbcreate command during the database creation phase.

    You should see a main menu such as this one.

  • If you are using internal (PostgreSQL based) Netmagis authentication, change your password (unless you want to keep the generated one).

    Change also your name (as well as other account informations) with the Admin / Internal Auth / Modify account menu.

  • If you are using the LDAP based authentication, go into the Admin menu (on the left vertical bar), then to the Application parameters menu.

    • In the "LDAP parameters" section, adapt values to your LDAP directory.
    • Then select the LDAP item in the Authentication method menu (in the "General parameters" section).
    • Then, click on the Submit button to register your new configuration.
    • Log out, or better, use an alternate browser to log in, in order to check that your authentication parameters are correctly configured.

    If some of the LDAP parameters are erroneous, you will not be able to log in. To recover from this state, use the following command to reset the authentication method:

    $ psql -U nm -c "UPDATE global.config SET value = 'pgsql' WHERE key = 'authmethod'" netmagis
  • If you are using the CAS based authentication, go into the Admin menu (on the left vertical bar), then to the Application parameters menu.

    • Adapt the "CAS authentication parameters" section.
    • Next, adapt the "LDAP parameters" section as explained in the previous item (LDAP based authentication).
    • Then select the CAS item in the Authentication method menu (in the "General parameters" section).
    • Then, click on the Submit button to register your new configuration.
    • Log out, or better, use an alternate browser to log in, in order to check that your authentication parameters are correctly configured.

    If some of the LDAP or CAS parameters are erroneous, you will not be able to log in. To recover from this state, use the following command to reset the authentication method:

    $ psql -U nm -c "UPDATE global.config SET value = 'pgsql' WHERE key = 'authmethod'" netmagis

Congratulations! The Web application is now installed.

Import your existing data

Your next task is to feed your database with referential data (users, groups, networks, etc.) as well as your existing DNS zones if any.

Configuring Netmagis

Documentation about common tasks in Netmagis configuration is provided in a separate page.

Utils package installation

The utils package is optional: it contains commands to add, delete or modify individual hosts, as well as DNS zone prologues. These commands are designed to be used wth the Unix shell.

Installation is very simple: only common netmagis.conf directives to connect to the database are needed.

Notice, however, that this file (including password to connect to the database) must be readable to all users of these scripts. This may be a security risk if you make this file readable for all users, but you can also control access to the configuration file and to the scripts based on a group policy.

The provided commands are:

  • dnsaddhost fqdn addr view: adds the specified host with the IPv4 or IPv6 address.
  • dnsaddalias fqdn-alias fqdn-host view: adds the specified alias to an existing host.
  • dnsdelhost fqdn view: deletes the specified host or alias.
  • dnsdelip addr view: deletes an IP address. If this is the last, deletes the host.
  • dnsmodattr fqdn view key val key val ...: modify an existing host. Possibles keys are:
    • mac: value is the MAC address
    • hinfo: value is the exact name of the the machine type, as listed in the Modify machine types administrator menu. or in the dns.hinfo SQL table

    • respname: name of user responsible for this host
    • respmail: mail address of user responsible for this host
    • comment: comment associated to this host
    • dhcpprofile: value is the exact name of the DHCP profile, as listed in the Modify DHCP profiles administrator menu. or in the dns.dhcpprofile SQL table
    • ttl: value is the specific TTL value for this host, in seconds, or empty string to remove the specific value
    • sendsmtp: value is 0 to remove the right to emit mail with unauthenticated SMTP, or 1 to add the right.
  • dnsreadprol zone view: displays on standard output the DNS zone prologue specified by its domain name.
  • dnswriteprol zone view file: use the named file as the zone prologue. Since this operation may have a great impact, manual confirmation is required.

With all these commands, you have to provide a view name. If you don't use multiple views, just use the name "default".

Note that users using this script must have a Netmagis account with the same login as the Unix one.

Servers package installation

The servers package must be installed on your main BIND server, on your DHCP server, and on your mail router, depending on the features you want.

On all servers

On all servers, you will have to configure .../etc/netmagis.conf with the following directive:

netmagis.conf directives
DirectiveDescription
diffCommand line used to signal differences between two versions of a server file (DNS zones, DHCP files, etc.).

BIND server

On your main BIND server, provide values for the following directives:

netmagis.conf directives
DirectiveDescription
zonedirDirectory where your DNS zone files are read by your DNS daemon
zonecmdCommand used to make your DNS daemon reload the generated zone files. The default command is for BIND 9.x.

Please note that if this command fails, zone generation is aborted and old zone files are restored.

If you have configured a cluster of name servers, you can use this directive to propagate inforamtion to other nodes.

Once you configured .../etc/netmagis.conf file, run (as the user who have access to the zonedir directory and to the zonecmd command):

$ .../sbin/mkzone -n -v
This command tests zone generation, without performing any file modification.

You can force generation of some zones by using the Admin / Force zone generation. This sets a flag in the dns.zone SQL table, which is read by the mkzone to find modified zones.

Test now a real zone generation by using:

$ .../sbin/mkzone -v

Each zone file will be named after the "Name" column in zone edition.

If you are using multiple views on different DNS servers, you can use the -w view option to limit zone generation to a single view. By default, the mkzone program generates zone files for all views.

If everything is ok, you can use the sample .../share/examples/netmagis/crontab.bind to install the periodic zone generation for the user.

An usual convention about the SOA serial number is to use a number of the form "yyyymmddnn" where yyyymmdd is the current date and nn is a sequence number for this date. Netmagis uses this convention when possible, but if there is more than 100 modifications in a day, it will simply increment the value. Our algorithm will automagically synchronizes again with the convention when possible. This way, you can run the mkzone script every minute without fear.

FreeBSD users: if you install this crontab in the global /etc/crontab, please provide a user name just before the command name.

ISC DHCP Server

Netmagis is designed to aliment a central ISC DHCP server. If you want to use this functionnality on other networks, you need to activate DHCP relay in your router configurations.

The Netmagis can also handle failover configuration in order to provide a redundant DHCP service based on 2 ISC DHCP servers.

On your DHCP server, provide values for the following directives:

netmagis.conf directives
DirectiveDescription
dhcpfileLocation of generated file for ISC dhcpd daemon. This file should be included by your main dhcpd.conf file (see you ISC dhcpd configuration), with a line such as:
include "/etc/dhcp-gen.conf"

Note: if you don't plan to use DHCP generation, don't panic! This directive (and the next ones) will not be used.

dhcpfailoverLine to add to generated pools if you use ISC DHCP failover feature (DHCP redundancy over 2 servers). Leave empty if you don't use this feature.
dhcptestCommand used to check DHCP configuration file before restarting ISC dhcpd daemon. This is needed since Netmagis administrators may add faulty DHCP profiles, and ISC dhcpd daemon is not very tolerant to configuration errors. Please note that this command contains the location of your main teletype dhcpd.conf file.
dhcpcmdCommand used to make your ISC dhpcd daemon reload the configuration.

Once you configured .../etc/netmagis.conf file, run as root:

$ .../sbin/mkdhcp -n -v
This command tests DHCP file generation, without performing any file modification.

Test now a real DHCP file generation by using:

$ .../sbin/mkdhcp -v

If you are using multiple views, you need to supply a specific view for which you are generating a DHCP file, with the -w option:

$ .../sbin/mkdhcp -v -w internal
By default, the mkdhcp program generates a DHCP file for all views, which is certainly not what you want if you are using multiple views.

If everything is ok, you can use the sample .../share/examples/netmagis/crontab.dhcp to install the periodic DHCP file generation. You should run the DHCP file generation soon after the DNS zone generation (such that new host names are not generated in DHCP file before they are known in the DNS).

FreeBSD users: if you install this crontab in the global /etc/crontab, please provide a user name just before the command name.

Mail relay cluster

Netmagis can be used to generate mail routes for you central mail router. Routes are generated following "mail roles", whose modifications are enabled individually for each allowed domain in the Admin / Modify users and groups menu.

netmagis.conf directives
DirectiveDescription
mroutefileFile read by sendmail (or Postfix) to get mail routing information. The default line format (see "mroutefmt" directive) ressembles:
specific.domain.com smtp.[host.domain.com]
which means that mail addressed to someone@specific.domain.com is to be sent to host host.domain.com.

Note: if you don't plan to use mail route generation, don't panic! This directive (and the next ones) will not be used.

mrouteprologuePrologue of the generated mail route file. This prologue allows you to mention specific mail route information which cannot be extracted from the database.
mroutefmtFormat of mail route entries: %1$s is the mail address and %2$s is the host accepting mails for this address.
mroutecmdCommand used to make your Sendmail daemon reload the generated route file.

If your mail router is a cluster of hosts, this command will have to propagate the file and restart the daemon on other nodes.

Once you configured .../etc/netmagis.conf file, run as root:

$ .../sbin/mkmroute -n -v
This command tests mail routes file generation, without performing any file modification.

Test now a real file generation by using:

$ .../sbin/mkmroute -v

If you are using multiple views, you need to supply a specific view for which you are generating mail routes, with the -w option:

$ .../sbin/mkmroute -v -w internal
By default, the mkmroute program generates a routes for all views, which is certainly not what you want if you are using multiple views. For example, if you have a mail role sales.example.com (in view external) which points to a mailbox host named server.example.com (in view internal), chances are that your mail router "sees" the "internal" view (otherwise, it could not route mail to server.example.com). Thus, you have to specify:
$ .../sbin/mkmroute -v -w internal

If everything is ok, you can use the sample .../share/examples/netmagis/crontab.mroute to install the periodic file generation.

FreeBSD users: if you install this crontab in the global /etc/crontab, please provide a user name just before the command name.

Authenticated SMTP filtering

If you use authenticated SMTP, you know than some hosts (copiers, embedded measurement machines, etc.) may not be able to use authenticated SMTP. Some exceptions must be done.

Netmagis helps you handle these exceptions, by providing a special flag to each host. Right to activate this flag is enabled for each group in the Admin / Modify users and groups menu.

To activate this functionnality, some prerequisites are needed on your mail infrastructure:

  • Authenticated SMTP message submission is enabled on TCP port 587
  • Unauthenticated SMTP access is enabled on TCP port 25
  • All your routers restrict use of port 25, such as internal hosts can only open connections to mail submission hosts
  • Your mail submissions hosts accept connections on port 25 only from authorized hosts.
    A configuration file for a *BSD PF (packet filter) is provided as an example in .../share/examples/netmagis/pf.conf.sample. You will notice that the smtpf table is read from a file, generated by Netmagis.

With this infrastructure, provide values for the following directives on your mail submission hosts:

netmagis.conf directives
DirectiveDescription
pffileFile where the PF packet filter (on *BSD hosts) expects to find the contents of the table giving IP addresses of hosts allowed to use unauthenticated SMTP, i.e. hosts allowed to connect to TCP port 25.

Note: if you don't plan to use Netmagis to control which hosts have access to unauthenticated SMTP, don't panic! This directive (and the next ones) will not be used.

pffmtFormat of generated file (containing the smtpf table) included in pffile
pfprologueLines to prepend, if any, to the generated file. May be set to /dev/null
pftestCommand used to check the generated file
pfcmdCommand used to reload the generated pf table

Once you configured .../etc/netmagis.conf file, run as root:

$ .../sbin/mksmtpf -n -v
This command tests filter file generation, without performing any file modification.

Test now a real file generation by using:

$ .../sbin/mksmtpf -v

As with mkmroute, you can use the -w view option to specify a particular view.

If everything is ok, you can use the sample .../share/examples/netmagis/crontab.smtpf to install the periodic file generation.

FreeBSD users: if you install this crontab in the global /etc/crontab, please provide a user name just before the command name.

Congratulations! You have installed the servers package. The Netmagis is ready to be used, for its core functionnality.

Detecteq package installation

The topo package needs to know when the configuration of a network equipment has been changed. These changes are detected via:

  • syslog for Cisco and Juniper equipments
  • radius accounting for HP equipements. In this case, since HP equipments are unable to report configuration changes via Syslog, the hack is based on Radius accounting which detects user logout (which do not strictly imply a configuration change).

Thus, the detecteq package must be installed on your central syslog (syslog, syslog-ng, etc.) server and/or your central radius server, depending on your equipments. Since modification events are sent to the database, the detecteq package does not need to be installed on the same server as the topo server.

Network equipment configuration

In order to automate configuration fetching from a central server, a unique user should be defined on every network equipment. Let us call it conf. This user should have the right to execute the show configuration commmand (or similar), which means it must have the "admin" privilege.

Moreover, at least on HP equipments, there should be another distinct user (let us call it modconf) which should be able to modify the configuration in order to change interface characteristics. This second user is needed for Radius to distinguish regular configuration fetching from configuration modification.

Syslog configuration

  • On Cisco and Juniper equipments, you need to configure a syslog server if not already done. For example:

    EquipmentExample
    Junipersystem {
        syslog {
           host syslog-server.example.com {
               any notice;
               facility-override local2;
           }
        }
    }
    Ciscologging syslog-server.example.com
    logging facility local2
  • Configure .../etc/netmagis.conf for access to the Netmagis database
  • Update your syslog configuration file and use a line such as:
    local2.* |exec /usr/local/sbin/detectconfmod

    Adapt this line to your syslog facility and the detecteq installation path. You can also use other syslog daemons such as syslog-ng if they provide the "pipe to program" functionnality.

    Note: on Linux (Debian/Ubuntu), the default syslog program (rsyslogd) does not support the "pipe to program" functionnality. An alternative is to install the syslog-ng package and configure it with lines such as:

    source remote {
        udp(port(514));
    }

    filter myequip {
        netmask ("10.20.30.0/24");
    };

    destination mylogs {
        file ("/var/log/myeq.log");
    };

    destination detectconfmod {
        program ("/usr/local/sbin/detectconfmod");
    };

    log {
        source(remote);
        filter(myequip);
        destination(detectconfmod);
        destination(mylogs);
        flags(final);
    }

  • Be sure to keep hostnames in generated logs. You should get lines such as (example on a Cisco switch):
    Dec 23 11:52:26 adm-sw %SYS-5-CONFIG_I: Configured from console by joe on vty0 (192.168.1.1)

Freeradius configuration

  • On HP equipements, configure Radius accounting using the following command:

    EquipmentExample
    HPradius-server host radius.example.com key secret acct-port 1813

  • Configure your existing Freeradius using the SQL accounting module and the example files provided in .../share/examples/netmagis/radius/ directory.
      sql {
        database = "postgresql"
        driver = "rlm_sql_${database}"
        server = "mydatabaseserver.example.com"
        login = "nmdetecteq"
        password = "mysecret1234"
        radius_db = "netmagis"
        sqltrace = no
        sqltracefile = ${logdir}/sqltrace.sql
        num_sql_socks = 5
        connect_failure_retry_delay = 60
        lifetime = 0
        max_queries = 0
        sql_user_name = "%{User-Name}"
        nas_query = ""
        authorize_check_query = ""
        authorize_reply_query = ""
        authorize_group_check_query = ""
        authorize_group_reply_query = ""
        accounting_onoff_query = ""
        accounting_update_query = ""
        accounting_update_query_alt = ""
        accounting_start_query = ""
        accounting_start_query_alt = ""
        accounting_stop_query = " \
            INSERT INTO topo.modeq (eq,login) \
            ( \
              SELECT rr.name || '.' || domain.name, \
              '%{User-Name}' \
              FROM dns.rr, dns.rr_ip, dns.domain \
              WHERE rr_ip.idrr = rr.idrr AND \
                rr.iddom = domain.iddom AND \
                rr_ip.addr = '%{NAS-IP-Address}' AND \
              '%{User-Name}' NOT IN ( \
                  SELECT login \
                  FROM topo.ignoreequsers) \
            ) \
            "
        accounting_stop_query_alt = ""
        group_membership_query = ""
        postauth_query = ""
      }
  • As described above, the Radius accounting server needs to distinguish regular configuration fetching from configuration modification. This is done using the topo.ignoreequsers table which contains login names of the users to ignore.

    The default user is named conf. If want to change names of users which perform read-only (i.e. fetch) operations, issue the following SQL commands with psql:

    DELETE FROM topo.ignoreequsers;
    INSERT INTO topo.ignoreequsers VALUES ('joe');
    INSERT INTO topo.ignoreequsers VALUES ('bill');

Topo package installation

One of the most innovative parts of Netmagis is the topo package, which provides a complete infrastructure to fetch and analyze network equipments configuration files and modelize the network as a graph. This allows for automatic generation of network L2 and L3 maps, equipment port configuration, as well as interactions with the metrology package.

At this time, Juniper, Cisco and HP equipments (switches and routers) are supported.

Relations between topo and www packages

You have the choice to run the topo and www packages on the same host or on distinct hosts.

For security reasons, one would often use a separate server to run the topo package. This server should be located within the administration VLAN of equipments, if you have one.

The interface between the topo package and the rest of Netmagis is:

  • the ability to send mail from the topo server

  • the ability for the www package to access data extracted from the network graph built on the topo server: the topo server pushes (via a SSH connection) the file containing the built graph, and the www server runs extract commands on demand.

    This means that the topo package must also be installed on the www server to provide these commands.

    This also means that the architecture of the two servers must be compatible (CPU and OS) since the graph is a binary file.

The rest of this section describes the topo package installation.

On the topo server

The first part is the configuration of the topo server.

  • The first step of installation is to configure the rancid software (prerequisite of the topo package) on the topo server.

    Detailed information can be found in the README coming with the rancid package. However, here are the minimal steps to a quick configuration:

    • Check that the rancid user is created
    • Choose a unique "group" name for all your network equipments. The topo package uses the name mygroup by default. The name must be configured in the file rancid.conf (location is system dependant), in the line beginning with LIST_OF_GROUPS. While you are here, notice the BASEDIR directory name.
    • Configure a mail alias rancid-groupname (in /etc/aliases or equivalent, don't forget to run the command newaliases or equivalent) to receive notifications when rancid detects modifications.
    • Configure the ˜rancid/.cloginrc file with authentication information. For example:
      add method * {ssh}
      add password myrouter {pw} {enablepw}
      add password sw-* {anotherpw} {anotherenablepw}
      See the cloginrc(5) manual page for details about this configuration file.

      Be sure that this file belongs to the rancid user and has mode 0600.

    • Initialize the directory tree by running (as the rancid user):
      rancid-cvs
      This program is located in /usr/local/libexec/rancid/rancid-cvs on FreeBSD and in /usr/lib/rancid/bin/rancid-cvs on Linux (Debian/Ubuntu).
    • In order to test your rancid installation, name some equipments in the router.db file, which should be located in $BASEDIR/group-name/router.db. See the rancid README for instructions.

      This file will be overwritten by the topo package with information from the Netmagis database.

    • Test your installation by running:
      rancid-run
      The test is ok if you received a mail with the equipment configuration, which is also stored in the $BASEDIR/group-name/configs/ directory.
    • The rancid README says to run rancid-run periocally with cron. Do not do this: the topo package will call rancid when needed.

    Notice: this is not related to Netmagis installation, but you should consider installing a Web interface to CVS (CVSWeb) to ease tracking of your equipement configuration history.

  • The topo package must know which equipments are on each side of a network link. Therefore, you have to add information in equipment configurations: a link name (or number) prefixed with L, bracketed with "<" and ">", and added to the description of the interface on each side of the link:

    EquipmentExample
    Juniperge-0/1/0 {
        description "blah blah <L233>;
    }
    Ciscointerface GigabitEthernet0/1
        description blah blah <L233>
    HPinterface A1
        name "blah blah <L233>"

    You are advised to use the Web application (Topo / Link number menu item) to get and register link numbers. This way, you keep track of used link numbers, and link numbers are ensured to be unique.

    Ports connected to individual hosts must be marked with a "<X>" to indicate a terminal node in the graph.

    A complete description of the topo configuration is available on a separate page.

  • Configure .../etc/netmagis.conf on the topo server:

    netmagis.conf directives
    DirectiveDescription
    rancidcmdPath to the rancid-run command
    ranciddirRancid group directory (usually $BASEDIR/group-name/) where are located:
    • the router.db file which lists all polled equipments
    • the configs/ directory which contains all configurations fetched by rancid
    sendcisco, sendhp, sendjuniperCommands used to propagate new port configuration on an equipment, if you choose to enable the functionnality of letting users modify interface configuration.
    fpingcmdCommand used to launch fping, in order to test equipment before sending commands to change port configuration
    loggerCommand used to send a message to syslog
    eqvirtdirDirectory used for "virtual equipments". Each file is a specially crafted configuration which represent one equipment whose configuration cannot be fetched by rancid
    eqgendirDirectory used to store results of analysis of rancid fetched configurations
    topobindirDirectory where all topo programs are searched
    topographPath name of the graph built during topo analysis
    topocpgraphParameters (user@host:remote-path) used to copy the graph built during topo analysis to the www server. Leave empty if the topo and www servers are the same host.

  • Change the ownership to the rancid user:
    # chown -R rancid .../var/netmagis
  • As the rancid user, run the following command:
    $ .../lib/netmagis/topo/anaconf

    You will undoubtly notice a large number of warning that you will correct with time. If no fatal error has been encountered, you will normally find in .../var/netmagis/network.graph the modelization of your network. This is a binary file.

    To check that things are ok, you can use the commands (which all take the binary graph as standard input):

    • dump the graph in clear text:
      .../lib/netmagis/topo/dumpgraph -a < .../var/netmagis/network.graph
    • display all interfaces characteristics of an equipment:
      .../lib/netmagis/topo/extracteq -a equipment < .../var/netmagis/network.graph
  • Be sure that the detecteq package is properly installed on your central syslog server and/or your central radius server.
  • Run the topographd and the toposendd daemons via the provided startup scripts:

    FreeBSD/usr/local/etc/rc.d/topographd
    /usr/local/etc/rc.d/toposendd
    Linuxcopy or link XXXXXXXXXXX .../etc/rc.d/topographd.sh and .../etc/rc.d/toposend.sh to /etc/init.d directory
  • If you have choosen to install the topo and www packages on different servers, you have to configure the SSH connection from the rancid user on the topo server to a user on the www server.

    This SSH connection is used to copy the network graph computed by the rancid user on the topo server to the www server. The destination user on the www server will own the copied graph, which must be readable by the CGI scripts running under the Apache user. This destination user may be, for example, the Apache user (called www, www-data, nobody, etc., see the "User" directive in Apache configuration files). Let us call it the w user. Make sure this w user (on the www server) has a home directory and a valid shell.

    Then, as the rancid user on the topo server, create a SSH keypair (without pass-phrase):

    ssh-keygen
    Copy the public key (˜rancid/.ssh/id_rsa.pub) to the ˜w/.ssh/authorized_keys on the www server.

    Make sure that the topocpgraph configuration parameter is initialized with a line such as:

    topocpgraph w@www.example.com:.../var/netmagis/network.graph
    Warning: the path supplied here is the path of the graph on the server, which should match the topograph directive on the www server.
  • If you have choosen to install the topo and www packages on the same server, make sure that the topocpgraph directive is empty.

On the www server

You need now to come back to the server which hosts the Web application.

  • If you have choosen distinct www and topo servers, you must first install the topo package (on the www server) in order to use programs which extract data from the network graph. Then, you have to configure the following directives in .../etc/netmagis.conf on the www server:

    netmagis.conf directives
    DirectiveDescription
    topobindirDirectory where all topo programs are searched (on the www server)
    topographPath name of the graph copied onto the www server.

  • Direct your web browser to the Netmagis application, and go to the Admin / Application parameters menu. In this menu:
    • activate the topo package
    • change the "Default domain" value: the domain name of your network equipments
    • change the "Topo mail sender" value: you may use an invalid address such as automatic mails generated by the topo server don't receive replies.
    • change the "Topo mail receiver" value: topo mails will be sent to this address rather than to the genuine rancid-groupname alias you've previousely defined.
    After activation of the topo package, you will see a new Topology entry in the left menu.
  • Make sure that all your network equipments (and the default domain) exist as hosts (and domain) in the Netmagis database.
  • Input your network equipments (as fully qualified domain names) and your VLANs via the Admin / Modify equipments and Modify Vlans menus.
  • You can monitor the topographd and toposendd daemons with the Admin / Topod status menu.
  • If you want users to be able to modify interface configuration within the Web application, enter the Admin / Modify users and groups menu, and input regular expressions in the Permissions on network equipments section.

Configuring Topo

The Topo configuration documentation has all information to complete the Topo configuration.

Congratulations! You've configured the topo package. You can now consult your equipments, your network maps, and even modify interfaces of your equipments if you enabled this feature!

Metro package installation

The Metro package provides metrology data collection on equipments. Data collected include:

  • traffic
  • ip<->mac associations (ARP tables)
  • port<->mac associations (Forwarding tables)

At this time, all kinds of equipments are supported for traffic and ip/mac collection, provided they run a standard SNMP MIB-2 agent. For port/mac collection, Cisco, Juniper and HP equipements are supported.

The metro application is made up of two main programs:

  • metrocreatedb which creates metrology sensors based on the metrology collection points defined in the Netmagis database, resulting from equipement configuration fetching by the topo package.
  • metropoller which performs data collection for each sensor

The first part of this section describes the configuration of the metro package.

  • Configure .../etc/netmagis.conf on the metro server:

    netmagis.conf directives
    DirectiveDescription
    metrodatadirMain data directory, where reporting, logs and rrdtool databases are stored (in sub-directories)
    rrdtoolPathname to the rrdtool command
    rrddbDirectory where rrdtool databases are stored. Reserve enough disk space for this directory. It is usually a sub-directory of metrodatadir but it can be a separate directory if you need more disk space.

    FIXME: remove and keep only metrodatadir

    sensorsurvivalGrace period (in days) before sensor removal. This is needed when a sensor is no longer updated by topo (this can be caused by a volontary removal or a bad configuration of the equipment). So for safety reasons, the polling is really stopped on this sensor only after this grace period.
    pollergroups FIXME: provide description of directive first, and explanation after.

    Sensors sharing the same characteristics can be put in the same group:

    • the running time is short, so running them in the same group allows a high level of parallelization ; for example, the traffic collection sensors are usually put in the same group
    • the sensors are specialized and the SNMP collection takes more time
    We recommand to start with the grouping provided in the default configuration and refine the configuration afterwards.

    You must declare all your groups with the directive "pollergroups". Names of the different groups are separated by a ",".

    FIXME: groups should be separated by a space instead of a comma

    gpopt_<pollergroupname>The gpopt_* directives define which sensor types are associated with each group defined with the directive "pollergroups".
    • If no sensor type is specified, all sensor types will be executed by the poller group.
    • If you want that a poller group collects only some of sensor types, write the list of type names separated by a "," to the "gpopt_GROUP" directive.
      Example: gpopt_GP2 ipmac,portmac.cisco
    • If you want that a poller group collects all sensor types except "ipmac" and "portmac.cisco", write a "!" before each type name of sensor separated by a "," to the "gpopt_GROUP" directive.
      Example: gpopt_GP1 !ipmac,!portmac.cisco
    gpnbproc_<pollergroupname>Set the number of processes launched simultaneously for each poller group. The data collection task is shared between the different processes which run in parallel. You can raise the number of processes to reduce the total execution time for all sensors.
    ifspeed_<ifaceprefixname>Set the speed (in Mbit/s) associated to the prefix name of an interface type. This configuration is useful to tune the maximum values of rrd databases when the SNMP ifspeed values collected on the interfaces are not reliable.
    For example: ifspeed_Dot11Radio 300
    will limit the maximum speed for all interfaces called "Dot11Radio..." (cisco Aironet radio interface) to 300 Mb/s.
    pollerlogfacilitySet the syslog facility for data collectors. Modify your syslog configuration accordingly. Example for syslog-ng:
        filter f_local0 { facility(local0); };
        destination d_pollerlog { file(".../metro/log/poller.log"); };
        log { source(s_src); filter(f_local0); destination(d_pollerlog); };
    majdblogfacilitySet the syslog facility for sensor creation. Modify your syslog configuration accordingly. Example for syslog-ng:
        filter f_local1 { facility(local1); };
        destination d_majdblog { file(".../metro/log/majdb.log"); };
        log { source(s_src); filter(f_local1); destination(d_majdblog); };
    pollerlogptSet the default syslog level. (FIXME: change directive to pollerloglevel)
    metrohostSet to the server name hosting the metro package ; if metro is installed on the same server as the www package set it to "localhost".
  • Activate the creation of new sensors and collection of data. For this, you must create a user that will run all the metro programs and make this user the owner of all the metrodatadir and rrddir directory trees. Let us call this user m.

    Install the sample crontab (.../share/examples/netmagis/crontab.metro) in your default cron system. This crontab should be run by the user m.

    After the fisrt execution of .../bin/metrocreatedb, polling can start.

  • Allow access from the www server to the metro webservice: NB: the www host must be trusted by the metro server. The access to the webservice is not authentified.
    Copy the file .../share/examples/apache-metro.conf to your apache configuration directory and adapt it to your environment. The line starting with Allow from must have the IP address of the server running the www package. If www and metro both run on the same host, just put localhost.

The metro package is now ready to work. You can create sensors in the equipment configurations:

  • To activate a traffic sensor on an equipment interface, the only thing to do is to add information in the interface description on the equipment, as you have done with topo package.

    This information is the string M concatened with the name of the sensor. You can use any unique name for a sensor. For example, Netmagis authors use the link number (used with the topo package) associated with the name of the equipement (since a link number is present on each end of the link), but you can use your own convention.

    For example:

    EquipmentExample
    Juniper ge-0/1/0 {
        description "core link <L233 M233.eqname>";
    }
    ge-0/1/1 {
        description "edge port <X Meqname.desc>";
    }
    Cisco interface GigabitEthernet0/1
        description core link <L233 M233.eqname>
    interface GigabitEthernet0/2
        description edge port <X Meqname.desc>
    HP interface A1
        name "core link <L233 M233.eqname>"
    interface A2
        name "edge port <X Meqname.desc>"

  • To collect ip-mac and port-mac session, you need to enable the mac package in the application configuration in the Admin web interface.
    To activate an ipmac sensor in your equipement configuration, you have to add information in the SNMP sysLocation description.

    This information is just the letter I, bracketed with "<" and ">" added to syslocation string.

    Important: The sensor works only if routing is enabled.

    EquipmentExample
    Juniper snmp {
        location "hi, I'm here <I>";
    }
    Cisco snmp-server location hi, I'm here <I>
    HP snmp-server location "I'm here <I>"

  • To activate a portmac sensor in your equipement configuration, it is the same process is an ipmac sensor; just use <P> (instead of <I>).

    Important note: if you want activate both ipmac and portmac sensors on the same equipment, insert the string "<I P> in the sysLocation description.

Metro data descriptions

The Metro package generates saveral datas into the following files or directories :

  • ../var/netmagis/metro/poller/
    This Directory contains the following files :
    • metro.sensor : list of parameters of all sensors for data collection. This file is generated automatically by metrocreadb.
    • metro.quarantine : lists all the sensors which are no longer updated by topo. The sensors are definitively removed after a grace period.
    • metro.plugins : this file is reserved to be manually updated in order to execute plugins or sensors which are not automaticaly activated.
  • ../var/netmagis/metro/rrddb/
    The rrdtool databases are stored in this directory.
  • ../var/netmagis/metro/log/poller.log
    You can find in this log file all the messages generated by sensors collection and plugins execution.
  • ../var/netmagis/metro/log/syncmetro.log
    All the informations about the creation, deletion of sensors are in this log file.
  • ../var/netmagis/metro/report/
    This directory contains the results of the ipmac and portmac collection before storing them in the mac database.
  • ../var/netmagis/metro/cache/
    This directory contains the caches files created by the sensors collection. By example, the (ip,interface name,SNMP interface index) relation.
  • ../var/netmagis/metro/lock
    This Directory contains the locks files created by some scripts (metrocreatedb, plugin-ipmac or plugin-portmac) in order to avoid multiple executions of the same script in the same time. Important : If a crash of a script occurs, the lock file must be removed.
Netmagis license Netmagis Web site