HowTo install reconnoiter on Debian Lenny
Ever since I sow the oscon presentation of reconnoiter I wanted to check it out and play with it. Yesterday, I finally had some time to do this and thought it would be a good idea to document it as a short howto. Most of the infos I used are from the readme (BUILDING), the wiki and the excellent writeup of Thomas Dudziak on how to install reconnoiter on ubuntu.
The daemons noitd and stratcond are written in C, and the database used is postgressql, while the web interface is written in php. We will need to install a few dependencies to be able to compile noitd/stratcond:
Dependencies
apt-get install autoconf build-essential \
zlib1g-dev uuid-dev libpcre3-dev libssl-dev libpq-dev \
libxslt-dev libapr1-dev libaprutil1-dev xsltproc \
libncurses5-dev python libssh2-1-dev libsnmp-dev \
sun-java6-jdk
In case you don’t have already installed svn, we will need that also (to download the sources):
apt-get install subversion
Compiling
Next we will download the source and compile it. I used the trunk version bellow, but you might want use a stable tag (Urskek is the latest when writing this):
cd /usr/local/src/
svn co https://labs.omniti.com/reconnoiter/trunk reconnoiter
cd reconnoiter
autoconf
./configure
make
make install
Database
We need to instal PostgreSQL 8.4 and this is not available in lenny; the simplest way we can do this is to use the backports package. To enable the backports sources add this line
deb http://www.backports.org/debian lenny-backports main contrib non-free
to your /etc/apt/sources.list and run apt-get update to refresh your apt indexes. And finally add the backports key into apt:
apt-get install debian-backports-keyring
Now we can install postgres 8.4 using:
apt-get -t lenny-backports install postgresql
Next we’ll need to edit /etc/postgresql/8.4/main/pg_hba.conf and replace this line
local all all identwith
local all all trust
and restart postgres:
/etc/init.d/postgresql-8.4 restart
to allow local access without password.
Next we can install the dbs, using:
su postgres
cd /usr/local/src/reconnoiter/sql
psql
\i scaffolding.sql
\q
DB crontab
You can find a sample crontab in the sql folder that needs to be customized for our particular setup (/opt/psql835/bin -> /usr/bin/), and it should look like this:
su postgres
crontab -l
# m h dom mon dow command
#rollup jobs
* * * * * /usr/bin/psql -d reconnoiter -U stratcon -c "select stratcon.rollup_metric_numeric(rollup) from metric_numeric_rollup_config order by seconds asc;" >/tmp/rollup.log 2>&1
#cleanup jobs
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_numeric_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_text_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.check_status_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.check_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "delete from prism.saved_graphs where saved = false and last_update/dev/null!!! I would suggest to run the daily crons manually once as they create the needed tables and in case you don’t have them you will receive various errors (I know I lost a lot of time trying to figure out why those were not there). This should be just like:
/usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_text_archive', 'whence', 'day', 7);"
and so on for each of the 5 daily crons from above.
keys
For a production install you will obviously create some real certs, but in my case where I was just testing this out the test certs build by the installer are just fine:
cd /usr/local/src/reconnoiter/test
cp test-*.crt test-*.key /usr/local/etc/
In dealing with the configuration files uuid’s you will find useful the uuidgen program from the uuid-runtime package:
apt-get install uuid-runtime
that will generate a new uuid on each run:
uuidgen
22426aa0-db34-437d-a891-e759ffbdb494
noitd
We will start from the sample configuration:
mv /usr/local/etc/noit.conf.sample /usr/local/etc/noit.conf
and edit/remove it as needed. Be sure to edit the sslconfig part with the keys you copied above. For more details on the config file look on the wiki.
Finally run noitd and see if all is good:
/usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D
If there are no errors stop it, and run it in the background (without -D):
/usr/local/sbin/noitd -c /usr/local/etc/noit.conf
stratcon
For stratcon we are going to use a similar approach:
mv /usr/local/etc/stratcon.conf.sample /usr/local/etc/stratcon.conf
and then customize the config based on our needs. Set the keys in sslconfig section and the db details
<dbconfig>
<host>localhost</host>
<dbname>reconnoiter</dbname>
<user>stratcon</user>
<password>unguessable</password>
</dbconfig> the password is “stratcon” or whatever you set in scaffolding.sql if you customized it (a good idea
).
and finally run it with debuging in the foreground:
/usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf -D
and if all is good (you did ran the db creation task, right?) you can start it in the background without -D and move to the final step:
/usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf
Web interface
Finally the last step is to configure the web interface. For this we need to be sure we have apache/php installed and the pgsql module:
apt-get install apache2 libapache2-mod-php5 php5-pgsql
and to enable the rewrite module
a2enmod rewrite
Next we would copy the web files ui to some place like /var/www/ui and create a vhost for reconnoiter inside /etc/apache2/sites-available:
<VirtualHost *:80>
DocumentRoot /var/www/ui/web/htdocs
<Directory "/">
Options None
AllowOverride None
Order allow,deny
Deny from all
</Directory>
<FilesMatch "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
<Directory "/var/www/ui/web/htdocs/">
php_value include_path /var/www/ui/web/lib
php_value short_open_tag off
Options FollowSymLinks Indexes
AllowOverride All
Order deny,allow
Allow from all
</Directory>
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b" common
ErrorLog /var/www/ui/web/logs/error_log
CustomLog /var/www/ui/web/logs/access_log common
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-httpd-php .php
DefaultType text/plain
RewriteEngine On
RewriteRule ^(/data/.+)$ http://localhost:80$1 [P,L,QSA]
</VirtualHost>
enable the config:
a2ensite reconnoiter
and restart apache to enable it: /etc/init.d/apache2 restart
After this hopefully you can open the web interface and start adding some metrics. My goal was to test this with collectd and if there is interest I could describe the details on how to make reconnoiter work with collectd in another post. let me know…
>
Tags: reconnoiter

16th July 2010, 15:05
Thanks! This is awesome.
25th August 2010, 11:52
Marius, have you already touched the UI? I’ve seen the OSCON presentation and it seems cool and more modern than Zabbix UI for example but how do you rate it? Real time graphs rule but can you for example create new checks/items graphically? Or you must use the CLI (which anyway it’s good looking as well)?
25th August 2010, 16:54
@Vide for my particular project I was interested in adding metrics with collectd from the end servers. It works fine. You can definitely add metrics using the cli, but you will have to automate this to make it useful. I would suggest looking into collectd first as it has already many plugins and makes it much easier to add the metrics automatically as new machines get added to the setup. Hth.
9th March 2011, 08:43
Hi Marius,
if someone gets the following message when running configure :
[...]
checking libcurl cflags… ./configure: line 5693: curl-config: command not found
checking libcurl libs… ./configure: line 5700: curl-config: command not found
[...]
sudo apt-get install libcurl4-gnutls-dev should solve the problem.
‘Hope that helps.
9th March 2011, 09:34
Thanks Jérôme for your update. Hopefully this will help others that have this issue.
8th December 2011, 01:50
Is there some guide for CentOS5?
I got too many ‘ warning: implicit declaration of function ***’ while make…
14th December 2011, 11:56
system info:
—————–
[akshatha@localhost src]$ uname -a
Linux localhost.localdomain 2.6.41.4-1.fc15.x86_64 #1 SMP Tue Nov 29 11:53:48 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
[akshatha@localhost src]$ cat /etc/issue
Fedora release 15 (Lovelock)
I am getting below warnings which acts as errors:
———————————————————————–
[akshatha@localhost reconnoiter]$ make
(cd src && make)
make[1]: Entering directory `/home/akshatha/Documents/reconnoiter/src’
Building version info from git
* version -> 1f06389c4f1d40aaeb808213c44b6beb2d8512af
* symbolic -> branches/master
* version unchanged
make[2]: Entering directory `/home/akshatha/Documents/reconnoiter/src/udns’
…..
…..
- compiling stratcon_realtime_http.c
stratcon_jlog_streamer.c: In function ‘noit_connection_schedule_reattempt’:
stratcon_jlog_streamer.c:244:30: error: variable ‘cn_expected’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c:244:19: error: variable ‘feedtype’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c: In function ‘stratcon_jlog_recv_handler’:
stratcon_jlog_streamer.c:395:29: error: variable ‘feedtype’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c:395:15: error: variable ‘cn_expected’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c: In function ‘noit_connection_ssl_upgrade’:
stratcon_jlog_streamer.c:560:44: error: variable ‘feedtype’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c: In function ‘noit_connection_complete_connect’:
stratcon_jlog_streamer.c:612:70: error: variable ‘feedtype’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c:612:56: error: variable ‘cn_expected’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c: In function ‘noit_connection_initiate_connection’:
stratcon_jlog_streamer.c:690:29: error: variable ‘feedtype’ set but not used [-Werror=unused-but-set-variable]
stratcon_jlog_streamer.c:690:15: error: variable ‘cn_expected’ set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
make[1]: *** [stratcon_jlog_streamer.o] Error 1
make[1]: Leaving directory `/home/akshatha/Documents/reconnoiter/src’
make: *** [all] Error 2
However if I remove all those warning lines, it compiles successfully.
Is it recommanded to remove the warning lines? since google shows that most of the solutions have ended in removing lines which contain warning.
Thanks,
Kiran Patil.
16th December 2011, 21:42
Hello,
I tried rabbitmq and I lost, so I revert back to stomp as mq.
Please send me the document links with which I can configure STOMP, AMQP,
RabbitMQ. I googled and only found RabbitMQ configuration tips.
I am running both noitd and stratcond on a single machine.
Below is the noitd and stratcond deamon output, where we can see the error
messages like,
>> Error writing jlog header over SSL -1 != 20
>> [2011-12-16 14:12:21.527885] bind failed[*]: Address already in use
>> [2011-12-16 14:12:22.538255] MQ connection failed
>> [2011-12-16 14:12:23.589642] STOMP send failed, disconnecting
failed to MQ submit.
I am running noitd in one terminal,
[root at localhost ~]# /usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D
[2011-12-16 14:07:23.163687] Processed 1 includes
[2011-12-16 14:07:23.164217] Found 31 /noit/logs//log stanzas
[2011-12-16 14:07:23.164410] Found 1 outlets for log ‘error’
[2011-12-16 14:07:23.164830] starting asynchronous jlog
writer[6016/0x7f641a386700]
[2011-12-16 14:07:23.164900] starting asynchronous jlog
writer[6016/0x7f6419385700]
[2011-12-16 14:07:23.168981] stopping asynchronous jlog
writer[6016/0x7f641a386700]
[2011-12-16 14:07:23.170705] [2011-12-16 14:07:23.170712] starting
asynchronous jlog writer[6016/0x7f6413fff700]
rlim { 1048576, 1048576 }
[2011-12-16 14:07:23.174251] Found 1 acl stanzas
[2011-12-16 14:07:23.174746] Generic module check_test successfully loaded.
[2011-12-16 14:07:23.175196] Module selfcheck successfully loaded.
[2011-12-16 14:07:23.175862] Module ping_icmp successfully loaded.
[2011-12-16 14:07:23.176253] Module dns successfully loaded.
[2011-12-16 14:07:23.235648] Module snmp successfully loaded.
[2011-12-16 14:07:23.236124] Module ssh2 successfully loaded.
[2011-12-16 14:07:23.236462] Module httptrap successfully loaded.
[2011-12-16 14:07:23.237404] Module varnish successfully loaded.
[2011-12-16 14:07:23.239298] Module http successfully loaded.
[2011-12-16 14:07:23.240706] Module resmon successfully loaded.
[2011-12-16 14:07:23.241609] Module smtp successfully loaded.
[2011-12-16 14:07:23.242888] Module jmx successfully loaded.
[2011-12-16 14:07:23.246370] Found 4 /noit/listeners//listener stanzas
[2011-12-16 14:07:23.369657] stopping asynchronous jlog
writer[6016/0x7f6419385700]
[2011-12-16 14:08:15.575539] noit_listener[control_dispatch] SSL_accept on
fd 16 [stratcon]
[2011-12-16 14:08:15.576704] noit_listener[control_dispatch] SSL_accept on
fd 19 [stratcon]
[2011-12-16 14:08:20.699808] jlog client stratcon disconnected while idle
[2011-12-16 14:08:25.882641] Error writing jlog header over SSL -1 != 20
I am running stratcond in another terminal,
[root at localhost akshatha]# /usr/local/sbin/stratcond -c
/usr/local/etc/stratcon.conf -D
[2011-12-16 14:12:21.517722] Found 6 /stratcon/logs//log stanzas
[2011-12-16 14:12:21.518219] Found 1 outlets for log ‘error’
Found 1 outlets for log ‘error/iep’
Found 1 outlets for log ‘error/eventer’
Found 1 outlets for log ‘error/datastore’
Found 1 outlets for log ‘debug/eventer’
Found 6 /stratcon/logs//log stanzas
[2011-12-16 14:12:21.521237] rlim { 1048576, 1048576 }
Found 6 /stratcon/logs//log stanzas
[2011-12-16 14:12:21.525469] Found 1 acl stanzas
[2011-12-16 14:12:21.525802] Found 3 /stratcon/listeners//listener stanzas
noit_listener(/tmp/stratcon, 0, 1, 5, noit_console, (nil))
noit_listener(*, 80, 1, 5, http_rest_api, (nil))
noit_listener(*, 43191, 1, 5, control_dispatch, (nil))
[2011-12-16 14:12:21.527885] bind failed[*]: Address already in use
[2011-12-16 14:12:21.528405] Generic module stomp_driver successfully
loaded.
[2011-12-16 14:12:21.535269] Loaded 13 uuid -> (sid,storage_node_id)
mappings
[2011-12-16 14:12:21.535760] Loaded 0 storage nodes
[2011-12-16 14:12:21.535940] Generic module postgres_ingestor successfully
loaded.
[2011-12-16 14:12:21.536238] Found 1 /stratcon/noits//noit stanzas
[2011-12-16 14:12:21.536486] initiating to 127.0.0.1
[2011-12-16 14:12:21.536883] Found 1 /stratcon/noits//noit stanzas
[2011-12-16 14:12:21.537698] Found 1 /*/noits//noit stanzas
[2011-12-16 14:12:21.537993] initiating to 127.0.0.1
Pushing durable/storage batch async [noit-test]: [0/3213]
ingesting payload: /var/log/stratcon.persist/
127.0.0.1/noit-test/0/4eeb046d00086f8a.pg
Pushing durable/storage checkpoint [noit-test]: [0/3213]
[2011-12-16 14:12:22.231768] IEPEngine ready…
[2011-12-16 14:12:22.538255] MQ connection failed
[2011-12-16 14:12:22.707518] Creating Statement:
6cc613a4-7f9c-11de-973f-db7e8ccb2e5c
[2011-12-16 14:12:22.721078] Creating Statement:
76598f5e-7f9c-11de-9f5b-ebb4dcb2494e
[2011-12-16 14:12:22.791201] Creating Statement:
ba189f08-7f99-11de-9013-733772d37479
[2011-12-16 14:12:23.351387] Creating Query:
ce6bf8d2-3dd7-11de-a45c-a7df160cba9e
Found 0 /stratcon/iep/queries[@master="stratcond"]//statement stanzas
Found 0 /stratcon/iep/queries[@master="stratcond"]//query stanzas
[2011-12-16 14:12:23.542606] Staged 13/13 remembered checks from (null)
into IEP
[2011-12-16 14:12:23.542818] Loaded all 13 check states.
Pushing transient/iep batch async [noit-test]: [0/3214]
[2011-12-16 14:12:23.589642] STOMP send failed, disconnecting
failed to MQ submit.
Pushing transient/iep checkpoint [noit-test]: [0/3214]
Pushing transient/iep batch async [noit-test]: [0/3215]
Pushing transient/iep checkpoint [noit-test]: [0/3215]
Found 0 /stratcon/iep/queries[@master="stratcond"]//statement stanzas
Found 0 /stratcon/iep/queries[@master="stratcond"]//query stanzas
[2011-12-16 14:12:25.633220] Staged 13/13 remembered checks from (null)
into IEP
[2011-12-16 14:12:25.633327] Loaded all 13 check states.
Pushing transient/iep batch async [noit-test]: [0/3216]
Pushing transient/iep checkpoint [noit-test]: [0/3216]
[2011-12-16 14:12:28.672501] STOMP send failed, disconnecting
failed to MQ submit.
Pushing durable/storage batch async [noit-test]: [0/3216]
ingesting payload: /var/log/stratcon.persist/
127.0.0.1/noit-test/0/4eeb0477000c22cf.pg
Pushing durable/storage checkpoint [noit-test]: [0/3216]
Pushing transient/iep batch async [noit-test]: [0/3217]
Pushing transient/iep checkpoint [noit-test]: [0/3217]
Found 0 /stratcon/iep/queries[@master="stratcond"]//statement stanzas
Found 0 /stratcon/iep/queries[@master="stratcond"]//query stanzas
[2011-12-16 14:12:33.716097] Staged 13/13 remembered checks from (null)
into IEP
[2011-12-16 14:12:33.716257] Loaded all 13 check states.
Thanks,
Kiran.