We are getting closer to the end of the project, so I decided to write a post/tutorial about using syslog-ng with redis and what I’ve implemented so far:
After I've learned some tricks at the beginning of this summer in C by implementing redis client, I started to develop redis destination for syslog-ng. We can collect now log messages and send them to redis via publish or set command, or we can just use incr or hincrby to make statistics.
But first, you’ll have to compile syslog-ng based on this former post.
A short summary without explanation:
git clone -b redis-destination https://github.com/ptichy/syslog-ng-3.4.git
or download: https://github.com/ptichy/syslog-ng-3.4/archive/redis-destination.zip
./autogen.sh && ./configure --prefix <install dir> --enable-debug && make && make install
or if you want just to be sure:
./autogen.sh && ./configure --prefix <install dir> --enable-debug --enable-redis && make && make install
If you want to make statistics in redis from the amount of log messages separated by programs, you can easily use the following config file (if you don’t want to change the default config file, save the following file as mynewconf.conf and run it with sbin/syslog-ng -f mynewconf.conf command):
@version: 3.4
options {
threaded(no);
};
source s_test1 {
system();
tcp( port(514));
};
destination d_redis_hincrby {
redis( command("HINCRBY" "programs" "${PROGRAM}" "1"));
};
log {
source(s_test1);
destination(d_redis_hincrby);
};
options {
threaded(no);
};
source s_test1 {
system();
tcp( port(514));
};
destination d_redis_hincrby {
redis( command("HINCRBY" "programs" "${PROGRAM}" "1"));
};
log {
source(s_test1);
destination(d_redis_hincrby);
};
Of course, you can use another type of sources, you can choose for e.g. tcp(ip(10.1.2.3) port(1999)); instead of unix-stream("/tmp/test-log");
The source statement above receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address. See syslog-ng guide for further information (Chapter 6: Collecting log messages — sources and source drivers).
In this case you will get a programs list with the monitored program names and the amount of them. Hence, if you type hgetall programs in redis client, you’ll get a similar message with the name of programs and the number of collected messages:
redis 127.0.0.1:6379> hgetall programs
1) "xchat"
2) "4"
3) "program2"
4) "10"
1) "xchat"
2) "4"
3) "program2"
4) "10"
If you type redis( set("$PROGRAM" "$MSG")); in config file, then it’s completely equivalent with redis( command("SET" "${PROGRAM}" "${MESSAGE}"));
And finally, see another example from everyday: let's make a stat about browsers that the visitors use: (from apache log, per minutes)
@version: 3.4
options {
threaded(no);
};
source s_apache{
file("/var/log/apache2/access.log");
};
parser p_apache {
csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
"APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
"APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
"APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
flags(escape-double-char,strip-whitespace)
delimiters(" ")
quote-pairs('""[]')
);
};
log {
source(s_apache);
parser(p_apache);
destination(d_redis);
};
destination d_redis {
redis( command("HINCRBY" "${MONTH_ABBREV} ${DAY} ${HOUR}:${MIN}" "${APACHE.USER_AGENT}" "1"));
};
options {
threaded(no);
};
source s_apache{
file("/var/log/apache2/access.log");
};
parser p_apache {
csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
"APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
"APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
"APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
flags(escape-double-char,strip-whitespace)
delimiters(" ")
quote-pairs('""[]')
);
};
log {
source(s_apache);
parser(p_apache);
destination(d_redis);
};
destination d_redis {
redis( command("HINCRBY" "${MONTH_ABBREV} ${DAY} ${HOUR}:${MIN}" "${APACHE.USER_AGENT}" "1"));
};
If you have experience/opinion and want to share with us as a feedback, please send it to balabit syslog-ng mail list and/or me.
