tag:blogger.com,1999:blog-74803370584716903712024-02-20T13:37:49.108+01:00gsoc 2013Syslog-ng: redis destinationUnknownnoreply@blogger.comBlogger5125tag:blogger.com,1999:blog-7480337058471690371.post-32132293262323125082013-09-26T00:04:00.000+02:002013-09-26T00:04:25.383+02:00The road so far<br />
We are getting closer to the end of the project, so I decided to write a post/tutorial about using syslog-ng with redis and what I’ve implemented so far:<br />
After I've learned some tricks at the beginning of this summer in C by implementing redis client, I started to develop redis destination for syslog-ng. We can collect now log messages and send them to redis via <i>publish</i> or <i>set</i> command, or we can just use <i>incr</i> or <i>hincrby</i> to make statistics.<br />
But first, you’ll have to compile syslog-ng based on this <a href="http://tichygsoc.blogspot.hu/2013/06/compiling-syslog-ng-ose-35-on-ubuntu.html" target="_blank">former post</a>.<br />
<br />
A short summary without explanation:<br />
<div id="terminal">
git clone -b redis-destination <a href="https://github.com/ptichy/syslog-ng-3.4.git">https://github.com/ptichy/syslog-ng-3.4.git</a></div>
or download: <a href="https://github.com/ptichy/syslog-ng-3.4/archive/redis-destination.zip">https://github.com/ptichy/syslog-ng-3.4/archive/redis-destination.zip</a><br />
<br />
<div id="terminal">
./autogen.sh && ./configure --prefix <install dir> --enable-debug && make && make install</div>
or if you want just to be sure:<br />
<div id="terminal">
./autogen.sh && ./configure --prefix <install dir> --enable-debug --enable-redis && make && make install </div>
<br />
If you want to make statistics in redis from the amount of log messages separated by programs, you can easily use the following config file (if you don’t want to change the default config file, save the following file as mynewconf.conf and run it with <i>sbin/syslog-ng -f mynewconf.conf</i> command):<br />
<br />
<div id="terminal">
<span style="font-size: x-small;">@version: 3.4</span><br />
<span style="font-size: x-small;">options {</span><br />
<span style="font-size: x-small;"> threaded(no);</span><br />
<span style="font-size: x-small;">}; </span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">source s_test1 {</span><br />
<span style="font-size: x-small;"> system();</span><br />
<span style="font-size: x-small;"> tcp( port(514));</span><br />
<span style="font-size: x-small;">};</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">destination d_redis_hincrby {</span><br />
<span style="font-size: x-small;"> redis( command("HINCRBY" "programs" "${PROGRAM}" "1"));</span><br />
<span style="font-size: x-small;">};</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">log {</span><br />
<span style="font-size: x-small;"> source(s_test1);</span><br />
<span style="font-size: x-small;"> destination(d_redis_hincrby);</span><br />
<span style="font-size: x-small;">};</span></div>
<span style="font-size: x-small;"> </span>
<br />
Of course, you can use another type of sources, you can choose for e.g. <i>tcp(ip(10.1.2.3) port(1999));</i> instead of <i>unix-stream("/tmp/test-log");</i><br />
The source statement above receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address. See <a href="http://www.balabit.com/support/documentation/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/pdf/syslog-ng-ose-v3.4-guide-admin.pdf" target="_blank">syslog-ng guide</a> for further information (Chapter 6: Collecting log messages — sources and source drivers).<br />
In this case you will get a <i>programs</i> list with the monitored program names and the amount of them. Hence, if you type <i>hgetall programs</i> in redis client, you’ll get a similar message with the name of programs and the number of collected messages:<br />
<br />
<div id="terminal">
redis 127.0.0.1:6379> hgetall programs<br />
1) "xchat"<br />
2) "4"<br />
3) "program2"<br />
4) "10"</div>
<br />
<br />
If you type <i>redis( set("$PROGRAM" "$MSG"));</i> in config file, then it’s completely equivalent with <i>redis( command("SET" "${PROGRAM}" "${MESSAGE}")); </i><br />
<br />
And finally, see another example from everyday: let's make a stat about browsers that the visitors use: (from apache log, per minutes)<br />
<br />
<div id="terminal">
<span style="font-size: x-small;">@version: 3.4</span><br />
<span style="font-size: x-small;">options {</span><br />
<span style="font-size: x-small;"> threaded(no);</span><br />
<span style="font-size: x-small;">}; </span><br />
<br />
<span style="font-size: x-small;"><span class="null">source s_apache{</span></span><br />
<span style="font-size: x-small;"><span class="null"> file("/var/log/apache2/access.log");</span></span><br />
<span style="font-size: x-small;"><span class="null">}; </span></span><br />
<br />
<span style="font-size: x-small;">parser p_apache {</span><br />
<span style="font-size: x-small;"> csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",</span><br />
<span style="font-size: x-small;"> "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",</span><br />
<span style="font-size: x-small;"> "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",</span><br />
<span style="font-size: x-small;"> "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")</span><br />
<span style="font-size: x-small;"> flags(escape-double-char,strip-whitespace)</span><br />
<span style="font-size: x-small;"> delimiters(" ")</span><br />
<span style="font-size: x-small;"> quote-pairs('""[]')</span><br />
<span style="font-size: x-small;"> );</span><br />
<span style="font-size: x-small;">};</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">log { </span><br />
<span style="font-size: x-small;"> source(s_apache);</span><br />
<span style="font-size: x-small;"> parser(p_apache);</span><br />
<span style="font-size: x-small;"> destination(d_redis);</span><br />
<span style="font-size: x-small;">};</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">destination d_redis {</span><br />
<span style="font-size: x-small;"> redis( command("HINCRBY" "${MONTH_ABBREV} ${DAY} ${HOUR}:${MIN}" "${APACHE.USER_AGENT}" "1"));</span><br />
<span style="font-size: x-small;">};</span></div>
<br />
If you have experience/opinion and want to share with us as a feedback, please send it to balabit syslog-ng mail list and/or me.<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7480337058471690371.post-7545347469606754712013-07-27T00:38:00.000+02:002013-08-27T01:12:24.761+02:00Redis destinationAfter I've learned some tricks in the last weeks in C by implementing redis client, I started to develop redis destination for syslog-ng.<br />
The first aim was to develop the basis of the plugin, write makefiles and compile it. For this purpose the afsmtp plugin was a good take-off. I wrote the initialization functions, that called by syslog-ng on startup to load the destination. This way I managed to write a test message in redis.<br />
My mentor, Viktor wrote me a tiny test program in python to check the exist of a predetermined key.<br />
It starts the syslog-ng, that writes a given key in redis. The program checks the exist of this key, shutting down syslog-ng and return with a message, if the writing was successful.<br />
<br />
<div id="terminal">
tichy@nb:~/syslog-ng-3.4-install-redis$ sudo ./test.py<br />
Pid of syslog-ng process: 29410<br />
PING: PONG<br />
syslog-ng starting up; version='3.4.2'<br />
syslog-ng shutting down; version='3.4.2'<br />
The test was successful.</div>
<br />
<span style="color: yellow;"><b>Update:</b> </span> I managed to write messages in redis via syslog-ng the following way: I sent a log message with logger command and I got it in redis.<br />
Of course, I have a lot of things to do yet, choose naming method of keys properly, optimizing, etc.<br />
<br />
<b><span style="color: yellow;">Update #2:</span></b> <span style="background-color: white;"></span>Pub/sub implemented, we can send now the log messages to channels separated by program names.<br />
Error handling fixed, so if redis isn't up on program starting or disconnect while running, syslog-ng collects log messages, try to reconnect and resend all messages after connection restored.<br />
<br />
<b><span style="color: yellow;">Update #3:</span></b> Set custom redis command added.<span style="color: yellow;"><span style="color: black;"><br /></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7480337058471690371.post-80963376001837928812013-06-29T20:18:00.000+02:002013-07-17T17:19:33.849+02:00Redis client in CI made a simple <a href="https://github.com/ptichy/redis-learning" target="_blank">redis client</a> in C based on the hiredis example (redis-2.6.14/deps/hiredis/example.c) to demonstrate how to use hiredis.
You can compile it with cc -o redisclient redisclient.c libhiredis.a command or run makefile.
If you want to specify the host from terminal, run ./redisclient 127.0.0.1:6379.
If you don't define, it will use the adress and port in this example.
Type exit to quit.
Supported commands: get, set, del, incr, decr.<br />
<br />
<div id="terminal">
PING: PONG<br />
redis: get key<br />
> value<br />
redis: incr key<br />
> ERR value is not an integer or out of range <br />
redis: set key value 2<br />
> SET: OK<br />
redis: get key<br />
> value 2<br />
redis: set age 21<br />
> SET: OK<br />
redis: decr age<br />
> 20<br />
redis: incr age<br />
> 21<br />
redis: del age<br />
> (null)</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7480337058471690371.post-17240403405800115142013-06-25T22:12:00.002+02:002013-08-01T23:28:01.927+02:00Compiling syslog-ng OSE 3.4/3.5 on ubuntuDownload the latest version of syslog-ng OSE from the <a href="https://www.balabit.com/downloads/files/syslog-ng/sources/">BalaBit website</a> or from <a href="https://github.com/balabit">GitHub</a>.<br />
<br />
You will need <i>build-essential, dpkg-dev</i>, <i>gcc, glib-dev,</i> <i>flex</i> and <i>bison, libnet, libffi </i>packages.<br />
You can install them with <i>apt-get install</i> command or with synaptic.<br />
<b><span style="color: yellow;">Hint:</span> </b>You should increase the terminal buffer up to 1000 lines. <br />
<br />
Then download the latest version of syslog-ng (if you want to download from github through terminal, git must be installed on your system; you can install it with: )<br />
<div id="terminal">
sudo apt-get install git </div>
then<br />
<div id="terminal">
git clone https://github.com/balabit/syslog-ng-3.4.git</div>
<br />
However, in some descriptions you can read: "Download and install the latest version of <i>eventlog</i>", it's enough to install <i>libevtlog-dev </i>and<i> libevtlog0.</i><br />
<br />
Then <i>cd </i>to your syslog-ng-3.4 directory and run<br />
<div id="terminal">
dpkg-checkbuilddeps</div>
to check build dependencies and conflicts.<br />
<br />
I needed to install the following packages on ubuntu 13.04:<br />
<i>libtool, debhelper, libevtlog-dev, libevtlog0, libevtlog0-dbg, libnet1-dev, libglib2.0-dev, libdbi0-dev, libssl-dev. </i><br />
<br />
I got an error message during compiling, so you should install <i>xsltproc </i>before next step.<i></i><br />
<br />
Type to compile syslog-ng with debug option (you can find more options in <a href="http://www.balabit.com/support/documentation/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/pdf/syslog-ng-ose-v3.4-guide-admin.pdf">syslog-ng ose 3.4 admin guide</a>):<br />
<div id="terminal">
./autogen.sh && ./configure --prefix <<i>install dir></i> --enable-debug && make && make install</div>
where <i><install dir> </i>is the destination folder.<br />
<br />
<b><span style="color: yellow;">Note:</span></b> If you ge<span style="font-size: xx-small;"><span style="font-size: small;">t <i>permission denied </i>messages,<i> </i>you need to </span></span>change the user ownership.<br />
<div id="terminal">
<span style="font-size: xx-small;"><span style="font-size: small;">chown -R <user> syslog-ng-3.4 </span></span></div>
<br />
<u>Sources:</u><br />
Syslog-ng OSE 3.4 admin guideUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-7480337058471690371.post-5200269950413432312013-06-12T14:05:00.000+02:002013-08-08T01:43:24.732+02:00Short introduction and first stepsHi,<br />
<br />
I'm tichy and I participate in gsoc 2013 with project name syslog-ng: redis destination.<br />
It's about to develop a new syslog-ng destination plugin, one that can add or
change data in redis and have a reliably database with the function of
getting real-time counters, statistics or in other cases caching.<br />
I started the zero phase of the project:<br />
<ul>
<li>I read tutorials and documentation about redis, syslog-ng</li>
<li>compiled the syslog-ng 3.4 on ubuntu</li>
<li>made a github account for <a href="https://github.com/ptichy/syslog-ng-3.4" target="_blank">code hosting</a></li>
</ul>
<b>Some thoughts about me</b><br />
I provided the hungarian
translation of pigeonplanner and I think this project is a very good
opportunity to get involved in open source development.<br />
I got
programming experience (c based languages) at university level which is a
solid base to start the project, learn and develop something new on a
higher level and get experience, which would be a good reference after
graduating the university.<br />
I decided to write a blog post for each milestone or when I just find a problem, that's worth to write some thoughts about.<br />
I have exams at the university, but I will be free from next week so I can focus all of my power on this project.Unknownnoreply@blogger.com0