Revision 59: Minor Updates to Documentation

Charles Smutz — Fri, 16 Sep 2011 20:03:04 -0000

Revision 59 has been uploaded which has some small additions to the documentation including clarification on derivative works and references to vortex related contributions from the community.

For those who don't want to download, the diff is below:

--- linux_sensors/vortex/rpm/vortex.README 2011/09/15 13:42:26 2456
+++ linux_sensors/vortex/rpm/vortex.README 2011/09/16 13:54:33 2457
@@ -21,6 +21,9 @@
The libraries to which Vortex links are distributed under the terms of their own licenses.
Please see those libraries for their applicable licenses.

+Applications created to analyze or process data that is outputted from
+Vortex are viewed as separate works and not derivative.
+

Introduction:

@@ -312,6 +315,27 @@
Vortex only supports TCP.

+Community Contributions:
+(The following are provided for information purposes only.)
+
+Securityfu Blog: Vortex IDS - Get Super Snagadocious on Ubuntu, http://securityfu.blogspot.com/2010/02/vortex-ids-get-super-snagadocious-on.html
+ - Introduction to vortex on Ubuntu
+SmuSec Blog: Vortex Howto Series http://smusec.blogspot.com/search/label/vortex%20howto
+ - Series of posts on how to use vortex for various tasks
+Security Onion LiveCD: http://securityonion.blogspot.com/, https://sourceforge.net/projects/security-onion/
+ - Ubuntu based LiveCD contain numerous netwerk security/IDS tools
+StreamDB: Network stream database framework, http://code.google.com/p/streamdb/
+ - Open source project that stores and retrieves streams recontructed by vortex
+Ruminate IDS: Modular System for Network Payload Analysis, http://ruminate-ids.org/
+ - IDS based on vortex focussing on analysis of payload object such as documents
+FreeBSD patch: https://sourceforge.net/news/?group_id=255425&id=298842
+ - Patch disabling some less portable features of vortex, making compilation easier on BSDs
+
+Note: the Smusec blog and Ruminate IDS are independent works of Charles Smutz, one of the authors of vortex.
+
+Newer community contributions can be found on the vortex news page: https://sourceforge.net/news/?group_id=255425
+
+
Improvements:

There are many improvements to be made to vortex. If you make useful improvements, please provide them to the authors so we can consider integrating them. Even improvements to documentation, packaging, etc are both needed and desired.

Compiling Vortex on BSD

Charles Smutz — Thu, 24 Mar 2011 18:59:59 -0000

Martin Holste has contributed the following patch and instructions on compiling vortex on BSD. This disables some features that aren't frequently used and aren't very portable (prioritization and CPU affinity). Hopefully this is helpful to others:

Ok, I got it to compile on FreeBSD 8.1 with the following minor patch to add a BSD define flag to disable CPU affinity:

52,55d51
< #ifdef BSD
< #include <limits.h>
< #define SIZE_MAX _POSIX_SSIZE_MAX
< #else
57d52
< #endif
73d67
< #ifndef BSD
76d69
< #endif
692d684
< #ifndef BSD
709d700
< #endif
775,776c766
<
< #ifndef BSD
---
>
794d783
< #endif
960d948
< #ifndef BSD
978d965
< #endif
1927,1928c1914
<
< #ifndef BSD
---
>
1947d1932
< #endif

I compiled (without BSF) using this command:
gcc -I/usr/local/include -L/usr/local/lib -O3 -o vortex vortex.c -lnids -lpthread -lnet -lpcap -lgthread-2.0 -DBSD

You can download the whole file from my Dropbox here:
http://dl.dropbox.com/u/8259829/vortex.c

StreamDB Project

Charles Smutz — Tue, 01 Feb 2011 20:09:56 -0000

The StreamDB Project has recently been announced: http://code.google.com/p/streamdb/
StreamDB is a high-performance framework for storing network streams. The current version uses Vortex IDS to read the streams from a file or network interface and saves them to an indexed DB and data file....

Updated 2.9.0 tarball

Charles Smutz — Wed, 05 Jan 2011 16:33:24 -0000

A new .tgz for vortex 2.9.0 has been released. This should fix the compilation issue that was occuring for some platforms. This download also includes xpipes, the exclusion of which was an oversight.

The RPMs are unaffected. No changes to functionality were made.

Issues Compiling 2.9.0

Charles Smutz — Wed, 29 Dec 2010 15:50:04 -0000

Some users are reporting trouble compiling vortex 2.9.0 on some platforms, including ubuntu. It appears the easiest fix is to change the include from <linux/limits.h> to <limits.h>. It should also be noted that people who are interested in performance should consider enabling optimization in their compiler. The RPM versions are compiled with the red hat default flags which include -O2.

The following demonstrates and initial failed attempt to compile vortex 2.9.0, the modification proposed, and then testing of the new version:

securityonion@securityonion:/tmp/vortex_testing$ echo "Thanks to Doug Burks for SecurityOnion"
Thanks to Doug Burks for SecurityOnion
securityonion@securityonion:/tmp/vortex_testing$ tar zxf vortex-2.9.0.tgz
securityonion@securityonion:/tmp/vortex_testing$ cd vortex-2.9.0/
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ gcc vortex.c -lnids -lpthread -Wall -O2 -o vortex
vortex.c: In function ‘print_stats’:
vortex.c:736: error: ‘UINT_MAX’ undeclared (first use in this function)
vortex.c:736: error: (Each undeclared identifier is reported only once
vortex.c:736: error: for each function it appears in.)
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ echo "fail :("
fail :(
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ cat vortex.c | sed 's/linux\/limits.h/limits.h/' > vortex_ubuntu.c
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ gcc vortex_ubuntu.c -lnids -lpthread -Wall -O2 -o vortex
vortex_ubuntu.c: In function ‘flow_name’:
vortex_ubuntu.c:567: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:567: note: initialized from here
vortex_ubuntu.c:569: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:569: note: initialized from here
vortex_ubuntu.c:575: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:575: note: initialized from here
vortex_ubuntu.c:577: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:577: note: initialized from here
vortex_ubuntu.c:582: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:582: note: initialized from here
vortex_ubuntu.c:584: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
vortex_ubuntu.c:584: note: initialized from here
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ diff -uNr vortex.c vortex_ubuntu.c
--- vortex.c 2010-12-15 09:32:07.000000000 -0500
+++ vortex_ubuntu.c 2010-12-29 10:12:35.770762272 -0500
@@ -49,7 +49,7 @@
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
-#include <linux/limits.h>
+#include <limits.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/time.h>
@@ -809,7 +809,7 @@
//Used to take all the connection data, provide it to consumers, free, and close
void dump_stream(struct conn_param *a_conn)
{
- //taken from linux/limits.h
+ //taken from limits.h
char temp_filename[PATH_MAX];

//these vars only used for debugging
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ sudo ./vortex -i eth0 -e -k -l &
[1] 3372
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ wget http://ruminate-ids.org
--2010-12-29 10:31:26-- http://ruminate-ids.org/
Resolving ruminate-ids.org... 66.173.221.158
Connecting to ruminate-ids.org|66.173.221.158|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4156 (4.1K) [text/html]
Saving to: `index.html.5'

100%[===================================>] 4,156 --.-K/s in 0.002s

2010-12-29 10:31:26 (2.41 MB/s) - `index.html.5' saved [4156/4156]

tcp-1-1293636686-1293636686-c-4463-192.168.159.134:49840s66.173.221.158:80
tcp-1-1293636686-1293636686-c-4463-192.168.159.134:49840c66.173.221.158:80
securityonion@securityonion:/tmp/vortex_testing/vortex-2.9.0$ sudo skill -9 -c vortex

Version 2.9.0

Charles Smutz — Sat, 25 Dec 2010 01:55:40 -0000

Version 2.9.0 is available for download. This release includes various fixes, huge performance improvements, better debugging/error reporting/statistics, removal of arbitrary 2GB limits, and even a new feature when replaying from capture files where output files have their timestamps set the to the appropriate times from the pcap.

Happy Holidays!

Changlog is as follows:

* Wed Dec 15 2010 Charles Smutz <opensource.tools.security@lmco.com> 2.9.0-57
- Rebuild for public distribution

* Tue Dec 14 2010 Charles Smutz <opensource.tools.security@lmco.com> 2.9.0-56
- Significant performance improvements (lower CPU use by ~300%) by making calls to realloc less frequently (-x to tune)
- Set timestamps of output files to match timestamps from pcap (-d to disable)
- Remove arbitrary limit (2 GB) on stream collection sizes, now depends on system
- Wrap 32bit pcap counters with 64bit counters
- Fix cpu locking/affinity and simplify interface
- Addition of error counters for file/IO and memory errors

* Fri Jul 23 2010 Charles Smutz <opensource.tools.security@lmco.com> 2.8.1-55
- Added more debug statement for tcp callback and idle queue

* Thu Jul 22 2010 Charles Smutz <opensource.tools.security@lmco.com> 2.8.1-54
- Added debug statements for connection open, close, and write

* Wed May 19 2010 William Hoyt <opensource.tools.security@lmco.com> 2.8.1-52
- set init script to send messages to /dev/null

Vortex Included on Security Onion LiveCD

Charles Smutz — Mon, 18 Oct 2010 10:57:42 -0000

Vortex has been included on Security Onion Live, http://code.google.com/p/security-onion/, an ubuntu based LiveCD devoted to IDS. See http://securityonion.blogspot.com/2010/10/security-onion-live-20101010-edition.html for the announcement.

SmuSec continues Vortex Howto Series

Charles Smutz — Thu, 08 Apr 2010 13:39:17 -0000

SmuSec has posted another entry in their vortex howto series: http://smusec.blogspot.com/search/label/vortex%20howto

RPMs for Libnids 1.24 released

Charles Smutz — Thu, 08 Apr 2010 13:37:27 -0000

RPMs for vortex built against libnids 1.24 are available for download. Look for the RPMs with revision number 51. Previous RPMs, including revision 49, were all built against 1.23. Hence, the difference between vortex-2.8.1-51.el5.x86_64.rpm and vortex-2.8.1-49.el5.x86_64.rpm is the version of libnids required.

Blog Posts on Vortex

Charles Smutz — Wed, 24 Mar 2010 12:34:16 -0000

Vortex has found itself in a few blogs lately.
Securityfu has an introduction to and overview of vortex. http://securityfu.blogspot.com/2010/02/vortex-ids-get-super-snagadocious-on.html
SmuSec is running a series of howtos on vortex. http://smusec.blogspot.com/2010/03/vortex-howto-series-network.html

Recent posts to news