New suricata release 0.8.2

New suricata release! Have a look at the new features and changelog!

The OISF development team is proud to introduce the 3rd beta release of
Suricata, the Open Source Intrusion Detection and Prevention engine. The
first release candidate is currently scheduled for early May, but check
https://redmine.openinfosecfoundation.org/projects/roadmap/suricata for
the up to date schedule!

Get the new release here:
http://www.openinfosecfoundation.org/download/suricata-0.8.2.tar.gz

New features

- Support for the following keywords: detection_filter, http_client_body
- The HTTP parser can now set server personalities
- threshold.config support
- The experimental CUDA code now also works on x86_64
- IP address only rules for IPv6 are now supported as well
- Suricata can now write a pid file (pass --pidfile )
- A fuzzer script was added to the code base
- Policy lookup for defrag module

Improvements

- Much better average and worstcase performance in the detection engine
- Memory footprint was reduced
- More validation at signature loading stage
- Libnet 1.1 is now optional
- Negated uricontent and http_cookie matching is now supported
- Lots of fixes of issues found by Valgrind's DRD, CLANG and Parfait.
- Threads are named now in "top" (Linux only atm).
- Unified1 file handling is improved

Bugs fixed

Many :)
Several segmentation faults, upgrading is highly recommended.

See
https://redmine.openinfosecfoundation.org/projects/suricata/issues?fixed_version_id=6&set_filter=1&status_id=c

Known issues & missing features

We have made significant progress towards reaching our first full
(non-beta) release of Suricata. Your feedback is always important to us
and we appreciate your time and effort. As always, we are doing our
best to make you aware of continuing development and items within the
engine that are not yet complete. With this in mind, please notice the
list we have included of known items we are working on.

- Using the http_cookie keyword seems to cause a match on all packets.
- Currently we dont' support the dce option for byte_test and byte_jump.
- Stream reassembly is currently only performed for app-layer code.
- Inconsistent time stamps in http log file due to handling & updating
of the http state.
- DCE/RPC over udp is not currently supported.
- dce_stub_data does not respect relative modifiers.
- Engine does not work properly on big endian platforms.
- Time based stats are not calculated correctly.

See https://redmine.openinfosecfoundation.org/projects/suricata/issues
for an up to date list and to report new issues.

Improved version of pcap2rawc

This version split each packet into layers, and each layer is pointed from an array of name like rawpktLayer_name[]

-- pcap2RawCLayers.py --
#!/usr/bin/python
try:
    from scapy.all import *
except:
    print "old way..."
    from scapy import *

import sys
from binascii import *

if len(sys.argv) ==2:
    print "Parsing "+str(sys.argv[1])
else:
    print "Usage: python "+sys.argv[0]+" file.pcap"
    exit(10)

pcap=rdpcap(sys.argv[1])
out=file(sys.argv[1]+".rawc","w")

out.write("// Generated from pcap2RawCLayers.py\n")

i=0
buff=""
arrays=[]

for p in pcap:
    print "// packet "+str(i)+": ***"

    while p.payload and len(p.payload) > 0:
        q=p.copy()
        q.payload = ''
        bytes=len(q)
        strbyte=""

        for j in range(0,bytes):
            if j %8 ==0:
                strbyte = strbyte +"\n        "
            strbyte = strbyte + "0x" + str(hexlify(str(q)[j]))
            if j < bytes-1:
                if j+1 %8:
                    strbyte= strbyte + ","
                else:
                    strbyte= strbyte + ", "

        rawpkt="    rawpkt" + str(q.name) + "["+str(i)+"] = {" + strbyte + " }; /* end rawpkt" + str(p.name) +"["+ str(i) +"] */\n"
        p=p.payload
        arrays.append("rawpkt" + str(q.name))
        buff = buff + rawpkt

    if not p.payload and p.load:
        q=p.copy()
        bytes=len(q.load)
        strbyte=""

        for j in range(0,bytes):
            if j %8 ==0:
                strbyte = strbyte +"\n        "
            strbyte = strbyte + "0x" + str(hexlify(str(q.load)[j]))
            if j < bytes-1:
                if j+1 %8:
                    strbyte= strbyte + ","
                else:
                    strbyte= strbyte + ", "

        rawpkt="    rawpktPayload["+str(i)+"] = {" + strbyte + " }; /* end rawpktPayload["+ str(i) +"] */\n"
        p=p.payload
        arrays.append("rawpktPayload")

    i=i+1
    buff = buff + rawpkt

declares=""
for l in arrays:
    declares = declares + "    uint8_t *"+ l +"["+str(i)+"];\n"
       
filebuff = declares+ "\n"+ buff + "\n"
out.write(filebuff)
out.close()

print filebuff
print "//"+ str(i) +" packets written in "+sys.argv[1]+".rawc"

Rule2Alert

Hi, some updates.
I have started a new project with Josh Smith and Will Metcalf. Talking about scapy Josh told me if I would like to get involved in the project, and we created a google project called "rule2alert".


It's written in python and use scapy. The purpose of this project is to read snort compatible rules and write a pcap with packets that should match the rules. This can be used later to test NIDS like suricata and snort and detect problems on the detection plugins. Of course this needs a lot of development, for each rule keyword, so we don't think we will generate payloads for all the rules, but also for the majority of them. At the moment we deal with content and content modifiers, and also content hexa data specification, and flow options, performing TCP 3 way handshakes. The next steps will be focussed on http protocol options, like uricontent.

We hope it will be a good QA tool. If you would like to get involved, feel free to get in touch.

I must feel lucky...

Not too much people can say that saw this line:
"the 'impossible' happened"...
lol!


valgrind: m_scheduler/scheduler.c:1144 (vgPlain_scheduler): the 'impossible' happened.
valgrind: VG_(scheduler), phase 3: run_innerloop detected host state invariant failure
==28716== at 0x3802A7AC: report_and_quit (m_libcassert.c:140)
==28716== by 0x3802AABA: vgPlain_assert_fail (m_libcassert.c:205)
==28716== by 0x3804E283: vgPlain_scheduler (scheduler.c:1165)
==28716== by 0x38060CB0: run_a_thread_NORETURN (syswrap-linux.c:89)

Ante la inclusión en el Anteproyecto de Ley de Economía Sostenible

Today in Spain ... The gov want to be able to close a webpage without a court procedure if they "consider that there's" Author rights infringed.

Ante la inclusión en el Anteproyecto de Ley de Economía sostenible de modificaciones legislativas que afectan al libre ejercicio de las libertades de expresión, información y el derecho de acceso a la cultura a través de Internet, los periodistas, bloggers, usuarios, profesionales y creadores de internet manifestamos nuestra firme oposición al proyecto, y declaramos que…

1.- Los derechos de autor no pueden situarse por encima de los derechos fundamentales de los ciudadanos, como el derecho a la privacidad, a la seguridad, a la presunción de inocencia, a la tutela judicial efectiva y a la libertad de expresión.

2.- La suspensión de derechos fundamentales es y debe seguir siendo competencia exclusiva del poder judicial. Ni un cierre sin sentencia. Este anteproyecto, en contra de lo establecido en el artículo 20.5 de la Constitución, pone en manos de un órgano no judicial -un organismo dependiente del ministerio de Cultura-, la potestad de impedir a los ciudadanos españoles el acceso a cualquier página web.

3.- La nueva legislación creará inseguridad jurídica en todo el sector tecnológico español, perjudicando uno de los pocos campos de desarrollo y futuro de nuestra economía, entorpeciendo la creación de empresas, introduciendo trabas a la libre competencia y ralentizando su proyección internacional.

4.- La nueva legislación propuesta amenaza a los nuevos creadores y entorpece la creación cultural. Con Internet y los sucesivos avances tecnológicos se ha democratizado extraordinariamente la creación y emisión de contenidos de todo tipo, que ya no provienen prevalentemente de las industrias culturales tradicionales, sino de multitud de fuentes diferentes.

5.- Los autores, como todos los trabajadores, tienen derecho a vivir de su trabajo con nuevas ideas creativas, modelos de negocio y actividades asociadas a sus creaciones. Intentar sostener con cambios legislativos a una industria obsoleta que no sabe adaptarse a este nuevo entorno no es ni justo ni realista. Si su modelo de negocio se basaba en el control de las copias de las obras y en Internet no es posible sin vulnerar derechos fundamentales, deberían buscar otro modelo.

6.- Consideramos que las industrias culturales necesitan para sobrevivir alternativas modernas, eficaces, creíbles y asequibles y que se adecuen a los nuevos usos sociales, en lugar de limitaciones tan desproporcionadas como ineficaces para el fin que dicen perseguir.

7.- Internet debe funcionar de forma libre y sin interferencias políticas auspiciadas por sectores que pretenden perpetuar obsoletos modelos de negocio e imposibilitar que el saber humano siga siendo libre.

8.- Exigimos que el Gobierno garantice por ley la neutralidad de la Red en España, ante cualquier presión que pueda producirse, como marco para el desarrollo de una economía sostenible y realista de cara al futuro.

9.- Proponemos una verdadera reforma del derecho de propiedad intelectual orientada a su fin: devolver a la sociedad el conocimiento, promover el dominio público y limitar los abusos de las entidades gestoras.

10.- En democracia las leyes y sus modificaciones deben aprobarse tras el oportuno debate público y habiendo consultado previamente a todas las partes implicadas. No es de recibo que se realicen cambios legislativos que afectan a derechos fundamentales en una ley no orgánica y que versa sobre otra materia.

Este texto se publica multitud de sitios web. Si estás de acuerdo, publícalo también en tu blog, twittéalo, facebookéalo.

Profiling with shark on Mac OS X - Snow Leopard

Mac OS X has a simple but useful tool called "Shark", that allows you to profile applications of different languages. The following are the steps to do a simple profile:

How to do profiling with Mac OS X.

The version I'm running is Mac OS X 10.6.1 (Snow Leopard). I have installed the Development toolkit X-code and the typical tools like autoconf, automake, etc. You may also need to run it as root depending on your configuration. Mac OSX provide a great program for profiling on different languages. It's called "Shark".

1. To run it type cmd+space bar and type Shark (Spotlight search is useful).

2. Now that you have Shark opened select "Launch" at the last combo (It list all the running processes, and you could also profile everything, but we are interested only on our app).

3. Now you can click on "Start", and a new dialog box will ask you for the executable path, and the arguments. Set them as you need. You may also set environment vars for debugging or whatever.

4. Now just click Ok and wait to see the results.

Now that the results are generated, you can order them by each of the columns "Self" and "Total" will indicate you the cost of each part, and this will give you the most critical sections. They can be simple code, but maybe it's executed a lot of times. So even appering very simple, if you can improve that part just a bit, you will win on performance!

Now you can check some features of Shark. You can expand in a hierarchical list, the traces of the function calls, and also the self/total cost% will be splitted into each function call, so you may see a grouped call with 24% but expanding it, you may have one call that use 10% another 5% and another 9%. This way you can get a draft in your mind of the execution flow. It is also able to show the profile by threads, and the call stack as "Heavy", "Tree" or both

If you don't have the source code, don't worry. Shark disassemble the application for you, lol!

So now you may think you need to check the code of that part. Shark will also help you on that. Double click on the function call and the code will be displayed automatically. The cpu cost will appear with each important line.

The rest is at your logic. You may need to determine where a loop can be costing a lot, or a performance improvement can be done. You can also have a look at the generated charts. The id of the cpu can be also specfied here.

pcap2rawc.py

This is another script that maybe help someone coding things with raw c Packets. It take a pcap file and create a file with all that packets declared as c arrays.

#!/usr/bin/python
# File: pcap2rawc.py
# Pablo Rincon Crespo [pablo.rincon.crespo at gmail]
#
try:
    from scapy.all import *
except:
    print "old way..."
    from scapy import *

import sys
from binascii import *

if len(sys.argv) ==2:
    print "//Parsing "+str(sys.argv[1])
else:
    print "Usage: python "+sys.argv[0]+" file.pcap"
    exit(10)

pcap=rdpcap(sys.argv[1])
out=file(sys.argv[1]+".rawc","w")

out.write("// Generated from pcap2rawc.py\n")

i=0
for p in pcap:
    i=i+1
    print "//processing packet "+str(i)+": ***"
    print p.command()
    bytes=len(p)
    strbyte=""
    for j in range(0,bytes):
        if j %8 ==0:
            strbyte = strbyte +"\n        "
        strbyte = strbyte + "0x" + str(hexlify(str(p)[j]))
        if j < bytes-1:
            if j+1 %8:
                strbyte= strbyte + ","
            else:
                strbyte= strbyte + ", "
    rawpkt="    uint8_t rawpkt" + str(i) + "[] = {" +strbyte + " }; /* end rawpkt" + str(i) +" */\n"
    print rawpkt

    out.write(rawpkt + "\n")

out.close()
print "//"+ str(i) +" packets written in "+sys.argv[1]+".rawc"