Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing msgtype and events? #14

Closed
juju4 opened this issue Sep 18, 2021 · 17 comments
Closed

Missing msgtype and events? #14

juju4 opened this issue Sep 18, 2021 · 17 comments

Comments

@juju4
Copy link

juju4 commented Sep 18, 2021

with laurel 0.1.1 on Ubuntu 18.04 and base laurel configuration

# jq -r 'keys | nth(1)' /var/log/laurel/audit.log | sort | uniq -c | sort -nr | head -20
   2268 SECCOMP
     66 ID
     33 USER_START
     33 USER_END
     33 USER_ACCT
     33 LOGIN
      3 SERVICE_STOP
      3 SERVICE_START

while

# cut -d ' ' -f 7 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  56855 type=SYSCALL
  56855 type=PROCTITLE
  43944 type=PATH
  30833 type=OBJ_PID
  13681 type=EXECVE
   7771 type=SECCOMP
    156 type=USER_END
    156 type=CRED_DISP
    155 type=USER_START
    149 type=USER_ACCT
    149 type=LOGIN
    149 type=CRED_ACQ
     16 type=SERVICE_STOP
     16 type=SERVICE_START
      6 type=USER_CMD
      6 type=CRED_REFR
      3 type=NETFILTER_CFG
# grep 1631967301.772 /var/log/audit/audit.log
type=SYSCALL msg=audit(1631967301.772:1842167): arch=c000003e syscall=59 success=yes exit=0 a0=7f8c1c4c9260 a1=7f8c1c6db4e0 a2=7ffe80304318 a3=9 items=3 ppid=18688 pid=18689 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=3993 comm="check-pid" exe="/usr/bin/env" key="rootcmd"
type=EXECVE msg=audit(1631967301.772:1842167): argc=4 a0="/usr/bin/env" a1="bash" a2="/opt/zeek/share/zeekctl/scripts/helpers/check-pid" a3="21898"
type=PATH msg=audit(1631967301.772:1842167): item=0 name="/opt/zeek/share/zeekctl/scripts/helpers/check-pid" inode=4655459 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1631967301.772:1842167): item=1 name="/usr/bin/env" inode=4152 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1631967301.772:1842167): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=18925 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1631967301.772:1842167): proctitle=2F7573722F62696E2F656E760062617368002F6F70742F7A65656B2F73686172652F7A65656B63746C2F736372697074732F68656C706572732F636865636B2D706964003231383938
# grep 1631967301.772 /var/log/laurel/audit.log
#

SYSCALL is the ones where I would expect process activity
Even if merging some of the different msgtype, same data should be found

@IppSec
Copy link

IppSec commented Sep 18, 2021

I believe you may be excluding the End Of Event (EOE) Records. If you use the Florian Roth auditd template it is excluded by default. It is this line in your /etc/audit/rules.d/audit.conf file.

## Ignore EOE records (End Of Event, not needed)
-a always,exclude -F msgtype=EOE

Just comment out the above, restart the services and you should be good to go

@hillu
Copy link
Collaborator

hillu commented Sep 20, 2021

@juju4 Does this help?

hillu added a commit to hillu/Neo23x0-audit-rules that referenced this issue Sep 20, 2021
While it is true that end-of-event messages are technically not needed
for the regular audit log, auditd will filter them out anyway (cf.
https://github.com/linux-audit/audit-userspace/blob/0f79b322856ad6f40aa7d0291c2ebc4947543057/src/auditd.c#L276).
However, auditd does forward EOE messages to audit dispatcher plugins
such as Laurel which may rely on those mesages being
there. (<threathunters-io/laurel#14>.)
@juju4
Copy link
Author

juju4 commented Sep 20, 2021

Sorry, I believed I had updated.

Yes, it fixed the issue.
But one thing that I was thinking are they are other types required/useful? CWD seems to be not required but useful for omigod per https://techcommunity.microsoft.com/t5/azure-sentinel/hunting-for-omi-vulnerability-exploitation-with-azure-sentinel/ba-p/2764093 and https://github.com/SigmaHQ/sigma/pull/2042/files
This is also excluded in https://github.com/Neo23x0/auditd/blob/master/audit.rules#L70

Still got very different count, so not sure if something else or just the result of laurel merging logs

# jq -r 'keys | nth(1)' /var/log/laurel/audit.log | sort | uniq -c | sort -nr | head -20
   2636 OBJ_PID
   1905 PARENT_INFO
   1275 ID
    687 SECCOMP
     12 USER_START
     12 USER_END
     12 USER_ACCT
     12 LOGIN
      4 PATH
      2 SERVICE_STOP
      2 SERVICE_START
# cut -d ' ' -f 7 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  26974 type=SYSCALL
  26974 type=PROCTITLE
  22666 type=PATH
  13163 type=OBJ_PID
   6026 type=EXECVE
   3358 type=SECCOMP
     70 type=USER_END
     70 type=CRED_DISP
     69 type=USER_START
     63 type=USER_ACCT
     63 type=LOGIN
     63 type=CRED_ACQ
      7 type=SERVICE_STOP
      7 type=SERVICE_START
      6 type=USER_CMD
      6 type=CRED_REFR
      3 type=NETFILTER_CFG

@hillu
Copy link
Collaborator

hillu commented Sep 20, 2021

I think that the jq call does not do what you want. Using it on the sample piece of laurel output from https://github.com/threathunters-io/laurel/blob/master/practical-auditd-problems.md gives us just the EXECVE key. Removing the nth gives me:

[
  "CWD",
  "EXECVE",
  "ID",
  "PATH",
  "PROCTITLE",
  "SYSCALL"
]

… and apparently the ordering is not retained.

CWD, PROCTITLE, SYSCALL should give you the same counts. PATH should be off by a factor 2-3 because an EXECVE event usually goes together with multiple PATH lines which are grouped into one entry in the JSON log. The EXECVE count may be off if there are long command lines that are split across multiple EXECVE lines. (Merging those was one of the main motivations for writing Laurel in the first place.)

@juju4
Copy link
Author

juju4 commented Oct 2, 2021

If comparing this way, got below which has still some big difference in count

# jq -r 'keys' /var/log/laurel/audit.log | egrep -v '(\]|\[)' | sort | uniq -c | sort -nr | head -20
    281   "ID",
    278   "SYSCALL"
    278   "PROCTITLE",
    278   "PARENT_INFO",
    199   "PATH",
    199   "CWD",
    152   "EXECVE",
     79   "OBJ_PID",
      2   "SECCOMP"
      1   "USER_END"
      1   "ID"
      1   "CRED_DISP",
# cut -d ' ' -f 8 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  60448 type=SYSCALL
  60445 type=PROCTITLE
  48570 type=PATH
  31322 type=OBJ_PID
  26632 type=CWD
  14536 type=EXECVE
   2803 type=SECCOMP
    164 type=USER_END
    164 type=CRED_DISP
    163 type=USER_START
    155 type=USER_ACCT
    155 type=LOGIN
    155 type=CRED_ACQ
     32 type=USER_LOGIN
     23 type=SERVICE_START
     20 type=SERVICE_STOP
     10 type=NETFILTER_CFG
      8 type=USER_CMD
      8 type=CRED_REFR
# jq -r 'keys' /var/log/laurel/audit.log-20211002 | egrep -v '(\]|\[)' | sort | uniq -c | sort -nr | head -20
  11694   "ID",
  11280   "SYSCALL"
  11277   "PROCTITLE",
  11266   "PARENT_INFO",
   6739   "CWD",
   6738   "PATH",
   4320   "OBJ_PID",
   2410   "EXECVE",
    278   "SECCOMP"
     72   "ID"
     36   "USER_END"
     36   "CRED_DISP",
     35   "USER_START"
     21   "USER_ACCT"
     21   "LOGIN"
     21   "CRED_ACQ",
     14   "USER_CMD"
     14   "CRED_REFR",
      4   "SERVICE_STOP"
      4   "SERVICE_START"
# cut -d ' ' -f 8 /var/log/audispd/audispd.log-20211002 | sort | uniq -c | sort -nr | head -20
 175266 type=SYSCALL
 175265 type=PROCTITLE
 130816 type=PATH
  97193 type=OBJ_PID
  70442 type=CWD
  48698 type=EXECVE
   7197 type=SECCOMP
    527 type=USER_START
    527 type=USER_END
    527 type=CRED_DISP
    513 type=USER_ACCT
    513 type=LOGIN
    513 type=CRED_ACQ
     66 type=SERVICE_START
     65 type=SERVICE_STOP
     30 type=NETFILTER_CFG
     14 type=USER_CMD
     14 type=CRED_REFR
      1 type=USER_LOGIN
      1 type=SOCKADDR

(not sure why the change in cut field number)

@hillu
Copy link
Collaborator

hillu commented Oct 2, 2021

alright. I'll be doing some remodelling to be able to implement event filtering fieatures, this will allow for some more diagnostics and statistics.

In the meantime, here's a stupid idea: Are you sure that you collected all the logs for laurel? In the default configuration, log file segments are rotated at 1 MB which translates to something between 1700 and 1800 messages on my boxes.

@juju4
Copy link
Author

juju4 commented Oct 2, 2021

I had increased it to 10MB because I wanted logrotation to be done by logrotate every day and it seems 0 is not the right way the way to disable this option from laurel.
looking at logdir, this seems enough for a quiet day but not enough when more activity like today when I'm playing on the system. increased to 1GB and will check again after few days.

yeah, some extra filtering options would be nice, auditd exclusions are not really the most granular/precise... Thanks

@juju4
Copy link
Author

juju4 commented Oct 9, 2021

I added report with both audispd and laurel output in my ansible role pipeline
One example here
https://github.com/juju4/ansible-auditd/runs/3847691965?check_suite_focus=true#step:12:1
number seems approaching from a type perspective, less when digging by keyword

also for file integrity monitoring, audispd have nametype with CREATED, DELETED... but laurel output is always NORMAL

@hillu
Copy link
Collaborator

hillu commented Oct 9, 2021

also for file integrity monitoring, audispd have nametype with CREATED, DELETED... but laurel output is always NORMAL

Not sure what you mean by that, can you provide an example?

@juju4
Copy link
Author

juju4 commented Oct 9, 2021

# audispd report
*** sensitive_file CREATE+DELETE
Total unique files: 9
Total unique apt .#lk0x files: 0
      2 item=1 mode=0100600 rdev=00:00
      1 name="/etc/systemd/system/snap-core-11743.mount.Xg4Jpb9hHSCL~" ouid=0 nametype=DELETE
      1 name="/etc/systemd/system/snap-core-11743.mount.Xg4Jpb9hHSCL~" ouid=0 nametype=CREATE
      1 name="/etc/systemd/system/snap-core-11743.mount" ouid=0 nametype=CREATE
      1 name="/etc/systemd/system/snap-core-11420.mount" ouid=0 nametype=DELETE
      1 name="/etc/systemd/system/multi-user.target.wants/snap-core-11743.mount" ouid=0 nametype=CREATE

# laurel report (not same day)
*** sensitive_file -CREATE+DELETE- (name, nametype, exe)
      3 /var/www NORMAL /usr/bin/aide
      3 /var/www/html/.well-known NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/fim-test-touch-auditd1 NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/fim-test-touch3 NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file5-long NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file3-long NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file2 NORMAL /usr/bin/aide

@hillu
Copy link
Collaborator

hillu commented Oct 10, 2021

Laurel does not do anything specific with the PATH entries that may contain different nametype entries. So if you paste the following into its STDIN…

type=SYSCALL msg=audit(1633859647.072:120613): arch=c000003e syscall=316 success=yes exit=0 a0=ffffff9c a1=7fff70af382e a2=ffffff9c a3=7fff70af3851 items=4 ppid=3219422 pid=3219423 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts9 ses=3 comm="mv" exe="/bin/mv" subj==unconfined key="sudo"
type=CWD msg=audit(1633859647.072:120613): cwd="/home/bengen/src/spyre"
type=PATH msg=audit(1633859647.072:120613): item=0 name="/etc/sudoers.d/" inode=38798702 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=1 name="/etc/sudoers.d/" inode=38798702 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=2 name="/etc/sudoers.d/x2goserver.disabled" inode=38819726 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=3 name="/etc/sudoers.d/x2goserver" inode=38819726 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PROCTITLE msg=audit(1633859647.072:120613): proctitle=6D76002F6574632F7375646F6572732E642F7832676F7365727665722E64697361626C6564002F6574632F7375646F6572732E642F7832676F736572766572
type=EOE msg=audit(1633859647.072:120613): 

it should give you 4 PATH entries, like so:

{
  "ID": "1633859647.072:120613",
  "SYSCALL": {
    "arch": "0xc000003e",
    "syscall": 316,
    "success": "yes",
    "exit": 0,
    "items": 4,
    "ppid": 3219422,
    "pid": 3219423,
    "auid": 1000,
    "uid": 0,
    "gid": 0,
    "euid": 0,
    "suid": 0,
    "fsuid": 0,
    "egid": 0,
    "sgid": 0,
    "fsgid": 0,
    "tty": "pts9",
    "ses": 3,
    "comm": "mv",
    "exe": "/bin/mv",
    "subj": "=unconfined",
    "key": "sudo",
    "ARCH": "x86_64",
    "SYSCALL": "renameat2",
    "AUID": "bengen",
    "UID": "root",
    "GID": "root",
    "EUID": "root",
    "SUID": "root",
    "FSUID": "root",
    "EGID": "root",
    "SGID": "root",
    "FSGID": "root",
    "ARGV": [
      "0xffffff9c",
      "0x7fff70af382e",
      "0xffffff9c",
      "0x7fff70af3851"
    ]
  },
  "CWD": {
    "cwd": "/home/bengen/src/spyre"
  },
  "PATH": [
    {
      "item": 0,
      "name": "/etc/sudoers.d/",
      "inode": 38798702,
      "dev": "fd:00",
      "mode": "0o40755",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "PARENT",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 1,
      "name": "/etc/sudoers.d/",
      "inode": 38798702,
      "dev": "fd:00",
      "mode": "0o40755",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "PARENT",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 2,
      "name": "/etc/sudoers.d/x2goserver.disabled",
      "inode": 38819726,
      "dev": "fd:00",
      "mode": "0o100440",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "DELETE",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 3,
      "name": "/etc/sudoers.d/x2goserver",
      "inode": 38819726,
      "dev": "fd:00",
      "mode": "0o100440",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "CREATE",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    }
  ],
  "PROCTITLE": {
    "ARGV": [
      "mv",
      "/etc/sudoers.d/x2goserver.disabled",
      "/etc/sudoers.d/x2goserver"
    ]
  }
}

@hillu
Copy link
Collaborator

hillu commented Nov 3, 2021

@juju4 Do you still see problems with missing messages?

@hillu
Copy link
Collaborator

hillu commented Dec 10, 2021

@juju4 ping?

@juju4
Copy link
Author

juju4 commented Dec 18, 2021

Sorry for delay, busy times.

On github action side, I lost laurel report between build of Nov 29th and Dec 4th. just repaired it

On my test server, for some reasons, laurel logs are empty since some time while /var/log/audit/audit.log is fine.

# ps aux |grep laurel
_laurel   1072  0.0  0.1  10248  9488 ?        S<   Nov03  28:47 /usr/local/sbin/laurel --config /etc/laurel/config.toml
# killall laurel
# ps aux |grep laurel
_laurel   7661  0.6  0.0   1272   976 ?        S<   09:32   0:00 /usr/local/sbin/laurel --config /etc/laurel/config.toml
# ls -l /var/log/laurel/audit.log
-rw-------+ 1 _laurel _laurel 64550 Dec 18 09:33 /var/log/laurel/audit.log

trying to review during holidays

I see many commits since last 1.0.5 release on Nov 1st. any plan for a new one?

Thanks

@hillu
Copy link
Collaborator

hillu commented Dec 18, 2021

I see many commits since last 1.0.5 release on Nov 1st. any plan for a new one?

Yeah, I somehow got sidetracked dealing with surprise features in Java software. ;-)

@juju4
Copy link
Author

juju4 commented Dec 18, 2021

no problem. same for many people :)

@hillu
Copy link
Collaborator

hillu commented Nov 7, 2023

I am closing this issue because we have not made an progress in almost two years. Feel free to reopen if you still see problems with a more current version.

@hillu hillu closed this as completed Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants