Starting with SNMP traps, Zabbix Proxy, CentOS 7, Event Correlation, varbinds, var bind, varbind

Open SNMP trap listener port 162 in firewall:

firewall-cmd --add-port=162/udp --permanent
firewall-cmd --reload

Check SELinux status:

getenforce

Turn off SELinux in session level:

setenforce 0

Turn off selinux globally:

sed -i "s/^SELINUX=.*$/SELINUX=disabled/" /etc/selinux/config

Install SNMP trap listener:

yum install -y net-snmp-utils net-snmp-perl net-snmp nmap

Enable and strart the service:

systemctl enable snmptrapd
systemctl start snmptrapd
systemctl status snmptrapd

Install 'netstat' command:

yum -y install net-tools

Check the listening ports:

netstat -tulpn | grep 162

Install "zabbix_trap_receiver.pl"

cd /usr/bin

Install jq utility on debian:

apt -y install jq

For RHEL family and Ubuntu:

curl -sL https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -o /usr/bin/jq && chmod +x /usr/bin/jq

Set the file executable:

chmod +x zabbix_trap_receiver.pl

Check if the temp directory is correct:

grep "zabbix_traps.tmp" /usr/bin/zabbix_trap_receiver.pl

Whitelist all allowed community names:

cat <<'EOF'> /etc/snmp/snmptrapd.conf
authCommunity execute Earth
authCommunity execute Moon
authCommunity execute Mars
perl do "/usr/bin/zabbix_trap_receiver.pl";
EOF

Test if server is accessible:

nmap -p 10051 ec2-35-166-97-138.us-west-2.compute.amazonaws.com

If proxy do not exist:

rpm -ivh http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-1.el7.centos.noarch.rpm
yum -y install zabbix-proxy-sqlite3 vim
sed -i "s|^Server=127.0.0.1|Server=ec2-35-166-97-138.us-west-2.compute.amazonaws.com|" /etc/zabbix/zabbix_proxy.conf
sed -i "s|^Hostname=Zabbix proxy|Hostname=CentOS7ProxY|" /etc/zabbix/zabbix_proxy.conf
sed -i "s|^DBName=zabbix_proxy|DBName=\/tmp\/zabbix.proxy.db|" /etc/zabbix/zabbix_proxy.conf
grep -v "^$\|#" /etc/zabbix/zabbix_proxy.conf

Restart SNMP trap daemon:

systemctl restart snmptrapd
systemctl status snmptrapd

Enable Zabbix Trapper in "zabbix_proxy.conf":

conf=$(find / -name "zabbix_proxy.conf" | grep "/etc/")
echo $conf
cp $conf $conf.`date +%Y%m%d%H%M`
ls -l $conf*
sed -i "s|^.*StartSNMPTrapper=.*$|StartSNMPTrapper=1|" $conf
sed -i "s|^SNMPTrapperFile=.*$|SNMPTrapperFile=\/tmp\/zabbix_traps.tmp|" $conf
grep -v "^$\|^#" $conf

Restart Zabbix Server:

systemctl restart zabbix-proxy

Check if there is any listener:

ps -aux | grep [s]nmp

This will output something like:

zabbix   20169  0.0  0.4  97576  4196 ?        S    15:15   0:00 /usr/local/sbin/zabbix_proxy: snmp trapper [processed data in 0.000034 sec, idle 1 sec]

Clear logs and restart services:

> /var/log/zabbix/zabbix_proxy.log
> /var/log/audit/audit.log
systemctl restart snmptrapd
chown zabbix. /tmp/zabbix_traps.tmp
systemctl restart zabbix-proxy

Check if services are up and running:

systemctl status {snmptrapd,zabbix-proxy}

Ckeck if proxy trapper process is started:

grep "snmp trapper" /var/log/zabbix/zabbix_proxy.log

Send a test trap:

ls -l /tmp # note down the /tmp/zabbix_traps.tmp do not exist

# send test trap
snmptrap -v 1 -c Earth 127.0.0.1 '.1.3.6.1.6.3.1.1.5.3' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.3 s "eth0"
snmptrap -v 1 -c Earth 127.0.0.1 'IF-MIB::linkDown' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.3 s "eth0"
snmptrap -v 1 -c Earth 127.0.0.1 'IF-MIB::linkDown' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.3 s "enp2s0"

# check the content
cat /tmp/zabbix_traps.tmp

# empty file it there is a lot
> /tmp/zabbix_traps.tmp

# send recovery trap
snmptrap -v 1 -c Earth 127.0.0.1 '.1.3.6.1.6.3.1.1.5.4' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.4 s "eth0"
snmptrap -v 1 -c Earth 127.0.0.1 'IF-MIB::linkUp' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.4 s "eth0"
snmptrap -v 1 -c Earth 127.0.0.1 'IF-MIB::linkUp' '0.0.0.0' 6 33 '55' .1.3.6.1.6.3.1.1.5.4 s "enp2s0"

# now the file must exist
ls -l /tmp/zabbix_traps.tmp

Also the zabbix server log should contain this trap:

cat /var/log/zabbix/zabbix_proxy.log

Appendix: Check SELinux errors:

grep denied /var/log/audit/audit.log

The trigger expressions look like this. Pay extra attention for:

PROBLEM event generation mode: Multiple
OK event closes: if tag values match
Tag for matching: Interface
tag: Interface, value: {{ITEM.VALUE}.regsub("IF-MIB::link.*(eth[0-9])",\1)}