Kategorie: Kernel-Error-Blog (Seite 26 von 45)

Mailgraph von David Schweikert ist ein simples Perl-Script, das Postfix-Logdateien parst und daraus RRDtool-Graphen erzeugt. Empfangene, gesendete, abgelehnte Mails auf einen Blick. Für kleinere Mailserver oder Testsysteme reicht das völlig.

Was mailgraph nicht kann: SPF-, DMARC– und DKIM-Ergebnisse darstellen. Wie viele Mails bestehen den SPF-Check? Wie oft schlägt DMARC fehl? Gibt es einen Trend? Ich habe zwar einen existierenden SPF-Patch für mailgraph gefunden, wollte aber alle drei Protokolle in einem Aufwasch.

Die folgenden Patches erweitern mailgraph 1.14 um drei zusätzliche Graphen: SPF (pass/none/fail), DMARC (pass/none/fail) und DKIM (pass/none/fail). Ausgelegt für Postfix mit postfix-policyd-spf, opendkim und opendmarc.

Installation

Auf einem Debian-System müssen zwei Dateien ersetzt werden: /usr/sbin/mailgraph (der Daemon) und /usr/lib/cgi-bin/mailgraph.cgi (die Weboberfläche). Vorher sichern.

Die Patches als Download: mailgraph-dmarc-spf-dkim.patch.tar.gz

Patch 1: /usr/sbin/mailgraph

Der Daemon-Patch fügt neun RRD-Datasources hinzu (spfnone, spffail, spfpass, dmarcnone, dmarcfail, dmarcpass, dkimnone, dkimfail, dkimpass) und parst die Logzeilen von policy-spf, opendmarc und opendkim:

*** mailgraph	2012-06-17 00:00:00.000000000 +0200
--- mailgraph	2014-04-24 08:59:58.964977886 +0200
***************
*** 4,9 ****
--- 4,10 ----
  # copyright (c) 2000-2007 ETH Zurich
  # copyright (c) 2000-2007 David Schweikert <david@schweikert.ch>
  # released under the GNU General Public License
+ # with dkim-, dmarc, spf-patch Sebastian van de Meer <kernel-error@kernel-error.de>

  ######## Parse::Syslog 1.09 (automatically embedded) ########
  package Parse::Syslog;
***************
*** 382,388 ****
  my $rrd_greylist = "mailgraph_greylist.rrd";
  my $year;
  my $this_minute;
! my %sum = ( sent => 0, received => 0, bounced => 0, rejected => 0, virus => 0, spam => 0, greylisted => 0, delayed => 0);
  my $rrd_inited=0;

  my %opt = ();
--- 383,389 ----
  my $rrd_greylist = "mailgraph_greylist.rrd";
  my $year;
  my $this_minute;
! my %sum = ( sent => 0, received => 0, bounced => 0, rejected => 0, spfnone => 0, spffail => 0, spfpass => 0, dmarcnone => 0, dmarcfail => 0, dmarcpass => 0, dkimnone => 0, dkimfail => 0, dkimpass => 0, virus => 0, spam => 0, greylisted => 0, delayed => 0);
  my $rrd_inited=0;

  my %opt = ();
***************
*** 396,401 ****
--- 397,411 ----
  sub event_rejected($);
  sub event_virus($);
  sub event_spam($);
+ sub event_spfnone($);
+ sub event_spffail($);
+ sub event_spfpass($);
+ sub event_dmarcnone($);
+ sub event_dmarcfail($);
+ sub event_dmarcpass($);
+ sub event_dkimnone($);
+ sub event_dkimfail($);
+ sub event_dkimpass($);
  sub event_greylisted($);
  sub event_delayed($);
  sub init_rrd($);
***************
*** 533,538 ****
--- 543,557 ----
  				'DS:recv:ABSOLUTE:'.($rrdstep*2).':0:U',
  				'DS:bounced:ABSOLUTE:'.($rrdstep*2).':0:U',
  				'DS:rejected:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:spfnone:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:spffail:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:spfpass:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dmarcnone:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dmarcfail:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dmarcpass:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dkimnone:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dkimfail:ABSOLUTE:'.($rrdstep*2).':0:U',
+ 				'DS:dkimpass:ABSOLUTE:'.($rrdstep*2).':0:U',
  				"RRA:AVERAGE:0.5:$day_steps:$realrows",   # day
  				"RRA:AVERAGE:0.5:$week_steps:$realrows",  # week
  				"RRA:AVERAGE:0.5:$month_steps:$realrows", # month
***************
*** 614,619 ****
--- 633,649 ----
  				event($time, 'bounced');
  			}
  		}
+ 	  elsif ($prog eq 'policy-spf') {
+ 	    if ($text =~ /Received-SPF: none/) {
+ 	      event($time, 'spfnone');
+ 	    }
+ 	    elsif($text =~ /Received-SPF: pass/) {
+ 				event($time, 'spfpass');
+ 			}
+ 	    elsif($text =~ /Received-SPF:/) {
+ 				event($time, 'spffail');
+ 			}
+ 	  }
  		elsif($prog eq 'local') {
  			if($text =~ /\bstatus=bounced\b/) {
  				event($time, 'bounced');
***************
*** 862,867 ****
--- 892,919 ----
  			event($time, 'virus');
  		}
  	}
+ 	elsif ($prog eq 'opendmarc') {
+ 		if ($text =~ /pass/) {
+ 			event($time, 'dmarcpass');
+ 		}
+ 		elsif($text =~ /none/) {
+ 			event($time, 'dmarcnone');
+ 		}
+ 		elsif($text =~ /fail/) {
+ 			event($time, 'dmarcfail');
+ 		}
+ 	}
+ 	elsif ($prog eq 'opendkim') {
+ 		if ($text =~ /DKIM verification successful/) {
+ 			event($time, 'dkimpass');
+ 		}
+ 		elsif($text =~ /no signature data/) {
+ 			event($time, 'dkimnone');
+ 		}
+ 		elsif($text =~ /bad signature data/) {
+ 			event($time, 'dkimfail');
+ 		}
+ 	}
  	elsif($prog eq 'avmilter') {
  		# AntiVir Milter
  		if($text =~ /^Alert!/) {
***************
*** 918,931 ****
  	return 1 if $m == $this_minute;
  	return 0 if $m < $this_minute;

! 	print "update $this_minute:$sum{sent}:$sum{received}:$sum{bounced}:$sum{rejected}:$sum{virus}:$sum{spam}:$sum{greylisted}:$sum{delayed}\n" if $opt{verbose};
! 	RRDs::update $rrd, "$this_minute:$sum{sent}:$sum{received}:$sum{bounced}:$sum{rejected}" unless $opt{'no-mail-rrd'};
  	RRDs::update $rrd_virus, "$this_minute:$sum{virus}:$sum{spam}" unless $opt{'no-virus-rrd'};
  	RRDs::update $rrd_greylist, "$this_minute:$sum{greylisted}:$sum{delayed}" unless $opt{'no-greylist-rrd'};
  	if($m > $this_minute+$rrdstep) {
  		for(my $sm=$this_minute+$rrdstep;$sm<$m;$sm+=$rrdstep) {
! 			print "update $sm:0:0:0:0:0:0:0:0 (SKIP)\n" if $opt{verbose};
! 			RRDs::update $rrd, "$sm:0:0:0:0" unless $opt{'no-mail-rrd'};
  			RRDs::update $rrd_virus, "$sm:0:0" unless $opt{'no-virus-rrd'};
  			RRDs::update $rrd_greylist, "$sm:0:0" unless $opt{'no-greylist-rrd'};
  		}
--- 970,983 ----
  	return 1 if $m == $this_minute;
  	return 0 if $m < $this_minute;

! 	print "update $this_minute:$sum{sent}:$sum{received}:$sum{bounced}:$sum{rejected}:$sum{spfnone}:$sum{spffail}:$sum{spfpass}:$sum{dmarcnone}:$sum{dmarcfail}:$sum{dmarcpass}:$sum{dkimnone}:$sum{dkimfail}:$sum{dkimpass}:$sum{virus}:$sum{spam}:$sum{greylisted}:$sum{delayed}\n" if $opt{verbose};
! 	RRDs::update $rrd, "$this_minute:$sum{sent}:$sum{received}:$sum{bounced}:$sum{rejected}:$sum{spfnone}:$sum{spffail}:$sum{spfpass}:$sum{dmarcnone}:$sum{dmarcfail}:$sum{dmarcpass}:$sum{dkimnone}:$sum{dkimfail}:$sum{dkimpass}" unless $opt{'no-mail-rrd'};
  	RRDs::update $rrd_virus, "$this_minute:$sum{virus}:$sum{spam}" unless $opt{'no-virus-rrd'};
  	RRDs::update $rrd_greylist, "$this_minute:$sum{greylisted}:$sum{delayed}" unless $opt{'no-greylist-rrd'};
  	if($m > $this_minute+$rrdstep) {
  		for(my $sm=$this_minute+$rrdstep;$sm<$m;$sm+=$rrdstep) {
! 			print "update $sm:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0 (SKIP)\n" if $opt{verbose};
! 			RRDs::update $rrd, "$sm:0:0:0:0:0:0:0:0:0:0:0:0:0" unless $opt{'no-mail-rrd'};
  			RRDs::update $rrd_virus, "$sm:0:0" unless $opt{'no-virus-rrd'};
  			RRDs::update $rrd_greylist, "$sm:0:0" unless $opt{'no-greylist-rrd'};
  		}
***************
*** 935,940 ****
--- 987,1001 ----
  	$sum{received}=0;
  	$sum{bounced}=0;
  	$sum{rejected}=0;
+ 	$sum{spfnone}=0;
+ 	$sum{spffail}=0;
+ 	$sum{spfpass}=0;
+ 	$sum{dmarcnone}=0;
+ 	$sum{dmarcfail}=0;
+ 	$sum{dmarcpass}=0;
+ 	$sum{dkimnone}=0;
+ 	$sum{dkimfail}=0;
+ 	$sum{dkimpass}=0;
  	$sum{virus}=0;
  	$sum{spam}=0;
  	$sum{greylisted}=0;

Patch 2: /usr/lib/cgi-bin/mailgraph.cgi

Der CGI-Patch erzeugt die drei neuen Graphen (SPF, DMARC, DKIM) und bindet sie in die HTML-Ausgabe ein. Jeder Graph zeigt pass als Fläche, none als Stack und fail als Linie:

*** mailgraph.cgi	2012-06-17 00:00:00.000000000 +0200
--- mailgraph.cgi	2014-04-24 09:03:11.917988368 +0200
***************
*** 4,9 ****
--- 4,10 ----
  # copyright (c) 2000-2007 ETH Zurich
  # copyright (c) 2000-2007 David Schweikert <david@schweikert.ch>
  # released under the GNU General Public License
+ # with dkim-, dmarc, spf-patch Sebastian van de Meer <kernel-error@kernel-error.de>

  use RRDs;
  use POSIX qw(uname);
***************
*** 15,21 ****
--- 16,25 ----
  my $xpoints = 540;
  my $points_per_sample = 3;
  my $ypoints = 160;
+ my $ypoints_spf = 96;
  my $ypoints_err = 96;
+ my $ypoints_dmarc = 96;
+ my $ypoints_dkim = 96;
  my $ypoints_grey = 96;
  my $rrd = '/var/lib/mailgraph/mailgraph.rrd';
  my $rrd_virus = '/var/lib/mailgraph/mailgraph_virus.rrd';
***************
*** 32,37 ****
--- 36,50 ----
  	sent       => '000099',
  	received   => '009900',
  	rejected   => 'AA0000',
+ 	spfnone    => '000AAA',
+ 	spffail    => '12FF0A',
+ 	spfpass    => 'D15400',
+ 	dmarcnone  => 'FFFF00',
+ 	dmarcfail  => 'FF00EA',
+ 	dmarcpass  => '00FFD5',
+ 	dkimnone   => '3013EC',
+ 	dkimfail   => '006B3A',
+ 	dkimpass   => '491503',
  	bounced    => '000000',
  	virus      => 'DDBB00',
  	spam       => '999999',
***************
*** 154,159 ****
--- 167,292 ----
  	);
  }

+ sub graph_spf($)
+ {
+ 	my ($range, $file) = @_;
+ 	my $step = $range*$points_per_sample/$xpoints;
+ 	rrd_graph($range, $file, $ypoints_spf,
+ 		"DEF:spfpass=$rrd:spfpass:AVERAGE",
+ 		"DEF:mspfpass=$rrd:spfpass:MAX",
+ 		"CDEF:rspfpass=spfpass,60,*",
+ 		"CDEF:dspfpass=spfpass,UN,0,spfpass,IF,$step,*",
+ 		"CDEF:sspfpass=PREV,UN,dspfpass,PREV,IF,dspfpass,+",
+ 		"CDEF:rmspfpass=mspfpass,60,*",
+ 		"AREA:rspfpass#$color{spfpass}:SPF pass",
+ 		'GPRINT:sspfpass:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rspfpass:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmspfpass:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:spfnone=$rrd:spfnone:AVERAGE",
+ 		"DEF:mspfnone=$rrd:spfnone:MAX",
+ 		"CDEF:rspfnone=spfnone,60,*",
+ 		"CDEF:dspfnone=spfnone,UN,0,spfnone,IF,$step,*",
+ 		"CDEF:sspfnone=PREV,UN,dspfnone,PREV,IF,dspfnone,+",
+ 		"CDEF:rmspfnone=mspfnone,60,*",
+ 		"STACK:rspfnone#$color{spfnone}:SPF none",
+ 		'GPRINT:sspfnone:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rspfnone:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmsrspfnone:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:spffail=$rrd:spffail:AVERAGE",
+ 		"DEF:mspffail=$rrd:spffail:MAX",
+ 		"CDEF:rspffail=spffail,60,*",
+ 		"CDEF:dspffail=spffail,UN,0,spffail,IF,$step,*",
+ 		"CDEF:sspffail=PREV,UN,dspffail,PREV,IF,dspffail,+",
+ 		"CDEF:rmspffail=mspffail,60,*",
+ 		"LINE2:rspffail#$color{spffail}:SPF fail",
+ 		'GPRINT:sspffail:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rspffail:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmspffail:MAX:max\: %4.0lf msgs/min\l',
+ 	);
+ }
+
+ sub graph_dmarc($)
+ {
+ 	my ($range, $file) = @_;
+ 	my $step = $range*$points_per_sample/$xpoints;
+ 	rrd_graph($range, $file, $ypoints_dmarc,
+ 		"DEF:dmarcpass=$rrd:dmarcpass:AVERAGE",
+ 		"DEF:mdmarcpass=$rrd:dmarcpass:MAX",
+ 		"CDEF:rdmarcpass=dmarcpass,60,*",
+ 		"CDEF:ddmarcpass=dmarcpass,UN,0,dmarcpass,IF,$step,*",
+ 		"CDEF:sdmarcpass=PREV,UN,ddmarcpass,PREV,IF,ddmarcpass,+",
+ 		"CDEF:rmdmarcpass=mdmarcpass,60,*",
+ 		"AREA:rdmarcpass#$color{dmarcpass}:DMARC pass",
+ 		'GPRINT:sdmarcpass:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdmarcpass:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdmarcpass:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:dmarcnone=$rrd:dmarcnone:AVERAGE",
+ 		"DEF:mdmarcnone=$rrd:dmarcnone:MAX",
+ 		"CDEF:rdmarcnone=dmarcnone,60,*",
+ 		"CDEF:ddmarcnone=dmarcnone,UN,0,dmarcnone,IF,$step,*",
+ 		"CDEF:sdmarcnone=PREV,UN,ddmarcnone,PREV,IF,ddmarcnone,+",
+ 		"CDEF:rmdmarcnone=mdmarcnone,60,*",
+ 		"STACK:rdmarcnone#$color{dmarcnone}:DMARC none",
+ 		'GPRINT:sdmarcnone:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdmarcnone:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdmarcnone:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:dmarcfail=$rrd:dmarcfail:AVERAGE",
+ 		"DEF:mdmarcfail=$rrd:dmarcfail:MAX",
+ 		"CDEF:rdmarcfail=dmarcfail,60,*",
+ 		"CDEF:ddmarcfail=dmarcfail,UN,0,dmarcfail,IF,$step,*",
+ 		"CDEF:sdmarcfail=PREV,UN,ddmarcfail,PREV,IF,ddmarcfail,+",
+ 		"CDEF:rmdmarcfail=mdmarcfail,60,*",
+ 		"LINE2:rdmarcfail#$color{dmarcfail}:DMARC fail",
+ 		'GPRINT:sdmarcfail:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdmarcfail:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdmarcfail:MAX:max\: %4.0lf msgs/min\l',
+ 	);
+ }
+
+ sub graph_dkim($)
+ {
+ 	my ($range, $file) = @_;
+ 	my $step = $range*$points_per_sample/$xpoints;
+ 	rrd_graph($range, $file, $ypoints_dkim,
+ 		"DEF:dkimpass=$rrd:dkimpass:AVERAGE",
+ 		"DEF:mdkimpass=$rrd:dkimpass:MAX",
+ 		"CDEF:rdkimpass=dkimpass,60,*",
+ 		"CDEF:ddkimpass=dkimpass,UN,0,dkimpass,IF,$step,*",
+ 		"CDEF:sdkimpass=PREV,UN,ddkimpass,PREV,IF,ddkimpass,+",
+ 		"CDEF:rmdkimpass=mdkimpass,60,*",
+ 		"AREA:rdkimpass#$color{dkimpass}:DKIM pass",
+ 		'GPRINT:sdkimpass:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdkimpass:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdkimpass:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:dkimnone=$rrd:dkimnone:AVERAGE",
+ 		"DEF:mdkimnone=$rrd:dkimnone:MAX",
+ 		"CDEF:rdkimnone=dkimnone,60,*",
+ 		"CDEF:ddkimnone=dkimnone,UN,0,dkimnone,IF,$step,*",
+ 		"CDEF:sdkimnone=PREV,UN,ddkimnone,PREV,IF,ddkimnone,+",
+ 		"CDEF:rmdkimnone=mdkimnone,60,*",
+ 		"STACK:rdkimnone#$color{dkimnone}:DKIM none",
+ 		'GPRINT:sdkimnone:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdkimnone:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdkimnone:MAX:max\: %4.0lf msgs/min\l',
+ 		"DEF:dkimfail=$rrd:dkimfail:AVERAGE",
+ 		"DEF:mdkimfail=$rrd:dkimfail:MAX",
+ 		"CDEF:rdkimfail=dkimfail,60,*",
+ 		"CDEF:ddkimfail=dkimfail,UN,0,dkimfail,IF,$step,*",
+ 		"CDEF:sdkimfail=PREV,UN,ddkimfail,PREV,IF,ddkimfail,+",
+ 		"CDEF:rmdkimfail=mdkimfail,60,*",
+ 		"LINE2:rdkimfail#$color{dkimfail}:DKIM fail",
+ 		'GPRINT:sdkimfail:MAX:total\: %8.0lf msgs',
+ 		'GPRINT:rdkimfail:AVERAGE:avg\: %5.2lf msgs/min',
+ 		'GPRINT:rmdkimfail:MAX:max\: %4.0lf msgs/min\l',
+ 	);
+ }
+
  [...]

+ 		print "<img src=\"$scriptname?${n}-s\" alt=\"mailgraph\"/></p>\n";
+ 		print "<img src=\"$scriptname?${n}-d\" alt=\"mailgraph\"/></p>\n";
+ 		print "<img src=\"$scriptname?${n}-k\" alt=\"mailgraph\"/></p>\n";

  [...]

+ 		elsif($img =~ /^(\d+)-s$/) {
+ 			my $file = "$tmp_dir/$uri/mailgraph_$1_spf.png";
+ 			graph_spf($graphs[$1]{seconds}, $file);
+ 			send_image($file);
+ 		}
+ 		elsif($img =~ /^(\d+)-d$/) {
+ 			my $file = "$tmp_dir/$uri/mailgraph_$1_dmarc.png";
+ 			graph_dmarc($graphs[$1]{seconds}, $file);
+ 			send_image($file);
+ 		}
+ 		elsif($img =~ /^(\d+)-k$/) {
+ 			my $file = "$tmp_dir/$uri/mailgraph_$1_dkim.png";
+ 			graph_dkim($graphs[$1]{seconds}, $file);
+ 			send_image($file);
+ 		}

Was die Patches tun

Der Daemon-Patch erweitert die RRD-Datenbank um neun Datasources und erkennt im Syslog die Ausgaben von drei Diensten:

Der CGI-Patch erzeugt drei neue Graphen unterhalb der bestehenden. Jeder zeigt pass als Fläche, none als Stack darüber und fail als rote Linie. Die Zeiträume (Tag, Woche, Monat, Jahr) werden wie bei den Standard-Graphen automatisch erzeugt.

Hinweise

Die Patches sind für mailgraph 1.14 geschrieben. Weil mailgraph die RRD-Datasources beim ersten Start festlegt, muss nach dem Patchen die bestehende mailgraph.rrd gelöscht werden. Die alten Daten gehen dabei verloren, die neuen Graphen fangen bei Null an.

Es gibt auch einen separaten Patch für DANE-Graphen in mailgraph.

Wer heute mit dem Monitoring anfängt: rspamd bringt eine eigene Weboberfläche mit und prüft SPF, DKIM und DMARC in einem Durchgang. Zusammen mit Prometheus und Grafana bekommt man deutlich flexiblere Dashboards als mit RRDtool. Mailgraph bleibt trotzdem nützlich wenn man schnell was Schlankes braucht, das ohne externe Abhängigkeiten läuft.

Fragen oder Verbesserungsvorschläge? Einfach melden.