Skab overblik over dine DMARC rapporter

(Last Updated On: 20. januar 2021)

Denne guide viser hvordan du kan opsamle dine DMARC rapporter og gemme data i en database som du herefter kan præsentere på en hjemmeside. Det hele er udført på Ubuntu 20.04 LTS med Apache 2.4, PHP 7.4, MariaDB 10 og Perl 5.3. Du benytter muligvis et andet Unix/Linux OS, og skal så blot foretage de tilrettelser der måtte være nødvendige.

Lad være med blot at kopiere kommandoer fra denne vejledning… Husk at tænke selv 😉

Jeg benytter et simpelt værktøj udviklet af TechSneeze.com, som de kalder dmarkts-report-parser. Det består af en parser der udtrækker data og lægger dem i databasen og et web GUI der fremviser data.

Parseren kan udtrække data på flere måder, bl.a. via en imap forbindelse. Men jeg har ikke konfigureret imap på denne server, da MTA’en (Postfix) primært benyttes til at sende post.

Mit setup fungerer derfor som følger:

Postfix modtager DMARC rapporterne.
Et cronjob dykker ned i disse mails, finder og udtrækker de vedhæftede filer og pakker dem ud til en dertil indrettet folder.
Herefter køres parseren (et perl script) som trækker data ud af de vedhæftede filer og lægger det i databasen
Nu kan data ses på hjemmesiden.

Indholdsfortegnelse

Konfiguration af Postfix

Mit Postfix setup er meget simpelt da den i dette tilfælde primært benyttes til afsendelse af mail. Der er kun 3-4 konti til modtagelse og jeg har derfor ikke et fancy setup med imap og database, men blot flade tekstfiler hvor disse konti er konfigureret.

Fra main.cf:

virtual_alias_maps = hash:/etc/postfix/virtual
virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox

1 2	virtual_alias_maps = hash:/etc/postfix/virtual virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox

I virtual filen skriver jeg ind hvortil dmarc rapporterne skal leveres:

vim /etc/postfix/virtual
dmarc@example.com dmarc@mailbox.example.com

1 2	vim /etc/postfix/virtual dmarc@example.com dmarc@mailbox.example.com

… og i virtual_mailbox filen fortæller jeg hvilken folder de skal placeres i. Den folder du angiver er relativ i forhold til den mappe du har angivet i virtual_alias_maps i main.cf.

vim /etc/postfix/virtual_mailbox
dmarc@mailbox.example.com mailbox.example.com/dmarc/

1 2	vim /etc/postfix/virtual_mailbox dmarc@mailbox.example.com mailbox.example.com/dmarc/

Herefter køres postmap på de to filer og Postfix genstartes:

postmap virtual
postmap virtual_mailbox
postfix reload

postmap virtual

postmap virtual_mailbox

postfix reload

dmarcts-report-parser

Nu henter du parseren fra TechSneeze … jeg benytter wget . Herefter pakker du den ud til den folder hvor du ønsker den skal ligge.

wget https://github.com/techsneeze/dmarcts-report-parser/archive/master.zip
unzip master.zip -d /sti/til/folder

1 2	wget https://github.com/techsneeze/dmarcts-report-parser/archive/master.zip unzip master.zip -d /sti/til/folder

På github forsiden kan du se hvilke afhængigheder der skal installeres på forskellige unix/linux operativsystemer.

Jeg har på Ubuntu installeret følgende:

apt-get install libfile-mimeinfo-perl libmail-imapclient-perl libmime-tools-perl libxml-simple-perl \
libclass-dbi-mysql-perl libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl \
libmail-mbox-messageparser-perl unzip

apt-get install libfile-mimeinfo-perl libmail-imapclient-perl libmime-tools-perl libxml-simple-perl \

libclass-dbi-mysql-perl libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl \

libmail-mbox-messageparser-perl unzip

Gå til den folder du pakkede zip filen ud til, og opret config filen:

cd sti/til/folder
cp dmarcts-report-parser.conf.sample dmarcts-report-parser.conf

1 2	cd sti/til/folder cp dmarcts-report-parser.conf.sample dmarcts-report-parser.conf

Da jeg ikke benytter imap delen kan jeg helt ignorere disse parametre, og kun forhold mig til dem der vedrører databasen:

$dbname = 'dmarc';
$dbuser = 'dmarc';
$dbpass = 'very_secret';
$dbhost = 'localhost'; # localhost, hostname or ip
$dbport = '3306';

$dbname = 'dmarc';

$dbuser = 'dmarc';

$dbpass = 'very_secret';

$dbhost = 'localhost'; # localhost, hostname or ip

$dbport = '3306';

Opret MySQL database og bruger

Du kan nøjes med at oprette databasen. Hvis dmarcts-report-parser ikke finder nogen tabeller, sørger den nemlig selv for at oprette dem:

CREATE DATABASE dmarc;
GRANT USAGE ON *.* TO `dmarc`@`10.0.0%` IDENTIFIED BY 'very_secret';
GRANT ALL PRIVILEGES ON `dmarc`.* TO `dmarc`@`10.0.0%`;

CREATE DATABASE dmarc;

GRANT USAGE ON *.* TO `dmarc`@`10.0.0%` IDENTIFIED BY 'very_secret';

GRANT ALL PRIVILEGES ON `dmarc`.* TO `dmarc`@`10.0.0%`;

Udskillelse af vedhæftede filer

Inden jeg kan bruge dmarcts-report-parser, skal jeg trække de vedhæftede rapporter ud af de e-mails der er modtaget. Til det formål benytter jeg mu som er en del af maildir-utils .

apt install maildir-utils

1	apt install maildir-utils

Nu oprettes et shell script der trækker filerne ud. Herunder har jeg angivet den essentielle del af dette script.

#!/bin/bash

DMARC_MAILACCOUNT="/var/spool/postfix/virtual/mailbox.example.com/dmarc/new"
DMARC_MAIL_NEW=""sti/til/dmarc/mail/new"
DMARC_MAIL_ERROR=""sti/til/dmarc/mail/error"
DMARC_MAIL_DONE=""sti/til/dmarc/mail/processed"
DMARC_ATTACHMENTS_NEW="sti/dmarc/attachments/new"
DMARC_ATTACHMENTS_DONE="sti/dmarc/attachments/processed"
MU=/usr/bin/mu
MV=/usr/bin/mv
LS=/usr/bin/ls

#Count regular files in mail account dir
mail_count=$($LS $DMARC_MAILACCOUNT/ | $WC -l)

if [[ $mail_count -eq "0" ]]; then
  echo "There are no mails to process (mail count = $mail_count)"
else
  #Move mails to dmarc maildir
  echo "Moving $mail_count mails to $DMARC_MAIL_NEW"
  $MV $DMARC_MAILACCOUNT/* $DMARC_MAIL_NEW/
fi
file_count=$($LS $DMARC_MAIL_NEW/ | $WC -l)

if [[ $file_count -eq "0" ]]; then
  if [[ $mail_count -eq "0" ]]; then
    echo "Nothing to do"
    exit 0
  fi
fi
#If number of files is not equal after moving the mails
#We may have leftovers from a previous run. 
#This should be investigated
if [[ $mail_count -ne "$file_count" ]]; then
  echo "mail count: $mail_count"
  echo "file count: $file_count"
  echo "Please investigate the diff between the above numbers"
  exit 0
fi

echo "There are $file_count dmarc reports"
exit 0

FILES="${DMARC_MAIL_NEW}/*"

for file in $FILES
do
  f=$(basename "$file")
  echo "Extracting attachments from $f"
  #List attachments
  output=$($MU extract $file)
  #Loop the list
  extract_error=0
  while IFS= read -r outline; do
    if [[ $outline =~ ^[[:space:]]*([0-9]+)(.*)\[attach\] ]]; then
      #Get the number of the file matched in the above line
      number="${BASH_REMATCH[1]}"
      extr_cmd="$MU extract $file --parts=$number --target-dir=$DMARC_ATTACHMENTS_NEW"
      #Extract the file
      $extr_cmd
      return_code=$?
      if [[ $return_code -eq "0" ]]; then
        echo "File successfully extracted."
      else
        extract_error=1
        echo "$extr_cmd exited with code $return_code"
      fi
    fi
  done <<< "$output"

  if [[ $extract_error -eq "1" ]]; then
    echo "Moving file to mail error folder"
    $MV $file $DMARC_MAIL_ERROR/
  else
    echo "Moving file to mail processed folder"
    $MV $file $DMARC_MAIL_DONE/
  fi
done

#Unpack gz files
gunzip $DMARC_ATTACHMENTS_NEW/*.xml.gz

#There may also be zip files to unpack 
unzip $DMARC_ATTACHMENTS_NEW/*.zip
#Remove the unpacked zip files
rm $DMARC_ATTACHMENTS_NEW/*.zip

#Add data to database
$DMARC_PARSER -x $DMARC_ATTACHMENTS_NEW/*

#Move attachments to processed/ folder
$MV $DMARC_ATTACHMENTS_NEW/* $DMARC_ATTACHMENTS_DONE/

#!/bin/bash

DMARC_MAILACCOUNT="/var/spool/postfix/virtual/mailbox.example.com/dmarc/new"

DMARC_MAIL_NEW=""sti/til/dmarc/mail/new"

DMARC_MAIL_ERROR=""sti/til/dmarc/mail/error"

DMARC_MAIL_DONE=""sti/til/dmarc/mail/processed"

DMARC_ATTACHMENTS_NEW="sti/dmarc/attachments/new"

DMARC_ATTACHMENTS_DONE="sti/dmarc/attachments/processed"

MU=/usr/bin/mu

MV=/usr/bin/mv

LS=/usr/bin/ls

#Count regular files in mail account dir

mail_count=$($LS $DMARC_MAILACCOUNT/ | $WC -l)

if [[ $mail_count -eq "0" ]]; then

echo "There are no mails to process (mail count = $mail_count)"

else

#Move mails to dmarc maildir

echo "Moving $mail_count mails to $DMARC_MAIL_NEW"

$MV $DMARC_MAILACCOUNT/* $DMARC_MAIL_NEW/

file_count=$($LS $DMARC_MAIL_NEW/ | $WC -l)

if [[ $file_count -eq "0" ]]; then

if [[ $mail_count -eq "0" ]]; then

echo "Nothing to do"

exit 0

#If number of files is not equal after moving the mails

#We may have leftovers from a previous run.

#This should be investigated

if [[ $mail_count -ne "$file_count" ]]; then

echo "mail count: $mail_count"

echo "file count: $file_count"

echo "Please investigate the diff between the above numbers"

exit 0

echo "There are $file_count dmarc reports"

exit 0

FILES="${DMARC_MAIL_NEW}/*"

for file in $FILES

f=$(basename "$file")

echo "Extracting attachments from $f"

#List attachments

output=$($MU extract $file)

#Loop the list

extract_error=0

while IFS= read -r outline; do

if [[ $outline =~ ^[[:space:]]*([0-9]+)(.*)\[attach\] ]]; then

#Get the number of the file matched in the above line

number="${BASH_REMATCH[1]}"

extr_cmd="$MU extract $file --parts=$number --target-dir=$DMARC_ATTACHMENTS_NEW"

#Extract the file

$extr_cmd

return_code=$?

if [[ $return_code -eq "0" ]]; then

echo "File successfully extracted."

else

extract_error=1

echo "$extr_cmd exited with code $return_code"

done <<< "$output"

if [[ $extract_error -eq "1" ]]; then

echo "Moving file to mail error folder"

$MV $file $DMARC_MAIL_ERROR/

else

echo "Moving file to mail processed folder"

$MV $file $DMARC_MAIL_DONE/

done

#Unpack gz files

gunzip $DMARC_ATTACHMENTS_NEW/*.xml.gz

#There may also be zip files to unpack

unzip $DMARC_ATTACHMENTS_NEW/*.zip

#Remove the unpacked zip files

rm $DMARC_ATTACHMENTS_NEW/*.zip

#Add data to database

$DMARC_PARSER -x $DMARC_ATTACHMENTS_NEW/*

#Move attachments to processed/ folder

$MV $DMARC_ATTACHMENTS_NEW/* $DMARC_ATTACHMENTS_DONE/

dmarcts-report-viewer og Apache konfiguration

Nu mangler du blot at hente den del der præsenterer data og konfigurere Apache:

wget https://github.com/techsneeze/dmarcts-report-viewer/archive/master.zip

1	wget https://github.com/techsneeze/dmarcts-report-viewer/archive/master.zip

Pak zip filen ud i dit webdir. På Ubuntu har jeg placeret filerne under /var/www/vhosts/dmarc .

Min Apache 2.4 konfiguration ser således ud:

<VirtualHost *:443>
  DocumentRoot /var/www/vhosts/dmarc
  ServerName dmarc.example.com
  DirectoryIndex dmarcts-report-viewer.php
  ErrorLog "|/usr/bin/rotatelogs -l /var/log/apache2/dmarc.example.com/error.%Y.%m.%d 86400"
  CustomLog "|/usr/bin/rotatelogs -l /var/log/apache2/dmarc.example.com/access.%Y.%m.%d 86400" combined
  php_value session.save_path /var/www/data/dmarc/sessions
  php_flag log_errors On
  php_flag display_errors Off
  php_value error_log /var/log/apache2/dmarc.example.com/php_error.log

  SSLEngine on
  SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf

  <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
  </FilesMatch>

  <Location "/">
    Options FollowSymLinks
    AllowOverride None
    AuthType Basic
    AuthName "DMARC report viewer"
    AuthUserFile /etc/apache2/passwd-ed-admin
    Require valid-user
  </Location>
</VirtualHost>

DocumentRoot /var/www/vhosts/dmarc

ServerName dmarc.example.com

DirectoryIndex dmarcts-report-viewer.php

ErrorLog "|/usr/bin/rotatelogs -l /var/log/apache2/dmarc.example.com/error.%Y.%m.%d 86400"

CustomLog "|/usr/bin/rotatelogs -l /var/log/apache2/dmarc.example.com/access.%Y.%m.%d 86400" combined

php_value session.save_path /var/www/data/dmarc/sessions

php_flag log_errors On

php_flag display_errors Off

php_value error_log /var/log/apache2/dmarc.example.com/php_error.log

SSLEngine on

SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem

SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem

Include /etc/letsencrypt/options-ssl-apache.conf

SSLOptions +StdEnvVars

</FilesMatch>

Options FollowSymLinks

AllowOverride None

AuthType Basic

AuthName "DMARC report viewer"

AuthUserFile /etc/apache2/passwd-ed-admin

Require valid-user

</Location>

</VirtualHost>

En særlig tak til den person der skrev dette blog-indlæg. Uden det var jeg ikke faldet over dmarcts-report-parser:

Self Hosted DMARC reports