For the longest time, I had unknowingly been reinventing the wheel. Cobbling together scripts and timers to notify of administrative concerns to the currently active desktop session. I wanted graphical dialogues, because that is what I’d grown up with. But how terribly uninformative it is for a dialog to appear that essentially says “Something is wrong, you should probably look into it”.
/var/log trying to tell you for the 1,435th time that you're running out of disk space
So many daemonized programs define arguments for sending mail when an issue arises and, foolish me, assuming that those options always meant email, never investigated any further. While it is true that with a mail transfer agent such notifications can be sent over the network to an email address which makes sense in a server environment, I always thought it a bit silly for an end user device. But what I didn’t realize was that user mail agents with local mailboxes were ever an option.
The common user mail agents, from my poking around Debian dependencies, are either bsd-mailx or GNU mailutils. And where bsd-mailx is minimalistic, and in fact only supports local mailboxes, it also lacks features such as scaling to different terminal sizes or complete previous/next navigation. I would recommend mailutils, which can be set as the preferred user mail agent through update-alternatives:
update-alternatives --config mailx
And select /usr/bin/mail.mailutils.
Mailutils can be invoked simply with:
mail
Some mail commands I’ve found useful:
z : next window (scroll down)
z- : previous window (scroll up)
$ : view newest message
^ : view oldest message
<number> : view message
d<number> : delete message
h : reprint message list window
n/next : next message
p/prev : previous message
Without having to lift a finger, stdout of actions invoked by both cron and anacron are sent to the user mailbox at /var/mail/$USER. It produces a chronological, easy to view means of checking system jobs. A lazy administrator could even just view the mailbox with pager /var/mail/$USER but will miss out on the cozy formatting afforded by a proper mail agent.
Return-path: <root@Computer>
Envelope-to: root@computer
Delivery-date: Thu, 05 Dec 2024 01:52:44 -0500
Received: from root by Computer with local (Exim 4.96)
(envelope-from <root@computer>)
id 1tJ5jA-000YGN-1M
for root@computer;
Thu, 05 Dec 2024 01:52:44 -0500
From: Anacron <root@computer>
To: root@computer
Subject: Anacron job 'cron.weekly' on Computer
Content-Type: text/plain; charset=UTF-8
Message-Id: <E1tJ5jA-000YGN-1M@Computer>
Date: Thu, 05 Dec 2024 01:52:44 -0500
/etc/cron.weekly/opensnitch-adlists:
[+] Checking list https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts.txt, urlhaus-filter-hosts.txt
[+] downloading new ads list... https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts.txt -> /home/blocklists/domains//urlhaus-filter-hosts.txt (64242, 63842) OK
[+] Checking list https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt, multiparty-trackers-hosts.txt
[-] ads list not updated yet: 507760, 507760 - https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
[+] Checking list https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt, firstparty-trackers-hosts.txt
[-] ads list not updated yet: 459805, 459805 - https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
[+] Checking list https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt, tracking-aggressive-extended.txt
[-] ads list not updated yet: 6205307, 6205307 - https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt
[+] Checking list https://adaway.org/hosts.txt, adaway-hosts.txt
[-] ads list not updated yet: 243454, 243454 - https://adaway.org/hosts.txt
[+] Checking list https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext, yoyo-adservers.txt
[!] No content-length header found: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
[.] Trying with Last-Modidifed
[+] downloading new ads list... https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext -> /home/blocklists/domains//yoyo-adservers.txt (94727, ) OK
[~] Done
I open my local mail with the same level of enthusiasm as はなちゃん, I swear.
Viewed items will be moved to $HOME/mbox upon close. You can always go back and check your read mail history by passing the mbox as the file to be read:
mail -f mbox
And often the programs which are not invoked from cron or anacron, such as those on systemd timers, can be configured to send mail when a condition is met. smartd, for example, can be configured to send messages to the local mailbox when some SMART parameter exceeds a safe threshold. Edit /etc/smartd.conf and append -m root into the entry for your hard drive. Which should look something like:
/dev/nvme0 -a -o on -S on -s (S/../.././02) -m root -M exec /usr/share/smartmontools/smartd-runner
And then restart smartd. You can test that mail notifications are working by temporarily replacing the -M exec argument with -M test and restarting smartd. Then check mail and you should see an entry with the subject “SMART error (EmailTest) detected on host:…”.
Other programs, whose default anacron entries instruct them to run quietly, may need some additional coaxing to make them talkative. For rkhunter, as one example, edit /etc/rkhunter.conf and uncomment MAIL-ON-WARNING=root. Now when rkhunter identifies suspicious changes, it will be sent to the local mailbox with the subject “[rkhunter] Warnings found for Computer”.
Return-path: <root@Computer>
Envelope-to: root@computer
Delivery-date: Sun, 24 Nov 2024 07:48:21 -0500
Received: from root by Computer with local (Exim 4.96)
(envelope-from <root@Computer>)
id 1tFC2H-0096VI-2m
for root@computer;
Sun, 24 Nov 2024 07:48:21 -0500
Subject: [rkhunter] Warnings found for Computer
To: root@computer
User-Agent: mail (GNU Mailutils 3.15)
Date: Sun, 24 Nov 2024 07:48:21 -0500
Message-Id: <E1tFC2H-0096VI-2m@Computer>
From: root@Computer
X-UID: 64
Status: OR
Please inspect this machine, because it may be infected.
All those failed sudo password attempt incident reports to which everyone likes to joke “I AM the administrator”? Those also find their way to mailbox entries with “*** SECURITY information for Computer ***”.
Mail logging can be further enhanced with the use of log readers like logcheck or logwatch. Logwatch analyzes system log files and mails items of interest (defined by perl script modules) to the user mailbox. Just set Output = mail within /etc/logwatch/conf/logwatch.conf.
Lastly, the most important component in all of this is you and your good habit of reviewing mail daily. A log is of no good to anyone if it never gets read. And a system such as this will help bring broken configs to your attention. It already helped me to discover a few hatches needing to be battened down which would have otherwise gone unnoticed.