Fighting PHP spam

A good friend owns a very old joomla website which has been hacked. It is massively sending spam, how can we investigate this?

First of all, reject outgoing mails in your firewall (ferm.conf style):

  chain OUTPUT {
    policy ACCEPT;

    # connection tracking
    mod state state INVALID DROP;
    # drop every outgoing mail
    proto tcp dport 25 REJECT;
  }

Now we have time to investigate. A good start is a look at the mailq:

...DD460113C27 1388 Tue Dec 1 22:30:58 emily_smith@foobar.de
(delivery temporarily suspended: connect to mailin-02.mx.aol.com[152.163.0.100]:25: Connection refused)
anahas5353@aol.com

-- 441 Kbytes in 270 Requests.

so already 270 spam mails, ups. You can force php-fpm to extend mail headers, log the mail() calls to a file but also disable the mail function:

php_admin_flag[mail.add_x_header] = on
php_admin_value[mail.log] = /home/www.fuckedupwebsite.de/logs/mail.php.log
php_admin_value[disable_function] = mail

Now we wait a few minutes until more mails were tried to be sent, than take a look in the mail.php.log:
[30-Nov-2015 11:48:09 Europe/Berlin] mail() on [/home/www.fuckedupwebsite.de/htdocs/templates/protostar/images/sql59.php(1962) : eval()'d code:775]: To: ercole.giona@poste.it -- Headers: Date: Mon, 30 Nov 2015 11:48:09 +0100 From: Bernice Lowe Message-ID: X-Priority: 3 X-Mailer: PHPMailer 5.2.9 (https://github.com/PHPMailer/PHPMailer/) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_ed6348634bcdc8a0ab1da27532c7f85d" Content-Transfer-Encoding: 8bit

So we found the first infected file, sql59.php. Delete it, clean the mailqueue with postsuper -d ALL and take a look at the /var/log/mail.log if there are new entries. It is possible that new mails appear, but nothing gets logged into the mail.php.log, than we need to take a look at the origin of the mail. We can dump a mail that is stuck in the postfix queue with postcat + the mail ID from mailq:
postcat -q DD460113C27 | grep X-PHP-Originating-Script
X-PHP-Originating-Script: 1044:template.php(1961) : eval()'d code

Now we know the name of the file, but not the path, go into the docroot via ssh and find all php files with this name + that do an eval():

find . -type f -name template.php -exec grep -i -l 'eval' '{}' \;
./tmp/install_5523c3dd4f727/installer/adapters/template.php
./tmp/install_553e2d7f39ee9/installer/adapters/template.php

perfect, just a few matches. take a look at them, identify the correct one, delete it, clear the queue, watch the mail.log, repeat for every new mail.

This is a good solution to detect the spamming php scripts, but this doesn’t fix the issue which allowed $bad_person to upload these files to the webspace.