Apache stats for bots

Last time my server got overload due to heavy queries from bots. I needed to know which bot is so malicious. I wrote simple script to parse apache logs to search boot.

#!/usr/bin/perl use File::Basename; use Time::Piece; use Term::ANSIColor qw(:constants); if (-T $plik){ open(PLIK,"$plik")||die "nie mozna otwoprzyc pliku: $plik!!!\n"; } elsif(-B $plik){ open(PLIK,"zcat $plik |")||die "nie mozna otwoprzyc pliku: $plik!!!\n"; } else { print "Pliku: $plik nie mozna otworzyc\n"; exit; } while(defined($log=)){ my ($host,$date,$reqtype,$url,$proto,$status,$size,$referrer,$agent) = $log =~ m/^(\S+) - - \[(\S+ [\-|\+]\d{4})\] "(GET|POST)\s(.+)\sHTTP\/(\d.\d)" (\d{3}) (\d+|-) "(.*?)" "([^"]+)"$/; if ($status eq "200" && $reqtype eq "GET" && $agent =~ m/bot/i){ my $dt = Time::Piece->strptime($date, '%d/%b/%Y:%H:%M:%S %z'); $date= $dt->strftime('%Y-%m-%d'); $slugnumber{$agent}{$date}{$host}++; $bot{$agent}++; } } close(PLIK); foreach $klucz (sort keys %slugnumber){ print "\n================================================\n"; print BOLD,BLUE,"\n $klucz \n",RESET; foreach $data (keys %{ $slugnumber{$klucz} }){ print BOLD,BLUE,"\n $data \n",RESET; foreach $ipek (keys %{ $slugnumber{$klucz}{$data} }){ print "$klucz $data [$ipek] : $slugnumber{$klucz}{$data}{$ipek}\n" } } }

Below is output:

testing> perl ipstats.pl /var/log/apache/access.log ================================================ Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/) 2014-08-05 Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/) 2014-08-05 [125.209.211.199] : 1 2014-08-04 Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/) 2014-08-04 [125.209.211.199] : 1 ================================================ msnbot/2.0b (+http://search.msn.com/msnbot.htm) 2014-08-05 msnbot/2.0b (+http://search.msn.com/msnbot.htm) 2014-08-05 [65.55.213.247] : 10 msnbot/2.0b (+http://search.msn.com/msnbot.htm) 2014-08-05 [65.55.213.243] : 4 msnbot/2.0b (+http://search.msn.com/msnbot.htm) 2014-08-05 [65.55.213.242] : 2

Grep - get first and last line

Last time I had to search my logs for certain message. I needed to connect this with user login/logout time.

I needed estimated time of "start" and "end " occurences in logs ( logs which contains huge messages with different time and same message).

I used sed and grep to this:

root@testing:~# for i in `ls /var/log/syslog/syslog*`;do zgrep 'port 1099' $i | sed -n '1p;$p'; done; Jun 25 08:18:01 testing sshd[33286]: error: connect_to x.y.z.c port 1099: failed. Jun 25 11:30:52 testing sshd[45831]: error: connect_to x.y.z.d port 1099: failed. Jun 24 07:55:04 testing sshd[64527]: error: connect_to x.y.z.d port 1099: failed. Jun 24 11:53:13 testing sshd[64527]: error: connect_to x.y.z.c port 1099: failed. Jun 23 08:59:52 testing sshd[34130]: error: connect_to x.y.z.c port 1099: failed. Jun 23 15:28:38 testing sshd[34130]: error: connect_to x.y.z.d port 1099: failed. Jun 20 08:24:51 testing sshd[64526]: error: connect_to x.y.z.c port 1099: failed. Jun 20 10:55:46 testing sshd[7805]: error: connect_to x.y.z.c port 1099: failed.

Read linux dmesg with date

Simple script to read dmesg from linux (contains real date no miliseconds)
----------------------------------------------------------------------------------------------------------
#!/usr/bin/python
import sys
import re
import time
import datetime
import fileinput
import os

try:
  uptime = open('/proc/uptime','r')
except IOError as (errno, strerror):
  print "Wystapil blad otwarcia pliku :\n %s\n" % (strerror)

czas = time.time()

uptime_seconds = czas-float(uptime.readline().split()[0])

p = os.popen('dmesg')

for line in p.readlines():

  re1='.*?'
  re2='(\\d+)'
  re3='(\\.)'
  re4='(\\d+)'

  rg = re.compile(re1+re2+re3+re4,re.IGNORECASE|re.DOTALL)
  m = rg.search(line)
  if m:
      int1=float(m.group(1)+m.group(2)+m.group(3))
      szukaj=(m.group(1)+m.group(2)+m.group(3))
      czas = uptime_seconds + int1
      czas = datetime.datetime.fromtimestamp(int(czas)).strftime('%Y-%m-%d %H:%M:%S')
   
      zwrot = re.sub(szukaj,czas,line.rstrip(),1)

      print zwrot

Log coloring with bash

Below is link to short script I've made to see different logs with highlight different patterns.
How to use it ?

shamrock@alucard:~/dev/shell$ ./cgrep.sh
Usage : ./cgrep.sh <-g|--grep,-l|--less,-m|--more,-t|--tail> file [line|pattern]
-g, --grep - use grep with highlight
-l, --less - use less with highlight
-m, --more - use more with highlight
-t, --tail - use tail with highlight
line - highlights whole line
pattern - highlights whole pattern
Available colours: black,red,green,yellow,blue,magenta,cyan,white
Available bold colours: blackb,redb,greenb,yellowb,blueb,magentab,cyanb,whiteb

It is possible to use sed different patterns inside quotation marks i.e.:

cgrep sample.log pattern redb "Process target" blueb [Cc]urrent magentab "[0-9]\{3,10\}"

How doe it work in practice?

shamrock@alucard:~/dev/shell$ cat test
Lorem ipsum
Test lorem ipsum
test test lorem ipsum
shamrock@alucard:~/dev/shell$ ./cgrep.sh --grep test redb [Ll]orem

Lorem ipsum
Test lorem ipsum
test test lorem ipsum

cgrep.sh