Grep - get first and last line

Last time I had to search my logs for certain message. I needed to connect this with user login/logout time.

I needed estimated time of "start" and "end " occurences in logs ( logs which contains huge messages with different time and same message).

I used sed and grep to this:

root@testing:~# for i in `ls /var/log/syslog/syslog*`;do zgrep 'port 1099' $i | sed -n '1p;$p'; done; Jun 25 08:18:01 testing sshd[33286]: error: connect_to x.y.z.c port 1099: failed. Jun 25 11:30:52 testing sshd[45831]: error: connect_to x.y.z.d port 1099: failed. Jun 24 07:55:04 testing sshd[64527]: error: connect_to x.y.z.d port 1099: failed. Jun 24 11:53:13 testing sshd[64527]: error: connect_to x.y.z.c port 1099: failed. Jun 23 08:59:52 testing sshd[34130]: error: connect_to x.y.z.c port 1099: failed. Jun 23 15:28:38 testing sshd[34130]: error: connect_to x.y.z.d port 1099: failed. Jun 20 08:24:51 testing sshd[64526]: error: connect_to x.y.z.c port 1099: failed. Jun 20 10:55:46 testing sshd[7805]: error: connect_to x.y.z.c port 1099: failed.

Get all files from remote directory using wget

wget -A pdf,jpg -m -p -E -k -K -np http://site/path/

Check SSL expiration date

Recently I needed to check out my domains due to SSL expiration date. I used script below - it frees up time.

How does it work ?

I need file which contains domeins in form : address:port In loop I check every domain with parameter "-enddate" (certificate expiration date) using openssl. I change dates to timestamp and subtract expiration date from date "now+30days". If result timestamp is lower than daystimestamp (which is number of days in seconds) it warnings me that certificate will expire soon.

shamrock@alucard:~$ cat check_ssl_expiration.sh days=30 daystimestamp=`expr $days \* 86400` for domain in `cat domeny.txt` do expire=`echo | openssl s_client -connect $domain 2>/dev/null | openssl x509 -noout -enddate | awk -F \= '{print $2}'` out=`date -d "$expire" "+%s"` in=`date --date="$days days" "+%s"` res=`expr $out - $in` if [ $res -lt 0 ]; then echo "ALARM !!! DOMAIN $domain EXPIRED ON $expire" elif [ $res -lt $daystimestamp ]; then echo "ALARM !!! DOMAIN $domain WILL EXPIRE ON $expire" else echo "$domain WILL EXPIRE ON $expire" fi done

Usage:

shamrock@alucard:~$ bash check_ssl_expiration.sh poczta.onet.pl:443 WILL EXPIRE ON Oct 14 06:10:33 2014 GMT ALARM !!! DOMAIN some.kind.of.mons.tr:443 EXPIRE ON Jul 3 13:19:18 2013 GMT

File contains domains:

shamrock@alucard:~$ cat domeny.txt poczta.onet.pl:443 some.kind.of.mons.tr:443

Day Against DRM

"Day Against DRM" on International Day Against DRM May 6

SSL certificate: DER, PEM and error 20

I had an SSL certificate in a https server, but whenever I connected to this my site, I saw certificate error. First I check that my certificate is in DER form which is not accepted by my server. I have to convert it to PEM :

root@prod:~/cert# openssl x509 -in my_cert.cer -out my_cert.pem -inform DER -outform PEM

Than I saw that I have stil problems with certificate

root@prod:~/cert# openssl verify my_cert.pem: [details removed] error 20 at 0 depth lookup:unable to get local issuer certificate

I googled and found that I should get issuer certificate and combine it with my certificate. I downloaded issuer certificate in DER format and converted it to PEM.

root@prod:~/cert# openssl x509 -in l4.cer -out l4.pem -inform DER -outform PEM

Than I checked if everything is OK:

root@prod:~/cert# openssl verify -CAfile l4.pem my_cert.pem: OK

Final step was combined my_cert.pem with l4.pem :

root@prod:~/cert# cat my_cert.pem l4.pem >> my_cert.crt

I copied my_cert.crt and my_cert.key to /etc/ssl/ and configured my nginx to use it. No more certificate error.

Solr configure logging to file

First download solr and commons-logging:

root@testing:# wget http://ftp.ps.pl/pub/apache/lucene/solr/4.7.1/solr-4.7.1.tgz root@testing:# wget http://ftp.piotrkosoft.net/pub/mirrors/ftp.apache.org//commons/logging/binaries/commons-logging-1.1.3-bin.tar.gz

Unpack both.

root@testing:# tar -zxf solr-4.7.1.tgz root@testing:# tar -zxf commons-logging-1.1.3-bin.tar.gz

Copy solr-4.7.1.war to temp directory i.e:

root@testing:# mkdir tmp root@testing:# cd tmp root@testing:tmp# cp ../solr-4.7.1/dist/solr-4.7.1.war .

Unpack solr*.war

root@testing:tmp# jar -xf solr-4.7.1.war

Copy from solr-4.7.1 log4j*.jar and slf4j*.jar to WEB-INF/lib/

root@testing:tmp# cp ../solr-4.7.1/example/lib/ext/{log4j-1.2.16.jar,slf4j*} WEB-INF/lib/

Copy from commons-logging commons-logging*.jar :

root@testing:tmp# cp ../commons-logging-1.1.3/commons-logging-1.1.3.jar WEB-INF/lib/

Create directory classess in WEB-INF/

root@testing:tmp# mkdir WEB-INF/classes

Copy from solr*/example/resources/ log4j.properties to WEB-INF/classes

root@testing:tmp# cp ../solr-4.7.1/example/resources/log4j.properties WEB-INF/classes/ root@testing:tmp# cat WEB-INF/classes/log4j.properties # Logging level solr.log=/var/log/solr log4j.rootLogger=INFO, file, CONSOLE log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x \u2013 %m%n #- size rotation with log cleanup. log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.MaxFileSize=4MB log4j.appender.file.MaxBackupIndex=9 #- File to log to and log format log4j.appender.file.File=${solr.log}/solr.log log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%-5p - %d{yyyy-MM-dd HH:mm:ss.SSS}; %C; %m\n log4j.logger.org.apache.zookeeper=WARN log4j.logger.org.apache.hadoop=WARN # set to INFO to enable infostream log messages log4j.logger.org.apache.solr.update.LoggingInfoStream=OFF

Create log directory which configured in log4j.properties and change owner to user which running tomcat

root@testing:tmp# mkdir /var/log/solr root@testing:tmp# chown tomcat7:tomcat7 /var/log/solr

Pack everything back to solr*.war

root@testing:tmp# jar -cf solr-4.7.1.war META-INF WEB-INF admin.html css favicon.ico img js tpl

If everything is OK you should enjoy logs in log directory:

root@testing:tmp# cat /var/log/solr/solr.log INFO - 2014-04-11 12:23:58.890; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() INFO - 2014-04-11 12:23:58.916; org.apache.solr.core.SolrResourceLoader; Using JNDI solr.home: /srv/solr INFO - 2014-04-11 12:23:58.919; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/srv/solr/' INFO - 2014-04-11 12:23:59.145; org.apache.solr.core.ConfigSolr; Loading container configuration from /srv/solr/solr.xml INFO - 2014-04-11 12:23:59.358; org.apache.solr.core.CorePropertiesLocator; Config-defined core root directory: /srv/solr INFO - 2014-04-11 12:23:59.372; org.apache.solr.core.CoreContainer; New CoreContainer 1409622974 INFO - 2014-04-11 12:23:59.372; org.apache.solr.core.CoreContainer; Loading cores into CoreContainer [instanceDir=/srv/solr/] INFO - 2014-04-11 12:23:59.398; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 0 INFO - 2014-04-11 12:23:59.398; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: null INFO - 2014-04-11 12:23:59.406; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 0 INFO - 2014-04-11 12:23:59.406; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20 INFO - 2014-04-11 12:23:59.406; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0 INFO - 2014-04-11 12:23:59.407; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647 INFO - 2014-04-11 12:23:59.407; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5 INFO - 2014-04-11 12:23:59.407; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1 INFO - 2014-04-11 12:23:59.407; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false INFO - 2014-04-11 12:23:59.588; org.apache.solr.logging.LogWatcher; SLF4J impl is org.slf4j.impl.Log4jLoggerFactory INFO - 2014-04-11 12:23:59.589; org.apache.solr.logging.LogWatcher; Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] INFO - 2014-04-11 12:23:59.591; org.apache.solr.core.CoreContainer; Host Name: INFO - 2014-04-11 12:23:59.677; org.apache.solr.core.CorePropertiesLocator; Looking for core definitions underneath /srv/solr INFO - 2014-04-11 12:23:59.678; org.apache.solr.core.CorePropertiesLocator; Found 0 core definitions INFO - 2014-04-11 12:23:59.680; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/var/lib/tomcat7 INFO - 2014-04-11 12:23:59.680; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done INFO - 2014-04-11 12:24:15.866; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={indexInfo=false&_=1397211686406&wt=json} status=0 QTime=69 INFO - 2014-04-11 12:24:15.941; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/system params={_=1397211686554&wt=json} status=0 QTime=16 INFO - 2014-04-11 12:24:20.785; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/logging params={_=1397211691410&since=0&wt=json} status=0 QTime=2 INFO - 2014-04-11 12:24:22.578; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={_=1397211693200&wt=json} status=0 QTime=0 INFO - 2014-04-11 12:24:24.230; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/properties params={_=1397211694858&wt=json} status=0 QTime=1

Packt’s amazing Buy One, Get One Free

Check out #Packt’s amazing Buy One, Get One Free offer #Packt2k