Stefan Norlin's Blog

Dealing with slow tests and JRuby on Rails

stefan — Tue, 09 Dec 2008 20:09:00 +0000

I have started using JRuby in a Rails project and mostly I am very happy with it. However, one major issue is that Rails startup under JRuby is very slow… and I mean veeery slow. Running a single unit test under JRuby takes 15-16 seconds while running the same test using Ruby (MRI) takes around 0.15 seconds, a factor 100.

The reason for this is that startup of the Rails environment takes a long time under JRuby. When everything is up and running however, performance is great!

So, how to deal with this?

I figured the best way would be if it was possible to run tests using either JRuby or MRI. This would boost the speed in the development process while keeping the possibility or running tests with JRuby once or twice before deployment. So, how could this be accomplished.

Actually it was very simple. By adding a few

if RUBY_PLATFORM =~ /java/
 ...
end

in strategic places I was able to run almost everything under MRI.

These conditionals also needed to find their way into database.yml since it is necessary to choose either JDBC or ruby db driver depending on which “mode” we are running in.

Excerpt from database.yml:

test:
<% if RUBY_PLATFORM =~ /java/ %>
  <<: *defaults_jdbc
  url: jdbc:mysql://127.0.0.1/db_test?useUnicode=true&characterEncoding=UTF-8
<% else %>
  <<: *defaults
  host: 127.0.0.1
  database: db_test
<% end %>

So, now I can enjoy the power of Java whenever I like and I can still run 95% of my unit tests (which do not excercise Java-dependent code) as fast as ever possible!

VMware server performance

stefan — Fri, 18 Jul 2008 18:33:22 +0000

We started using VMware server for virtualization running CentOS 5 as host and CentOS 5 and other operating systems as guests. The servers were Dell PowerEdge 2950s with 4 SAS disks in RAID-10 setup.
However, performance was really poor especially for I/O intensive applications. After browsing around a bit I did the following:

Change I/O scheduler (host and guest)
echo deadline > /sys/block/sda/queue/scheduler
Increase disk read-ahead (host and guest)
/sbin/blockdev –setra 32768 /dev/sda
Set size of /dev/shm on host server to same as memory size on server (8 GB) via
mount -o remount,size=8G /dev/shm
And adding an entry in /etc/fstab for it to be set up on boot-time as well
tmpfs /dev/shm tmpfs defaults,size=8G 0 0
And then add some configuration options to the .vmx files (restart of virtual machines needed after the change):
mem.ShareScanTotal=0
mem.ShareScanVM=0
mem.ShareScanThreshold=4096
sched.mem.maxmemctl=0
sched.mem.pshare.enable = “FALSE”
mainMem.useNamedFile = “FALSE”
MemTrimRate = “0″
MemAllowAutoScaleDown = “FALSE”

This indeed the trick. Performance increased significantly!

Specifically I/O intensive operations does not affect the host server and all virtual machines as previously.

Adding new subnet to external Netscreen interface

stefan — Fri, 18 Jul 2008 18:20:13 +0000

We ran out of public IP addresses on our site where we map public IP addresses to services on internal servers via MIPs.

When we received more IP addresses in a different subnet I could not easily figure out how to use them until I ran across an article on the Juniper support site.

The solution was really simple:

Add a routing entry for the new subnet on the external interface (ethernet1) with “0.0.0.0″ as gateway
Network -> Routing -> Destination -> New
Add MIPs using addresses in the new IP subnet

How simple and it works great!

SMPP benchmark tools

stefan — Sat, 17 Mar 2007 12:42:39 +0000

It’s been a while since I got the time to write anything here. There has been a lot at work and some family stuff to take care of as well. Also I have been trying to get a hold Ruby on Rails, which is a really nice environment to build nice little web applications.

Yesterday we finally managed to publish our light-weight SMPP benchmarking tools as well on smpp.com. The emgload/emgsink utilities can really bring any SMSC on it’s knees. It also supports SMTP so it can be used to check mail server performance as well. On a Linux with Intel Xeon 3.0 GHz CPUs we reach around 25000 messages per second using emgload directly connected to emgsink.

smpp.com - Buying a domain

stefan — Mon, 22 Jan 2007 09:41:59 +0000

Last week I added a new experience to my list. Our company purchased a domain (smpp.com) from buydomains.com. I must say I was a little skeptical about paying a “horrendous” amount of money, using a credit card, for a domain. However, I must say the process was very swift and completed within 36 hours. Kudos to buydomain.com for that.

When the purchase completed the domain can be maintained via a control panel at domaindiscovery.com. From there DNS servers could be changed and possibly a transfer to another registrar initiated. Simple enough.

So, if you find a domain that you want badly, it is for sale from buydomains.com and you can agree on a price, then I think you can safely go ahead and acquire the domain.

CentOS and JFS

stefan — Thu, 28 Dec 2006 11:07:23 +0000

JFS (Journaled File System) is a file system from IBM. I have always really liked this file system since I first came across it, working with AIX, but I have never used it in a Linux environment until now. When you read comparisons of different file systems it seems JFS always comes out in a top position. For Linux it is sometimes beaten by XFS, but I think that is due to the fact that XFS is more widely used on Linux.

For some reason RedHat seems to have taken out support for JFS in RHES, but it is available for CentOS using the kernel in the CentosPlus repository. The process to upgrade an existing Centos installation to support JFS is simply to update the kernel with the kernel in centosplus and install jfsutils:

yum –enablerepo=centosplus install kernel-smp
yum –enablerepo=centosplus install kernel-smp-devel
yum install jfsutils

Then you probably need to change the “default” row in grub.conf and reboot to start using the new kernel with JFS support.

To create a JFS file system on a new partition use fdisk and mkfs -t jfs.

I think you need to stick to ext3 for the root and boot file systems though and there is no path (at least that I know of) for migrating an existing ext3 file system to JFS “in place”.

Upgrading Postfix on Centos (and RedHat)

stefan — Sun, 12 Nov 2006 15:04:00 +0000

Postfix is a great e-mail server and you would usually want to keep up with new versions more aggressively than RedHat publishes RPMs for new version. However, I have never succeeded well in compiling Postfix from source with SASL support enabled. The resulting binaries (compiled towards the RedHat SASL rpms) always seem to core dump and mis-behave for reasons I have never understood.However, there is a solution… building an RPM from source and using that RPM for updating.

The process is described in great detail at the Postfix RPM Packages site (http://postfix.wl0.org/), which contains both binary RPMs (without SASL and TLS support) as well as source RPMs.

I found out about this site when I (once again) had tried to compiled from source but kept receiving error messages similar to the ones below which I found out also caused Postfix smtpd to core dump.

postfix/smtpd[27650]: warning: SASL per-connection security setup; invalid parameter supplied
postfix/smtpd[27650]: fatal: SASL per-connection initialization failed

However, when downloading the Postfix 2.3.4 source RPM, building my own binary RPM and updating using that everything worked perfectly. The OS was CentOS 4.4.

Postfix and greylisting using postgrey

stefan — Sun, 12 Nov 2006 12:53:41 +0000

Our mail server receives around 10.000 e-mails per day. On weeekdays around 70% is spam, during weekends more than 90% is spam. Lately our mailserver load has been very high leading to poor reponse times. As a result we decided to implement greylisting.

Greylisting means that the first time a new combination of sender/recipient is seen by the receiving mail server the e-mail is rejected with a temporary SMTP error code. Something like “temporary failure, please try again later”. Most (if not all) mail servers will respect this and retry delivery a bit later. However, spam senders usually try delivery only once and therefore does not come back after receiving the temporary failure. The only disadvantage is a bit longer delay, from a few minutes up to an hour, before delivery.
We use Postfix as mail server and our choice fell on Postgrey (http://isg.ee.ethz.ch/tools/postgrey/) since it seemed easy to implement, efficient and reliable.

After implementing Postgrey (with a one minute delay) the amount of e-mails actually received by our mail server went from around 10.000 per day to 1.000 on weekends and the corresponding share of spam consequently went from over 90% to around 40%. Also, before greylisting the server could spend 40-50% waiting for I/O (spam filters seem to take resources) while afterwards the server is almost always idle.

Please note that we choose to whitelist info and support e-mail addresses (in other words, those are not subject to greylisting) in order to allow for incoming information requests (new potential customers) and support tickets (highest possible service level towards existing customers) to be delivered as quickly as possible.However, the amount of spam to these addresses seems to be a relatively small part of the total spam volume.

OTP via SMS for PAM

stefan — Tue, 07 Nov 2006 08:41:18 +0000

Now I have finished a PAM module for sending One-Time Passwords (OTPs) via SMS when using Pluggable Authentication Module (PAM) in Linux.

For more information »

HTTP sniffer on Linux - urlsnarf

stefan — Wed, 04 Oct 2006 06:15:07 +0000

We have experienced high load on one of our web servers lately (Apache 2.0.59 on Linux). So, I started thinking about how to analyze this in a bit more detail. We are running multiple virtual hosts (around 60) so when using tcpdump it is not easy to see which URLs are requested, therefore I looked around for some HTTP-oriented sniffer. My search ended with “urlsnarf” from the “dsniff” package.

Last change in dsniff was in 2000 and it has libnet and libnids dependencies. It took me a couple of tries (and a minor patch) to get it to compile but when done I was happy with the result. urlsnarf simply sniffs HTTP traffic on one or more parts and generates output in the common log format making it easy to see which URLs are reqeusted.

I could then see what URL were the most frequently requested and continue on to the awstats logs which we have for each virtual host. Then it was quite easy to find a couple of pages with very large images that were very frequently requested and from there we could take relevant actions to reduce the HTTP load.