Benchmarking Mongrel, Apache, Rails, etc.

I'm doing some benchmarking of a Rails based web app (technically, a web service) to try to establish some baselines and to use in assessing build outs/hardware deployment, etc. To start off, I wanted to establish a baseline by using pure static content. The performance I got didn't seem quite right to me. Below is a message I sent to the mongrel-users mailing list. I figured this was a good blog item, and would like to solicit help on this...

I'm trying to do some initial benchmarking of our setup, mainly just to establish baselines. I'm essentially using the process Zed outlines in a previous message:
http://rubyforge.org/pipermail/mongrel-users/2006-May/000200.html

What I'm running into is that Mongrel appears only half as fast as Apache when serving a small static HTML file. If I then add in Apache with mod_proxy_balancer, going to a single Mongrel, it drops down to nearly about a third of what pure static Apache will do. This seems bogus to me, and I suspect I have either some configuration problem, or something. My understanding from what I've read is that Mongrel should be fairly close to Apache when serving static content (at least not only 50% as fast). Is that right as a generalization?

Here's some info to back this up.
- Server: HP DL360 dual 3.0GHz Xeons, 4GB RAM, 10k RPM SCSI disks
- OS: Fedora Core 6, up to date as of a few days ago
- Apache 2.2.3-5, with mod_proxy_balancer
- Mongrel 1.0.1, mongrel_cluster 0.2.1
- serving a static HTML file from the Rails app's public directory
- Testing using an identical server, that sits above this one in the rack, connected to a switch (so it's one hop)
- Using httperf for testing, with the following command:
httperf --server lab05 --port 80 --uri /mongrel_alive.html --num-conns 10000
The number of connections I vary to be near the 10 second mark...

Here's the results:

- Just Apache, num-conns=15000, ~1400 req/sec
- Direct to Mongrel on port 5000, num-conns=8000, ~740 req/sec
- Apache mod_proxy_balancer to a single Mongrel, num-conns=5000, ~475 req/sec

So, I'm incurring a massive penalty for the balancer/cluster setup. In production we will be using hardware load balancers, but even still, my understanding was that instead of a drop from 1400 to 740, it should be somewhat closer (say at least over 1000).

What would folks suggest, or what comments do you have?

Shortly after that, I decided to throw Pen and Nginx in the mix just as a random check on this. I haven't used them before so relied on what I found on the net, etc. for config, thus I could potentially get better results, but they yielded:

• Nginx with one Mongrel: ~612 req/sec
• Pen with one Mongrel: ~670 req/sec

I can see Apache being slower than these, as it's a much bigger and more complex app. So the question remains, should straight Mongrel be half as fast as Apache? And, should the load balancers affect the performance that much (although Pen is having only a 10% impact here).

It's Comcastic!

Well, I hope it will be Comcastic! Ordered up new phone, cable TV, and Internet service all in one with Comcast today, for our new house in Eugene. Yep, that's phone over cable. They had the fastest net service in the area (8MBit down, 784k up). We have a cable modem now, and have had really good luck with it, so I hope that continues.

The cable will include HD, and we have two of the new TiVo Series 3 HD units on the way from Amazon. Comcast will give us four cable cards (two per box) to put in them. Pretty cool, although more for my wife than me, as I barely watch TV these days.

The phone is pretty good too. It includes unlimited local and long distance calling, and 12 features (stuff like voice mail, call forwarding, etc.). All of this, setup for two rooms, etc, etc., is $130/month ($15 of that is for the 3 extra cable cards - they give you one for free). Seems fairly good, although if you did a lot of long distance calling this would probably start to pay off pretty well. Our cable modem now costs me $68/month (this is for the "residential router" package, which means I can "legally" (i.e. I don't violate the ToS) put a router and as many computers as I want behind it), my phone typically costs $45, and our DirecTV (with Tivo) costs about $70/month. So, that's $183, and thus I figure we're saving about $58/month (minus some, as the $130 through Comcast doesn't include the Tivo service).

ApolloCamp

For those of you interested in Apollo, check out the free ApolloCamp, on March 16th at the Adobe San Francisco office. See you there!

Using multiple Perforce depots and TextMate's P4 Bundle

I'm a big TextMate user. We're also a Perforce shop. TextMate has a P4 bundle, and it works great, but it's main problem is that it has minimal knowledge of your P4 settings, and certainly doesn't handle multiple P4 clients or depots. For example, I have a few different clientspecs I work with, and the server, port, clientspec, etc. all differ. So, it's not feasible to use a single .p4config file, or set the values in TextMate or globally in the environment. This makes using the P4 bundle essentially impossible for all but whichever one you pick as your main clientspec.

To alleviate this, and based on how I organize my code on my machine, I wrote a simple little script that is what I set the "TM_P4" textmate environment variable to. This script knows that all my code is kept in subdirectories off my ~/Code directory. So, I then stash specific .p4config files at the root of each of these as needed, and the script looks for those based on the currently open file, and sets that for the P4 settings. This seems to work well. Here's the script in case it's useful to you:

#!/bin/sh
#
# Script to run p4 commands in TextMate, and set the P4 variables
# depending on the location of the file being operated on.

# Where all my code lives
CODE_PATH="${HOME}/Code/"

# figure out the Code subdirectory for the current file in TextMate
P4_CONFIG_PATH=${TM_FILEPATH##$CODE_PATH}
P4_CONFIG_PATH=${P4_CONFIG_PATH%%/*}

# Reconstitute as path to potential config file
P4_CONFIG_FILE="${CODE_PATH}${P4_CONFIG_PATH}/.p4config"

# if dir contains a .p4config, set P4CONFIG to that
if [ -f $P4_CONFIG_FILE ]; then
 export P4CONFIG="$P4_CONFIG_FILE"
else
 export P4CONFIG="~/.p4config"
fi

# now do p4
/usr/local/bin/p4 $*

Loving the Blackjack, Wishing For Ubiquitous 3G

I've really been loving my Blackjack. The one bummer is how spoiled I am getting with 3G. When we move to Eugene there won't be 3G and I already find EDGE sucks. Hey Cingular - please put 3G everywhere!

Twitter: cool or just silly?

Check out Twitter, a relatively new web app. It's very simple, it shows the status of people (here's ). Status is any short bit of text the person wants to state. When I first saw this, I thought it was just plain silly. Gee, do you really need up to the minute (or whatever) updates on what people are doing? Could people really keep it up to date enough to be http://www2.blogger.com/img/gl.link.gifinteresting/useful? Would people actually use it?

Tonight I was checking it out, and noticed that they have an ActionScript 3 library so as to make using their public API trivial from Flash/Flex apps (I'm using Flex a fair bit these days). I then noticed that you can update your status via IM as well as via SMS from your mobile phone. Ok, now it's getting a bit more interesting. Maybe you can keep it fairly up to date with relative ease.

Also the AS3 lib and API, as well as the ability to shield your status from the public (only your Twitter "friends" can see it), made me think about tieing this in to the Agile tasking tool we're working on for RailsConf. It could potentially be interesting to have a "work" Twitter account, and use the API to show people's status. You could then tie it in to the tasking app, and anytime anyone changed a task's status (for example, they set a task to now be "in progress", the app would shoot out a Twitter message (aka a "tweet") to indicate that. Then using the public API you could show your team's "timeline" on a web page. Useful? Questionable. Geeky fun? You bet.

moblog from my new Blackjack

I got a Blackjack mobile phone the other day and just set it up for moblogging. This is my first test. As for the phone, I am truly loving it so far!

start

Essential COM autographed by Don Box, and other books

I've put up more books on eBay, including a copy of "Essential COM", signed by Don Box himself (eBay listing). I've also got books on Zope and Plone (listing, which has links to the others), and various other Windows books. I didn't sell a single one of the Java books I had up there, which kind of surprised me (nearly all were listed at $4).

Moving to Eugene, Oregon

We (my wife, kids, and I) will be moving to Eugene, Oregon in a couple months (probably in April). We signed the counter-offer on a house today, so hopefully escrow goes well, and we'll own it in March. We're pretty excited, but nervous of course as well. We don't know anyone there. My wife's parents and brother and his family will move later this year (no commune jokes please!). Looking forward to it though. It was a move completely by choice, and I will continue to work for Adobe from there (from the home office).

We spent last Saturday driving around with our real estate agent, and wound up still loving a house we'd seen back in October, which had recently had the price dropped, so we went for it. I have a map showing our drive and various waypoints up on Discover Machine. I recently got a GPS unit, so used it on the drive, which was really quite handy. I put a waypoint in for every house we went in to, as well as lunch spot, and so on. It was cool to look at the overall route afterwards and see where all we went from a 10,000 foot type of view. I uploaded my data from my GPS into Google Earth Plus, and then saved a KML file out, which can be uploaded to Discover Machine (a site done in Rails :).

This will of course be a very convenient location for RailsConf later this year, as well as cool conferences like OSCON.

I'm Presenting at RailsConf 2007

I just found out yesterday that the proposal my co-worker and I submitted for RailsConf 2007 was accepted! Quite exciting. I hope to say more later, but for the moment, the talk will be about using Rails and Adobe's new Apollo technology.

Adobe to Release PDF for Industry Standardization

Check out the press release! Adobe is releasing the FULL PDF 1.7 spec to get published as an ISO standard. Pretty cool.

I'm Now A GPS Geek

A good friend who is quite into GPS, and happens to work on the Google Earth team finally got me hooked on GPS. I picked up a Garmin GPSmap 60CSx unit at REI after attending their intro class the other day. I then proceeded to figure out how to get it all going on the Mac. Many thanks to Ben Sinclair, as he pretty much covers every angle in his article, "How to use a Garmin GPS with your Mac."

I'm using the combination of an Intel Mac, with Google Earth Plus, and then Parallels for the Garmin MapSource City Navigator software. I haven't gone for any TOPO stuff yet, and Google Earth seems to cover most of what I want trail/bike/ski wise so far (if I get more into backcountry skiing, then I will likely want the TOPO stuff).

Anyway, this is really cool. I can't wait to use it on my mountain bike. The first heavy use will be this weekend when we go to Eugene to scout houses and so on. I will track the entire day on Saturday and then import into Google Earth, etc. Also, because I have the City Navigator stuff on there, I've got intricate detail of city streets, businesses, turn-by-turn directions, and so on. Slick.

Note, the "x" series units from Garmin (I think all of them anyway) have microSD cards. You can store a ton of map data on these things. For example, the City Navigator stuff I put on mine, for which I did most of Oregon, and a good chunk of Northern CA, was about 50MB. The 60CSx comes with a 64MB microSD, but I have a 512MB on the way (I'd have gone bigger, but I heard that with the 1GB cards it creates a noticeable slowdown on the unit while it sifts through all that data, and 512MB seemed like plenty given what you can put in about 50-100MB).

Java Books for Sale

I just posted a slew of Java books for sale on eBay. They are all priced between $3-6. Some are current, some aren't. Here's a listing (and the link goes to the eBay listing):

• Graphic Java 2, Volume 2: Swing (3rd Edition)


• Graphic Java 1.2, Volume 1: AWT, Third Edition


• The Java Class Libraries, Volume 1: java.io, java.la...


• Patterns in Java, Volume 1, A Catalog of Reusable De...


• Patterns in Java, Volume 2


• Programming Web Services with XML-RPC (O'Reilly Inte...


• Java Web Services


• JSTL: Practical Guide for JSP Programmers (The Pract...


• Web Development with Java Server Pages


• Designing Enterprise Applications with the Java(TM) ...


• JavaServer Pages, 3rd Edition


• Java Threads, Second Edition Java 2


• Java(TM) Native Interface: Programmer's Guide and Sp...


• Art of Java Web Development: Struts, Tapestry, Commo...


• Building Imaging Applications with Java(TM) Technolo...

I will have more, on other subjects to come, but I've started to prune my library, in preparation for our move to Eugene.

For the curious, I use iSale on the Mac to post on eBay, it's superb. It easily filled in most of the auction on each book simply by me letting it search Amazon for the ISBN and importing the info (all automatically). The new version of iSale also has iSight support for taking quick pics, etc. But, the listing above was a simple hack by exporting my auction items from iSale (as CSV), plopping that in Excel and dropping out all but the eBay ID column and title column, then dropping that in TextMate, and simply wrapping those with the above HTML, trivially done by using TextMate's awesome column editing abilities (a regex search and replace would've done fine too, but this seemed even faster).

Skiing Silverton in Early January

Just firmed up a quick trip with two friends to ski Silverton (Colorado) in early January. One of the guys, Matt, just skied there today and said it was stellar. It's an interesting place, as it only has one lift, and during the main part of the season you must ski guided. This will be my first real backcountry excursion, although it should be relatively mellow in that regard (no skinning, hopefully minimal hiking as my fitness level is quite poor right now).

I also am hoping to take a lot of pictures. And, given my weak fitness, I expect to not last a whole day of this kind of skiing, so hopefully can also shoot some action shots of Matt and Barry. Matt will easily rip it up, as he's been skiing this kind of stuff all over CO and UT for the last few weeks (the lucky bum).

Dtrace: the anal probe

So I'm listening to the ps pipe grep podcast the other day, and with his always interesting sense of humor, Jason provides this superb quote,

Dtrace is an anal probe for your application.

Classic.

John Nack Slashdotted My Flickr Photos of Adobe

The good John Nack mentioned my photos of Adobe that are on Flickr. And, of course this was about as close as they will get to getting Slashdotted, er, I guess that'd bhttp://beta.blogger.com/img/gl.link.gife Dugg these days.

I should of course make one correction. John mentions I'm a Photoshop engineer. This is only partly true. I used to be full time on Photoshop, but my features are complete (wait for the Photoshop CS3 Beta tomorrow, and I'll blog about them). I'm now actually back to doing web services and web applications (hopefully more on that early next year).

The Photoshop beta is very exciting for those of us who've worked on it though. And of course it's great for folks who've been clamoring for a Mactel version. This will be on tomorrow.

There are some other interesting things on Labs recently as well. For example, check out Kuler and Soundbooth. Really fun and great stuff that's been happening at Adobe. And of course there's all the great Flash and Flex related bits, which I have personally been using a lot.

Amazon's UnSpun done with Ruby on Rails

Pretty cool... Amazon's latest service/site, UnSpun, was implemented with Ruby on Rails. I knew Amazon was looking at Rails, but this is great to see.

Winter Evening in Old Town Auburn

Winter Evening in Old Town Auburn
Originally uploaded by Christopher Bailey.

Gary and I went out for a quick photo shoot in the drizzle this evening. We started near the courthouse in Auburn, and then walked around Old Town. I will have to go back, as it took a while to start finding shots I was interested in. But, I got a few. Gary also took some nice ones.

Adobe Sixth Floor At Night and Other Pics

AdobeSixFloorAtNight.jpg
Originally uploaded by Christopher Bailey.

I took some night shots the other night, while working on servers in the data center (had some time to kill while waiting for Linux to install). This one is shot from the six floor walkway and basketball court area. There are some other shots that I did time-lapse that show some of the planes that fly directly over our buildings.