Showing posts with label RubyOnRails. Show all posts
Showing posts with label RubyOnRails. Show all posts

12 May 2009

Passenger Fails to Regenerate Cached/Aggregate Assets for Rails

At DealBase, we've been testing Nginx with Passenger, and have mostly had good results. There are two issues that have come up, hopefully only one of which may broadly affect others.

The first issue, which likely affects anyone using this, is that it appears that if you combine and cache CSS or JavaScript via tags like this in Rails:


<%= stylesheet_link_tag :standard, :cache => 'standard' %>
<%= javascript_include_tag :jquery, :cache => 'jquery_all' %>


The ":standard" and ":jquery" symbols are expansion symbols for multiple CSS/JavaScript files defined in a Rails initializer. On the first request Rails gets, it's supposed to combine all the files as per the expansion symbol definition, and then produce a "cache" file, so you have a single file that is included in your HTML.

This worked fine for us under Mongrel, but it didn't seem to regenerate under Passenger after doing a deploy, even with a restart to Passenger. We had to do a second restart of Passenger, and then hit the server at least twice, if not more to see it get picked up.

Thanks to a tip from Engine Yard, one solution can be found on the overstimulate blog, where they detail how you can add a rake task that you call during deploy to regenerate those cache files. This is really even a nicer solution under Mongrel and others, as it will mean it doesn't occur during your first request.

The second problem seems confined to my MacBook Pro laptop (no problem on my MacPro tower). That is, I simply cannot get Nginx+Passenger to work. It installs fine, Nginx runs, but I get some odd permissions problem from Passenger:


2009/05/12 11:57:52 [alert] 19611#0: could not create /var/folders/7m/7m7ezMSTHdiBHz5bzVyNDE+++TI/-Tmp-//passenger.19596/control_process.pid (13: Permission denied)
2009/05/12 12:01:19 [crit] 19611#0: *1 connect() to unix:/var/folders/7m/7m7ezMSTHdiBHz5bzVyNDE+++TI/-Tmp-//passenger.19596/master/helper_server.sock failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: dealbase.dev, request: "GET / HTTP/1.1", upstream: "unix:/var/folders/7m/7m7ezMSTHdiBHz5bzVyNDE+++TI/-Tmp-//passenger.19596/master/helper_server.sock:", host: "dealbase.dev"


The first line of those two lines occurs when I start Nginx. The second happens when I try to surf to a page from my app in my browser (which makes sense given the first error :) If anyone has any suggestions on this one, let me know!

11 March 2009

Front End Rails Developer Job at DealBase.com

At DealBase, we have an opening for a part-time front end Rails developer at DealBase.com. The opening is for US residents only, and for individuals, no agencies or recruiters please. Most likely you'd be working remotely/telecommuting. The job posting, which is posted in several places, such as Rubynow, Working With Rails, and Rubyjobs.in, covers all the details, but I'll relist it here for ease:




DealBase.com, a startup hotel deals site, is looking for a stellar front end web developer who will adapt our current look/feel to new features, leverage JavaScript for useful and fun features, and is eager to apply their skills to enhance the user experience of our site. We're looking for you to share your knowledge and make an impact, be passionate about your work, and up-to-date on the latest technologies. If this is you, and you enjoy working with a small, distributed, agile team, then we'd love to talk with you.



Requirements for this position:



  • Deep knowledge of XHTML and CSS

  • Familiarity with browser capabilities and restrictions for all major browsers

  • Solid JavaScript skills

  • Experience with/demonstrated use of Git

  • You use and demand MacOS X as your primary development environment

  • Comfortable at the command line

  • Basic skills for image editing and optimization for the web

  • Exposure to and basic knowledge of Ruby on Rails

  • Great communication skills

  • Attention to detail

  • Ability to work both independently and on a team

  • Eagerness to share ideas and problem-solve creatively

  • Experience working on consumer oriented web applications/consumer focus

  • Quick learner, and good at digging in to problems

  • Agile development practices

  • You are based in the US.

  • Individuals only (no multi-person firms, agencies, etc.)


Nice to have:



  • jQuery experience

  • GitHub experience

  • MySQL experience

  • Use of test frameworks, TDD, and BDD

  • experience with Linux



If you'd like to work with us at DealBase.com and think you're a good fit for this position, send us a resume and sample work, or let us know where we can see your resume and work/code, by emailing jobs@dealbase.com. Please note, we are only considering candidates based in the US.




I'm excited to find a great developer to work with. DealBase has been an awesome company and app to work on, and we're already experiencing great success. We have some pretty cool features planned, and it'd be ideal to get some real CSS and JavaScript ninja skills making those features even better. So, if this is you, please do get in touch, making sure to send email to the right email address as outlined in the job description.

02 March 2009

Deploying per-server crontabs with Capistrano

There's been a couple cool writeups/solutions to deploying your crontab files when you deploy with Capistrano, which I think is great. I can't find the first one I saw (mention in comments and I'll update), but on GitHub, javan has the whenever gem that is really more about allowing you to define crontabs with Ruby/Rails' time methods so you don't have to remember the crontab file syntax which none of us ever seems to be able to remember. You can of course integrate this with Capistrano (and that's covered in his Readme). The point of all this: no more having to remember to go put in or uncomment a crontab entry once you deploy a certain build, and keeping your crontabs under version control. However, for us, none of the solutions out there worked quite right, and I just use what I find to be a simpler setup.


First, we have multiple servers with different crontabs per server. Also, we have some environment variables that get defined within the crontab so that they work properly on our Engine Yard slices. I just found that, while yes, I sometimes don't remember all the crontab syntax perfectly, I also don't do this often enough for that to be an issue, and would rather just have the real deal right there, so I knew exactly what I was going to install on my server. Lastly, I didn't really want to have yet another gem dependency for something pretty straight forward like this (IMHO).


So, get on with it you say, what's the solution? Two pieces. First, I create a crontab directory within my Rails app's config directory. In that I store crontab files named by the hostname of the server - the same thing you'd get by doing a hostname on the server. You could add an extension or whatever you want, but the hostname is what makes this work easily, so you want that somewhere in your file naming convention. We only have a couple servers and I know them well, so I just went with pure hostname for now. The contents of each file are exactly what you'd see in the crontab file on the server, for the user you set it up under.


Second, a simple Capistrano task to affect the given crontab file on the server, with an after hook to run it:



task :write_crontab, :roles => :app do
puts "Installing server-specific crontab."
run("cd #{deploy_to}/current/config/crontab; crontab `hostname`")
end
after "deploy:restart", "write_crontab"

That's it. You can obviously tweek this for your own setup, for example, maybe you need to run it on all roles, or different roles, or what not. Your run command might need to be more robust (or run a shell script or rake task) for example if not all servers have crontabs or you have something more dynamic. But, as you can tell, setting this kind of thing up is pretty straight forward, and it's great to keep your crontab setups in version control. Thanks to you guys that stimulated the idea in the first place.


04 February 2009

MySQL Performance Issues and acts_as_versioned

Recently we ran into an interesting performance issue with MySQL. We have an automated process we run at night a few nights a week that does data harvesting for hotel rates and such. This data is versioned so that we can look at historical values. However, this script had begun to really crawl. Originally it took a couple hours to run. But it had gotten to the point where it could take almost a day.



I tracked this down to being a SQL MAX call used by acts_as_versioned to determine the next version for one of these records. The problem is that it had to sift through nearly 10 million records. In testing this on my local machine, just one of these SQL queries could take 45 seconds! Think about doing this across oh say 100,000 hotels, ya, not good.



The good folks at GitHub ran into this same thing (with a table of 36M records) on nearly the same day. Their approach is similar to the approach I'll be taking on another table (which isn't currently affecting us this way, but will have different benefits), which was to split it into two tables, one with older data. I could have done this, and would have, but the reality was that we simply didn't need to keep these versions, as we weren't using the data. So, luckily, I was able to just no longer version this particular model, and throw out that table. After doing that, I ran the script and it took just over an hour. Yea!



So, this is something to note if you use acts_as_versioned with models that have frequent changes and a decent number of those models to begin with (think multipliers). One of the things I'll be looking into in the future is whether that MAX needs to get done, or whether acts_as_versioned can be smarter about how it does it. On first glance you'd think you could just use the version number on the original model itself, but that number isn't guaranteed to be the latest number, since you can rollback versions and so on.



19 November 2008

Using View Helpers and Controller Actions from Rails' script/runner

Recently I coded up a controller and view that produces a large data file. It was done this way because it needs to generate the proper URL's and take into account a fair bit of stuff at the view level in our application (like pagination, and the custom URL's we have, so extensive use of our URL helpers and other things at this level). The action takes a long time to run (about 3 minutes), so of course I page cache it. However, 3 minutes breaks the web server and/or Mongrel timeouts in our environment, so won't get served up in production and staging environments.

The solution, and I'd love to hear other ways, as I certainly won't claim this is the best way, was to create a small script/runner script that simply executed this route within our app. However, script/runner doesn't normally give you controller and view layer access, plus it doesn't have the context of a web request, so it won't know say the host it's being run against and so on. However, one can leverage an Integration::Session and manually set the host to get that. Thus, the script becomes as simple as:


#!script/runner

require 'action_controller/integration'

session = ActionController::Integration::Session.new
session.host = 'www.yourdomain.com'

session.get_via_redirect '/controller/action'

Note, I'm using get_via_redirect here because the particular action I call does a redirect after it expires the cached page of the action it's redirecting to. If you don't have a redirect going on, then you can just call get instead. This does not output anything of course, but for us, just caches the resulting page, which is exactly what we want.

14 November 2008

DealBase - Check out the new features

Since doing our initial public access to the DealBase.com site, you know, the web site that has the most hotel deals on the web (50x more than anyone else), we've been working hard on improving the experience for users. Today we rolled out the latest major revision to the site, which includes some pretty cool stuff. First, let me list these things out, and then I'll cover a few interesting technical bits that occurred along the way.


  • New home page: yes, a brand new, much nicer home page. This really helps tell you what's great about the site. It also has a new search box that I call the "omniscient search" - a single search field that auto-completes on our deal locations and hotels, which is much nicer than the separate search boxes we had before. This same search box is also used everywhere else on the site.

  • Deal filtering: this was a big one. Seems fairly simple on first blush, but lots of interesting stuff going on in the background. You can now filter deals on any deal listing page, by various criteria. You can combine filters too. So, for example, you can narrow down the deal listing by dates, prices, and hotel ratings, allowing you to, for example, look for 4 star hotel deals valid from April through August of 2009, in a certain price range, for a given city. Very handy. Here, take a look at the deals for Hawaii page as an example.

  • Deal sorting: in addition to the filtering, you can now sort the deal listing a myriad of ways, from most percent savings or most dollar savings to high or low rates, or most recently posted, etc. Combining this with filters allows you to really narrow down what deals are best for you.

  • Various UI/visual improvements. Meagan, our talented designer, has done a lot of work here.

  • Speed improvements. Various database queries and other operations for the site have been sped up, sometimes in small amounts, sometimes in extremely drastic ways.

  • More deals: we should probably have about 10,000 deals on the site by the time you read this. This is very exciting for us, and shows how serious we are. These are all very real deals, no link bait, no BS. These are true deals, checked by our team of editors. This gives us about 50x more deals than any other hotel deals site.

  • Comments! You can now comment on deals. No login required (just like the rest of the site). Comments do get reviewed, and we'll be watching for spam and so forth, but this is a great way to tell other people about a good deal or a hotel you like, etc. Comment box is at the bottom of a deal's page. For example, check out this wild personal fireworks show deal at the Ritz-Carlton in New York.

  • Chrome browser support. DealBase works and looks great in Chrome.


And now, since this is a geek blog anyway, a few technical bits...

  • We're running Rails 2.1.2, which is the latest release of Rails as of this writing. We try to stay up to date regardless, but this was a key release, as it fixed some tricky ActiveRecord named_scope issues when using SQL JOINs. Our filtering and other work requires various JOINs and the fixes here prevented us from having to explicitly hand craft a bunch of queries. Thanks Rails team.

  • My current favorite gem is Ryan Bates' scope-builder. This is just so nice for building up big, conditionally chainged named_scopes. As you can imagine this is heavily used in building up combined filtering and sorting of deals.

  • More jQuery goodness. I continue to love jQuery, and use it extensively. It is used heavily in the filtering features, pulling in some nice slider UI elements, and also using it for the "updating" status and dimming effects when the AJAX filtering operations are running.

  • One issue we ran into with Chrome was using the :cache feature of Rails' javascript_include_tag. If we used this to combine and create a cache file of a bunch of separate JavaScript files, Chrome failed to properly load/parse the resulting JavaScript file. This broke pretty much everything JavaScript wise in Chrome, but the simple fix was to not use :cache to achieve this.

  • As a helpful, and economical testing tool, we've been using CrossBrowserTesting.com to give us VM's of a slew of different OS and browser combinations. I tend to run VMWare Fusion and do a lot that way, but it's also a pain to keep up a bunch of different VM images, or have to fire that up for a quick test, etc. We're also using BrowserCam.

  • Finally, another shout-out to the Hoptoad service/folks. This continues to be an outstanding service for us. It works really really well, and it's free, so to me it is the winner amongst the competitors.

  • We've done a fair number of modifications to our tagging plugin (acts_as_taggable_on_steroids), although they're all particular to our app, so not sure if any will get contributed back. Things like enforcing all our rules about tag naming and so on. The same goes for the will_paginate plugin. But in this case, I'm hoping to contribute these back as soon as I can properly contribute the patches and ensure they'll work in any app.


I'm sure there's more, but that's what I can think of for the moment. It's been a busy couple weeks, and I'm really excited about the state of the site these days. We've been getting some great feedback, and I've had a few friends book deals they've found on the site (one friend saved $800 on a trip!). If you have feedback, don't hesitate to add a topic or question in our Get Satisfaction feedback system. This tells us what things you'd like to see, or any problems you're finding, etc.

04 November 2008

Speed Up and named_scope acts_as_taggable_on_steroids Finds

I use the acts_as_taggable_on_steroids plugin for tagging. I've been happy with it, but recently have been adding a lot of searching, sorting, filtering, etc. functionality to an app, and needed the find by tag functionality to work as a named_scope, so that I can have it within a chain of many named_scope finders.

This turned out to be trivially easy to do (without having to copy the SQL and put that into my named_scope). To add a "tagged_with" named_scope to your model that is already acts_as_taggable, you can just do this:


named_scope :tagged_with, lambda { |tags| YourModel.find_options_for_find_tagged_with(tags) }


I've been doing a lot of benchmarking and performance improvements to our SQL as well, and decided to see if this was any different in performance compared to just doing YourModel.find_tagged_with that acts_as_taggable_on_steroids adds. As it turns out, the named_scope version, which is really identical at the core, is faster, especially if called more than once (per thread/Rails request)! Here's the benchmarks to prove it:


Single query/1 Iteration:
user system total real
named_scope 0.040000 0.000000 0.040000 ( 0.045085)
find_tagged_with 0.020000 0.010000 0.030000 ( 0.108233)

10 Iterations:
user system total real
named_scope 0.030000 0.000000 0.030000 ( 0.040282)
find_tagged_with 0.420000 0.030000 0.450000 ( 2.040245)


I repeated these multiple times and got the same/similar results each time. So, for a single query, it's only about 2x faster, but if you start issuing this same find multiple times per request then I believe it's the Rails query caching that kicks in with named_scope, but apparently not with the generic find with all the options (I'd love to hear some commentary on this from someone who knows the details).

Regardless, using a named_scope is nice because now you can more easily chain a tag find together with other named_scope items.

16 September 2008

Mocked by Default, but Unmocking in Some Cases with RSpec

Uh, ya, another great blog title, but we'll get over it. We use geocoding in our app, and that's a relatively costly operation time wise, especially when you may be doing it hundreds or thousands of times when your test suite runs. I can't stub out the objects that use it in many cases, so I wanted to stub out the actual geocode call unless I truly needed real geocoding (which is only when I'm testing the actual geocoding itself, and thus is a very small part of the test suite).

We use RSpec Specs and Stories and I wanted to mock out the geocoding by default, but unmock it in a few places. I asked about this on the mailing list, Googled and so on, but didn't find a solution that was working. So here is what I wound up doing...

In my spec_helper.rb file, I added:


Spec::Runner.configure do |config|
config.before(:each) do
# Setup fake geocoding unless told not to
unless @do_not_mock_geocoding
fake_geocode = OpenStruct.new(:lat => 123.456, :lng => 123.456, :success => true)
GeoKit::Geocoders::MultiGeocoder.stub!(:geocode).and_return(fake_geocode)
end
end
end


What this does is mock the geocoding unless a test has set the @do_not_mock_geocoding variable to true. One caveat, at least from what I've found, is that you need to set that to true in a before(:all) block in your tests, so that it happens before the before(:each). This is minor, as you can just have something like:

describe "with real geocoding" do
before(:all) do
@do_not_mock_geocoding = true
end

# your tests that want real geocoding
end


The impact this has had on our test suite is tremendous. I had already had some partial mocking of the geocoding in place, but was sweeping the system to put it in because the time it took to run our test suite was out of hand at about 13 minutes! Now that I've got this in, it runs in 2 minutes! Geocoding is used in two of our most core objects, which is why it has such a big impact on the test suite. This is one place mocking has really proved to be a massive value!

02 September 2008

We're Hiring for a Rails or Similar Developer

The startup I work for, DealBase, is in need of another developer. This is pretty exciting for us, and the business is quite exciting as well! We're looking for someone with Rails experience, as well as MySQL, Git, TDD (BDD is good too), agile development practices, and so on. You can read all the details in our posting on the 37signals Gig Board. Please make sure you respond to the ad, as opposed to sending me email (or asking via a blog comment). You can mention in your email though that you found it via my blog entry.

One thing I want to specifically note, you don't have to be a "Rock Star"! Sure, we want the best people, but I too am sick of this "Rock Star" designation. I'd like to see someone who's passionate about software development, web apps, technology, TDD, JavaScript, and so on.

Another thing to note, we're a partially distributed team. I myself live in Eugene, Oregon, others are in the Bay area, one person Boston.

11 July 2008

Fork of acts_as_versioned to provide version diffs and more

Update: Added the earliest? and latest? methods, see below.

On my current project, I've recently begun using Technoweenie's acts_as_versioned (I also looked at simply_versioned which also looks great - you'll need to evaluate for your own needs). This project has some particular needs around versions that aren't covered by the existing plugin, so of course I forked it on GitHub and have been adding my enhancements to my fork. These are publicly available.

Right now there are two enhancements that may be of interest to others:


  1. For each version, there is now an updated_attributes field that stores an Array of the attribute names that were changed in the creation of this version. This is essentially the same array provided by the changed method from ActiveRecord (requires 2.0 (or 2.1?) or later for the "dirty" handling stuff). This provides a nice way of being able to show what changed between versions, without having to compute that yourself (as well as compute it each time you need to display it). Since the data is right there when making the version, I just store it off in a serialized column. This will not record "non-versioned columns". This obviously requires another column in your DB table, and I've amended the migration method, but if you already are using acts_as_versioned, you'd need to manually add the updated_attributes column (of type "text").

  2. A small tweak, but the revert_to! method now has an optional second parameter that, if set to true, will remove all newer versions. In our workflow, when you do a revert/rollback, you no longer want the newer versions, so we delete them. You can achieve this in your app without deleting them, but it just seems cleaner in our case. You can also do this manually by calling the same method that revert_to! uses, delete_newer_versions.

  3. Added the earliest? and latest? methods to the model's Version object as well, so that you can call these on an individual version instance - very convenient for me at least.



I have only tested this under Rails 2.1 and MySQL and SQLite (all the unit tests for acts_as_versioned use SQLite). I have not sent Rick a pull request, because a) I've only been using this for a few days, and I'd like to have it in use a bit longer to be sure of the approach and quality. For example, to really be ideal for the mainline distro, I suspect that adding an option to acts_as_versioned when you define it in your model, would be best for the updated_attributes storage aspect, e.g. making this part optional. It's never optional for my needs, so I haven't spent the time to do this. If you use my fork and do that, please send me a pull request.

The next piece I'm looking at is supporting versioning of associated models. This will be a bit more involved. There is acts_as_versioned_association, but it says it is not maintained and doesn't work with Rails 2.x. However, I may be able to fork that and bring it up to speed for the latest Rails and acts_as_versioned...

01 July 2008

Update on Rails, jQuery, autocomplete

Today I changed my auto_complete_jquery plugin (which I blogged about previously) to work with a different jQuery autocomplete plugin (that's not confusing is it?!). Previously, I'd been using the "jquery-autocomplete" plugin, but had been having problems with it always being case sensitive, and with it being pretty darn slow. To solve that, I wound up switching to Dylan Verheul's jquery autocomplete plugin, which is fantastic! So, I had to update my plugin (literally changed two lines, along with the readme and some comments).

So, what does it all mean? First, things are no longer case-sensitive (although you can tweak that if you need it, see the docs for DV's jquery plugin). Second, the speed is near instant, and if it's taking any time at all, there's now a CSS style you can set to show an indicator while the AJAX call is running (nice!). Further, there are a slew of options you can set, but one of the coolest things is the ability to set a formatter JavaScript function to adjust the display of the returned results, but without affecting the actual value that is placed in your text field. This is really cool for providing further information about a given matched item. For example, I use it to display, on a line below the matched name, the location of the item. There's an interactive example of this on Dylan Verheul's site.

I highly recommend updating all around on this. For those that might be using my prior version, the changes you'd need to make are:


  • Change your JavaScript calls that make a HTML input into an auto-complete from this format:

    $("input#post_title").autocomplete({ ajax: "auto_complete_for_post_title" })

    to this:

    $("input#post_title").autocomplete("auto_complete_for_post_title")


  • Update to the HEAD of my plugin.

  • Remove the old jquery.ui.autocomplete*.js files, and install Dylan's single jquery.autocomplete.js file, updating your JS includes accordingly. Same goes for the stylesheet.



Note, to do the cool 2nd line output of auto-complete items, you will need to write your own auto-complete Rails action, which means you don't need my plugin :) I may update my plugin at some point so you can pass a block in to the autocomplete function to create this same scenario, but this is pretty specific stuff, so we'll see. As a very brief set of instructions to do this, you can essentially copy-paste the auto-complete method that gets defined by my plugin, and then update what you return as text. To do the 2nd line bit, you want to return items in the format, "item|2nd line stuff", (so use the pipe symbol to separate the two lines). Then, you can use a simple JavaScript formatter function like:

function formatAutocompleteItem(row) {
return row[0] + "<br><i>" + row[1] + "</i>";
}


Then, update your call to the jQuery autocomplete that sets up a field as autocomplete to be:

$("input#post_title").autocomplete("auto_complete_for_post_title", { formatItem:formatAutocompleteItem })


Enjoy!

17 June 2008

RSpec View Testing Problems

I've been using RSpec exclusively on my latest project. I'd say I'm still fairly new to it, but it has won me over for the time being. However, the view testing has been a real problem. You can test views quite easily and nicely with RSpec, however, when you do something subtly wrong, it can cause RSpec to fail without any visible error! In fact, Autotest doesn't report a failure at all, and the only way I know it is failing is either I pay close attention to the number of examples it said it ran, or more likely, my cruisecontrol.rb CI server will actually fail (because it detects the commands exit status).

For example, while Autotest says everything is fine, the CI server will show something like this (note for easier reading I've trimmed this a bit):


/widgets/index.html.erb
- should render list of widgets

/bobbles/index.html.erb

Finished in 4.977586 seconds

175 examples, 0 failures
rake aborted!
Command /usr/bin/ruby1.8 -I"/var/cruisecontrolrb/projects/myproject/work/vendor/plugins/rspec/lib" "/var/cruisecontrolrb/projects/myproject/work/vendor/plugins/rspec/bin/spec" "spec/controllers/bobbles_controller_spec.rb" ... --options /var/cruisecontrolrb/projects/myproject/work/spec/cruisecontrol_rcov.opts failed

(See full trace by running task with --trace)


As you can see, RSpec is reporting 0 failures, yet rake fails. This is because in reality RSpec is returning an exit code of 1 instead of 0. But, looking at the output, it's certainly not revealing. Running with --trace is of no help either.

What I now know to pay attention to in situations like this is the fact that that last spec it ran, the /bobbles/index.html.erb one has no examples listed under it. That is thus the culprit (you can also argue that it's the last thing to "run" and then rake fails, so it's likely in this, etc.).

The real pain comes when you try to figure out what the heck is causing this. You have zero feedback, and no way that I know of to somehow debug or inspect the test to see what's failing. In my experience to date with these, it boils down to some problem in your mocks and stubs, but this can be difficult to figure out. I admit, the one that prompted me to write this blog entry is one I've still yet to figure out, and finally just punted on.

I've been hearing a lot about not testing views, or testing very little of the views. I agree in general on this, and am now looking into using Webrat to do integration testing to really test "view" functionality, and leave the rest of my view testing mostly to testing my helpers, controllers, and models. Here are a couple of blog entries related to all this:

11 June 2008

Changelogs and Deployment Notification for Capistrano and Git

Early warning: this is a hack, which doesn't mean it's bad, just that it's not polished. However, I am documenting my solution for myself thus far, as well as figured others might find it useful...

Update: Added my shell command for doing deploys (see end of this post).

I wanted a way to automate a few things around deployments, and integrate this a bit with my continuous integration server. I use CruiseControl for the CI server, and previously blogged about setting up CC.rb with Git. The goals for this next task, and subject of this blog post are:


  • Tag the code on successful deploys. My CI server already tags the code anytime it does a successful build, but since I didn't cover that previously, I'll mention it here as well.

  • Notify a list of people via email whenever a new deploy happens.

  • Generate a changelog, based on Git commit messages (better make sure they're suitable reading for whoever gets your deploy notices!), and include this changelog in the deploy emails.

  • Have the CI tag I want to deploy as the only required piece of info/parameter when issuing a deploy command.



Tagging


First, I tag the code on any successful CI run. This tag is what I can then use as the Git tag to deploy. Capistrano supports this via the branch variable (set its value to the tag name). As you can guess, you can use pretty much any Git ID/tag/branch name for this. To do this, add a task to your cruise.rake file (or similar - wherever you define your custom CruiseControl command), and then ensure you run that task during a CruiseControl session. Here's my task:

desc "Tag the code on successful CI build"
task :ci_tag do
timestamp = Time.now.strftime("%Y%m%d%H%M%S")
tag_name = "CI_#{timestamp}"
# Create an empty file with our tag name, so we can easily go grab the tagname
# from the CI output page and do deploys, etc.
system("touch #{File.join(ENV['CC_BUILD_ARTIFACTS'], tag_name)}")
system("git tag -a -m 'Successful continuous integration build on #{timestamp}' #{tag_name}")
system("git push --tags")
end


From the above, you can see that I'll get tags of the form: CI_timestamp. Next up, I want to tag a successful deploy to indicate which commit/tag actually got deployed and when. This is handled via an after task in my Capistrano deploy.rb:

after "deploy:restart", "tag_last_deploy"
task :tag_last_deploy do
set :timestamp, Time.now
set :tag_name, "deployed_to_#{rails_env}_#{timestamp.to_i}"
`git tag -a -m "Tagging deploy to #{rails_env} at #{timestamp}" #{tag_name} #{branch}`
`git push --tags`
puts "Tagged release with #{tag_name}."
end


This will create tags like, deployed_to_staging_1213223458, and works for both staging and production (or any environment you're targeting - note the use of the rails_env variable - you may need to use something else). One thing to pay particular attention to, is that this tag is actually tagging another tag, as defined by the branch variable (mentioned above). In order for this to work though, you need to ensure that your tags are up to date locally. Thus, somewhere in your workflow you'll need to do a git pull --tags, if like me, your CI server is elsewhere and is generating those tags.

Ok, we're all tagged up, let's move on...

Notification



It turns out there's a nifty new plugin called Cap Gun that will take care of emailing a list of folks on deploy. Setup is covered in their README, but the one bit they don't mention, is that you can include a comment in the email message that goes out. I wanted to include a changelog in these emails, so I tapped into this comment attribute, setting it to the text of my changelog. To use the comment, you can either set it via -s comment="my lovely comment" on your Capistrano deploy command, or you can set the comment variable in your Capistrano deploy.rb or included script. More on that in a minute.

Changelogs


My changelog, so far, is very simple, it just pulls the comments for the Git commits that occurred since the last deploy (for the appropriate target), up to the tag specified (which in this case will be the CI tag you are about to deploy). To handle this, I use a small Ruby script, combined with the great Grit gem that lets one manipulate Git via a nice Ruby API. The script simply spits out a simple chunk of text that will be what gets put into the comment Capistrano variable for our deployment notifications. This is in particular where the "hack" comes into play. This script is not robust, does essentially no error checking, etc, etc. Use at your own risk! And with that, here it is:

#!/usr/bin/env ruby

require 'rubygems'
require 'mojombo-grit'
include Grit

unless ARGV.length == 2
puts "Usage: changelog.rb staging|production <commit-or-tag>"
puts " where commit-or-tag is the commit ID or tag you are planning to deploy"
exit -1
end

repo_location = File.expand_path(File.dirname(__FILE__) + '/..')
target = ARGV[0]
about_to_deploy_commit = ARGV[1]
repo = Repo.new(repo_location)

# Find the tag for the last deployed
tags = repo.tags.collect {|tag| tag.name }
tags.delete_if {|tag| !(tag =~ /^deployed_to_#{target}_/)}
tags.sort!
last_deployed_tag = tags[-1]

commits_for_changelog = repo.commits_between(last_deployed_tag, about_to_deploy_commit)
commits_for_changelog.reverse!

puts "Changes since last release:"
commits_for_changelog.each do |commit|
puts " "
puts " #{commit.message}"
end


To run through it briefly, it takes two parameters (and clearly, you can change this for your own deployment targets, etc.): a deployment target, and a tag (which can actually be a tag, a commit ID, branch, etc.). It sets up a repo variable for your Git repository using Grit, and then proceeds to find the last deployed tag for that deployment target. After that, it gets all the commits between that last deployed tag and the tag you specified as the second script argument, and prints out the commit messages.

To integrate this, I added this line to my Capistrano deploy.rb:

set :comment, `script/changelog.rb staging #{branch}`

As you can see, that one is specific to my staging environment, and lives inside my "staging" task in deploy.rb. Same, appropriately edited version goes for production.

Deployment Command


Lastly, I define a simple shell function to do my deploys, which ensures I have done a git pull so I have all the tags, and makes the command easier to remember and get right, etc:

stagemyproject () {
git pull
cap -s branch=$1 staging deploy:migrations
}


You would thus have a command line to do a deploy like this:

stagemyproject CI_20080612052417

That's it, and if you've managed to read this far, congrats, and if you've not only managed to read this far, but payed attention and got value out of it, well, cool.

For anyone who uses/adapts this, please do let me know improvements you make, or suggestions, or tweaks/changes, and so on. I've been using this for all of about a half dozen deploys so far. If (more like when) I make improvements, I'll update.

04 June 2008

Rails, jQuery, auto-complete, and a New Plugin

Update: I've switched which jQuery autocomplete plugin I use for this, see my newer blog entry.

The other day, I made a whole switch from Prototype & Scriptaculous to jQuery. I've had the bug to do this for a while, and this is a new project, so I went for it. I don't have anything against Prototype, so my main impetus for this was a move towards Unobtrusive JavaScript, and also the speed aspect (the site I'm currently working on, if things go accordingly to plan, will do some pretty serious traffic). But, the unobtrusive JavaScript was the key, and really, my switch is more of a philosophy of approach rather than say a dislike for Prototype, etc. And, of course, it's something new to play with :)

Before I go any further, I'll state right now, I am not a JavaScript expert, and I've been using jQuery now for all of a couple hours.

One of the results of my switch however, was that I hacked DHH's auto_complete Rails plugin, to work for jQuery. Simple change. I tweaked the controller macro, and then gutted the JS helpers, as you just don't need those when using jQuery in this way. It does require the jquery-autocomplete plugin for jQuery. I've published my Rails plugin for this on GitHub as auto_complete_jquery.

Circling back around, here's what I did to get all this going. I did run into one issue (see step 10 below) that I'm still tracking down (easy solution in the interim, but I'd like to understand what's happening, so if you have comments, please let me know):


  1. Removed the Prototype and Scriptaculous JS files from the public/javascripts dir of my Rails app. You don't have to do this, but I am no longer using them, so saw no need to keep them there, and it helps ensure I don't mistakenly use something from them or include them in the view. This includes: prototype.js, controls.js, dragdrop.js, and effects.js.

  2. Removed the prototype-based Rails auto_complete plugin from vendor/plugins.

  3. Installed the latest minified jQuery file in public/javascripts/jquery.

  4. Installed the JS files for the jquery-autocomplete plugin, and its dependencies: jquery.templating.js, and jquery.ui.autocomplete.js. (see the jquery-autocomplete plugin for these files).

  5. Added the jquery.ui.autocomplete.css file to public/stylesheets.

  6. Installed my auto_complete_jquery plugin.

  7. Put the proper includes for the CSS file and the JS files in my application layout file:


    <%= stylesheet_link_tag 'jquery.ui.autocomplete' %>
    <%= javascript_include_tag 'jquery/jquery.min', 'jquery/jquery.templating', 'jquery/jquery.ui.autocomplete.ext', 'jquery/jquery.ui.autocomplete', :cache => 'jquery' %>

    Note that I keep my jQuery JS files in a subdir for organizational purposes, but you can modify as needed.

  8. Changed my existing auto-complete text fields that used the Rails Prototype based auto_complete plugin's helpers to just be plain old text fields, such as:
    <%= coffee.text_field :drink, :autocomplete =>"off" %>
    This is doing an auto-complete for the "drink" attribute of the Coffee model.

  9. I can simply leave any auto_complete_for calls that existed in my controller, as that works the same. If you had custom versions that were based on the code from the Prototype-based Rails plugin, just go look at the code in my plugin to see the differences, it's a simple change.

  10. Add the JavaScript that sets up the auto-complete for the given text field. This will typically look like:

    $(document).ready(function() {
    $("input#coffee_drink").autocomplete({ ajax: "auto_complete_for_coffee_drink" })
    });

    Where does this go? It depends. What I've been liking is using the JavaScript auto-include plugin, which creates a Rails-style convention for JavaScript files that pertain to individual actions, or are controller-wide. So in my case, this code would get placed in public/javascripts/views/coffees/new.js, or likely one directory up, as simply coffees.js (so that I can use it in any CoffeesController action that needs to auto-complete on coffee.drink. Without that plugin, you just put it in whatever JS file is appropriately included for the view you're using it in, etc. You can of course put it directly into the view in a script block, but then you aren't doing the whole Unobtrusive JavaScript thing as rigidly.
  11. Finally, what I found is that I had to add a route for this. This is the issue I mentioned above. It sort of makes sense, but what I'm unclear on is, why the prior standard/Prototype-based Rails auto_complete plugin didn't require a route. They both seem to use a GET, define the action the same way, and so on. I'm hoping I'm just missing something obvious. So, the route I added is:
    map.connect ':controller/auto_complete_for_coffee_drink', :action => 'auto_complete_for_coffee_drink', :format => 'json'



A bunch of steps, but pretty simple work. The app I'm doing this on is all of a few days old, so I hadn't gotten into use of much else in Prototype and so on, thus making the wholesale switch easy.

If you'd like to learn more about any of these things, and as a comprehensive set of links:


Enjoy!

Update: I removed the jQuery Dimensions JS file and include for it in my layout, as this is now included in the latest jQuery JS file itself.

Update 2: I don't know how the standard auto_complete plugin manages to do without routes, but here is a generic route for all auto-complete actions across controllers:

map.auto_complete ':controller/:action',
:requirements => { :action => /auto_complete_for_\S+/ },
:conditions => { :method => :get }

I hesitate to put this into the plugin, as routes can be quite tricky in more complex apps, and I wouldn't want to auto-hose someone :)

02 June 2008

Fixing Capistrano 2.3.0 and Git Deploy Problem

If you upgrade to Capistrano 2.3.0, and are doing deploys from a Git repository, you may find that all of a sudden you can no longer deploy. This is the case if you have no tags in your Git repo. Cap 2.3.0 changed one of the Git commands it uses and that apparently doesn't work right if you don't have tags. So, to solve the problem, you can simply create a single tag in your Git repository. The tag does not have to relate to your build at all, you only need one tag in the repo (not one per build or anything like that), etc. Once you create the tag, you can now deploy again.

To create a tag in Git, or, I think the "cooler" kind of tag, an annotated tag, you can do:

git tag -a tag_name

Replace "tag_name" with your tag name of course. The "-a" option says to make it an annotated tag, which lets you enter a comment about the tag. You can put whatever you want in there. I'm liking this potential use with my continuous integration server when it makes tags on successful builds. Lots of possibilities.

Finally, if you deploy from a remote repo, or if you have a remote repo (say on GitHub), you will need to push your tag. This does not automatically occur on a push, you need to add "--tags" option to git-push to include your tags:

git push --tags

Now you'll have your tags on your remote repo, and listed under the "all tags" tab on GitHub.

05 May 2008

Announcing ShipFu!

Have you ever needed to quickly compare UPS and USPS shipping rates, and just wanted a simple web page/app to do it? If so, check out my new ShipFu app. It's incredibly simple. No gratuitous web decoration, no hidden motivation behind it to promote some other product or shipping service, etc., just a simple calculator. Enter your package origin and destination, and dimensions and weight, and voila, you'll be presented with all the rates available for that from the two services.

I built ShipFu to address a need of my own, and the fact that I just didn't seem to like any other solutions out there - seemed to be too many hidden agendas or ungainly UI's, etc. Also, I built this as a bit of an experiment with a few technology aspects. It's a Rails app, and is running on the new-ish Thin server, etc. I plan to have a blog post that goes into a lot more depth on some of the fun things that occurred as part of building this super simple app.

Also, please, by all means, send me feedback (use the feedback link shown near the bottom of the site) if you'd like to see some feature or enhancement or what not. Or, just use it to help you quickly gauge shipping rates.

13 March 2008

Seed Data for Your Rails 2 Apps - Another Approach

Historically, I've used migrations to set up standard data that my database must contain in a Rails app. This would be things like standard Roles for the system, or maybe country codes or such things. However, it appears this simply won't work in Rails 2.x, because as far as I can tell, when you run something like "rake test", it blows away ALL data in your database (not just fixture data). If I'm wrong about that, please correct me. This makes sense given that it seems the drive is towards schema.rb being the official way to create a DB from scratch, and that you have the equivalent of Foxy fixtures which do lots of magic to make creating your fixtures easy (but likely quite painful to figure out how to explicitly clean up those fixtures in certain cases - so it's easier just to wipe the DB clean).

There are various solutions for creating seed/standard/structured data for your app. However, from what I've seen none address this problem that that data will get wiped out when testing. For many people that may not matter, their tests may not hinge on it. But, I like to stay DRY, and when you have standard roles, or similar types of data, there is no reason I should have to recreate those in fixtures (and risk being out of sync), or leave them out, etc. I likely have app functionality that directly depends on such things, and thus I need this during testing as well.

My solution as of now is a simple one, and one that does not scale well for large amounts of data, but for the five records I need at this point in the particular app I'm working on, it's an approach (I very much welcome better approaches!)... I simply created a "seed_data.rb" file in my config/initializers directory. Within this file I have code that does a create_or_update (or similar) of the standard data I need. This seems to work out quite well.

Update: the above breaks things like "rake db:reset", because when the initializers run, as part of the Rake environment, and the DB has been dropped, the initializer fails, and thus fails rake.

06 March 2008

Rails Applications and Gems: Solving the Dependency Problem

There's a post today on the Relevance blog about Frozen Gems Generator. I tried posting a comment there, but it seems to have not gone through, so I'll blog my solution here instead.

Chad Woolley at Pivotal Labs created GemInstaller to solve the problem of specifying exactly what gems you want your Rails app (or other Ruby code) to use. I've dealt with this issue a lot over the course of building Rails apps, and while at first blush I didn't think this was a good solution, I'm now really like it, and use it on most of my projects (basically all the projects I control or can :)

So, why is it better than other solutions, or at least the other solutions I've seen? First, let me give a quick synopsis: it is a simple gem that allows you to create a geminstaller.yml file that specifies the version(s) of gems your app wants. This can be an explicit version, or can use things like >= version, etc. It can then automatically install the gems for you on app launch, on deploy, or just at the command line. The benefits of this solution for a Rails app include:


  • Solves the arhictecture/platform-specific gems problem. I haven't seen any of the other solutions do this, or do it well. Most just punt on it, others require a convoluted process or hacking up your other code. Because geminstaller simply relies on the gems being installed on your system, it will use the proper version for whatever system it is running on. This also ties into the next point...

  • No polluting your source control with gems. This speeds up your source control, as well as your deploys. Further, for architecture specific gems, you now don't have to have every version of each gem in your source control for each platform you need (which is quite likely at least two: your dev boxes (e.g. Macs) and your deployment boxes (Linux), but could be even more).

  • Easier, single location, statement of what gems your app requires. By using the geminstaller.yml file, you have a single place to go see what gems and which versions of those gems your app uses. This is much better than trying to look through your vendor directory, and determine what version of a gem you might have.

  • Great for bootstrapping your development environment. Sure, frozen gems usually solve this too (except for the architecture specific ones!). You can just run geminstaller after pulling down the code and it'll go install all the specific version gems you need.

  • Allows for multiple config files, so that you can build common ones you use across projects, etc. Or even cooler, your plugins or whatever can provide a file to specify what they need and you can integrate that into your config!

  • Easy to install and use. In Rails 2 environments, you can simply drop the few lines needed to use it into its own file in config/initializers. In Rails 1.x, you add these lines to your environment.rb.

  • You determine what level of function you want geminstaller doing in your app: e.g. do you want it automatically installing missing gems or just warning you? Should it put them on the load path so you are guaranteed the proper version loads, or do you want to just use it to bootstrap and live dangerously otherwise ;-)

  • Makes it easier to experiment with new versions of gems. Since you'll have to install the gem anyway (or most solutions need that to freeze them in, but not all), you can experiment by simply changing the version number in your geminstaller.yml file. To undo it, just change the number back. No need to copy the gem into vendor or a private gem repo, etc. Easy.

  • GemInstaller can tell you what gems you have on your system, but are not in your config file, as a way to see what you might need.



Check out the GemInstaller page for more details. I highly recommend this, and thanks Chad for creating it.

24 September 2007

Installing Ruby MySQL Gem with MacPorts MySQL

Blogging this more for my own record, but maybe others will find it useful... Tonight I was having a hard time getting the MySQL Ruby Gem installed on a new MacBook Pro. I have installed Ruby, Rails, RubyGems, MySQL, etc. via MacPorts (or via the Ruby that was installed via MacPorts). Anyway, this is the command that finally got it to work:

sudo gem install mysql -- --with-mysql-include=/opt/local/include/mysql5 --with-mysql-lib=/opt/local/lib/mysql5 --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config


Update: as I mention in my comment below (updating here in case folks don't read the comments), when doing this on Leopard/MacOS X 10.5, I needed to change it to:

sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-include=/opt/local/include/mysql5 --with-mysql-lib=/opt/local/lib/mysql5 --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

15 June 2007

Fixes for Capistrano 2's Perforce

I'm starting to convert to Capistrano 2. We use Perforce, and I've found a simple typo/bug in Capistrano 1.99.1. The fix is simple:On line 57 of recipes/deploy/scm/perforce.rb, change the use of "revno" to "rev_no".


Now, the question is, where is the Capistrano bug DB, or where/how do I submit this to them?


I reported this to the mailing list, and Jamis has already checked in the fix in SVN. Capistrano lives in the Rails Trac.