blog.josh-nesbitt.net

Simple database backups using a nightly CRON job and Capistrano

2010-05-06T00:00:00+01:00

Limerick Rake by Thoughbot provides a useful set of Rake tasks to help simplify common tasks. One that I use quite a lot is the backup task which performs a mysqldump of the specified environments’ database. It looks roughly like this:

# http://github.com/thoughtbot/limerick_rake/blob/master/tasks/backup.rake
# lib/tasks/backup.rake
namespace :application do
  desc "Backs the applications database up for the given environment"
  task :backup => :environment do
    
    config           = ActiveRecord::Base.configurations[RAILS_ENV || 'development']
    database         = config['database']
    filename         = "#{database.gsub(/_/, '-')}-#{Time.now.strftime('%Y-%m-%d-%H%M%S')}.sql"
    backup_directory = File.expand_path(File.join(RAILS_ROOT, '..', '..', "shared", "backups"))
    backup_path      = File.join(backup_directory, filename)
    binary           = %x[ which mysqldump ].strip
    
    options   =  "-e -u #{config['username']}"
    options   += " -p#{config['password']}" if config['password']
    options   += " -h #{config['host']}"    if config['host']
    
    raise RuntimeError, "I only work with mysql." unless config['adapter'] == 'mysql'
    raise RuntimeError, "Cannot find mysqldump." if binary.blank?
    
    FileUtils.mkdir_p(backup_directory)
    
    puts "Backing up #{database}"
    %x[ #{binary} #{options} #{database} > #{backup_path} ]
    puts "Backed up #{database} => #{backup_path}"
    
  end
end

There’s a couple of ways I like to use this. Firstly, to ensure that the database is cloned before a deploy using Capistrano. By hooking it into a before callback I can ensure the database is backed up before any code is updated:

# config/deploy.rb
namespace :deploy do
  before "deploy:update_code", "deploy:backup"

  desc "Backs up the applications database to RAILS_ROOT/../shared/backups"
  task :backup do
    run "cd #{deploy_to}current && rake application:backup RAILS_ENV=#{rails_env}"
  end
end

This does the job for small to medium sized sites, larger ones would have much more rigourous backup procedures. Another way is to use it on a nightly CRON job, which might be more useful for preserving day-to-day snapshots of your database. In your crontab:

0 1 * * * cd /path/to/project && RAILS_ENV=production /usr/bin/env rake application:backup >> log/backup.log 2>&1

This way you cover your back between deploys and keep daily copies of the database should the worse ever happen.

iPhone 4.0 OS and developer choice

2010-04-12T00:00:00+01:00

There is currently a massive hoo haa within the iPhone and iPad developer community surrounding Apple’s recent disclosure of requiring iP(hone/ad/od) applications (distributed on the app store) to be developed in Objective-C, C, C++, or JavaScript (as executed by the iPhone OS WebKit engine). In particular this implicitly highlights the barred usage of Adobe Flash to Objective C language compilers.

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

There have been many takes on the matter, all with valid opinions.

However, one point that I feel people are missing is the ongoing issue with the app store development/release process and static downloadable application binaries. People are worried about these restrictions enforced by Apple crushing their personal choice of language by locking them in to using only a select few languages for the development process. In a way I’m not suprised Apple are doing this, however you look at it Apple are creating a closed environment for them to capitalise as best as they can on the dominance of their very popular platform. They want to do this in a way that applications distributed on the platform are unique to that platform, a niche application on an iPad is another reason to get an iPad.

For quite a few applications the web (as a platform) would suffice. A lot of apps don’t need access to these private API’s, multi tasking abilities or {insert other feature here} as a reason to choose the native application development path. I for one see this as a ongoing point that developers are free to harness the web as a platform for these devices, where there is ~~little~~ no chance of being locked into a corner. When the dust has cleared I’m sure we’ll all move on and find a way to make these platforms work in our favour, whatever route we take.

Introducing Configr, an elegant configuration interface in Ruby

2010-03-01T00:00:00+00:00

I’ve never really been happy with my previous solution to making configuration easy and elegant. It still felt too clunky and lacked the syntactical sugar that I’ve become accustomed to within Ruby. So configr was born.

Configr aims to provide a clean interface for configuring and reading a set of configuration values. The idea evolved from using a standard hash as a configuration store into a more elegant way to declare and read values from within a hash.

I wanted a way to be able to declare my configurations in a simple format, whether that be inline (in the format of a block), from a YAML file or as a mixture of the two. I also wanted to be able to have a simple way of overriding values for certain environments (for frameworks such as Sinatra and Rails). The project is far from done but it’s the start of a more elegant solution to these problems over my previous method (simple hash based configuration).

Using Configr is simple:

  configuration = Configr::Configuration.configure do |config|
    config.example_one = "One"
    config.example_two = "Two"
  end

  puts configuration.example_one
  puts configuration.example_two

Declaring values inline is great, but its also useful to pull them in from a YAML file:

  configuration = Configr::Configuration.configure("/path/to/file.yml")

  puts configuration.value_one
  puts configuration.value_two

Or as a YAML string:

  yaml = <<YAML

  example_one: "Hello"
  example_two: "It loads from YAML too!"
  
  YAML
  
  configuration = Configr::Configuration.configure(yaml)

  puts configuration.example_one
  puts configuration.example_two

Or a mixture of both methods:

  yaml = <<YAML

  example_three: "three"
  example_four: "four"
  
  YAML
  
  configuration = Configr::Configuration.configure(yaml) do |config|
    config.example_one = "one"
    config.example_two = "two"
  end
  
  puts configuration.example_one
  puts configuration.example_two
  puts configuration.example_three
  puts configuration.example_four

The idea is for Configr to be quite flexible in how you declare your configurations. The end result being able to read them in a more natural format:

  puts configuration.one.two.three

Sometimes when using configuration values you might want to assert whether the value exists before using it. Configr allows you to do this with the #key_name? syntax:

  puts configuration.one.two.three? # => true

Configr is intended to be framework agnostic, but it’s easy to get it going inside of any of the latest frameworks. I don’t plan to provide any “out of the box” way to hook Configr into Rails but it’s easy to implement:

  # In config/configuration.rb
  yaml_file = Rails.root.join("config", "environments", "#{Rails.env}.yml")
  
  Configuration = Configr::Configuration.configure(yaml_file) do |config|
    config.my.configuration.value = "value"
  end
  
  # In config/initializers/configuration.rb
  require Rails.root.join("config", "configuration.rb")
  
  # Anywhere in your Rails app
  Configuration.my.configuration.value

By requiring a different YAML file based on the environment it is easy to override global values with environment specific ones. For more examples of using Configr with other frameworks see the readme.

By design a configuration value is not meant to be edited or created after the configuration block has been run (if you do you will run into a Configr::ConfigurationLocked error). In my opinion configurations such as those created by Configr are meant to be read-only during the lifetime of the application.

The gem is now available on rubygems.org:

  gem install configr

It’s still early days for the project but I’m already enjoying using it in a few personal and client projects, for me it certainly beats the previous method. If you use it and find any bugs please file an issue to help me wean out any problems.

Lockfile now pushed to Gemcutter

2010-02-16T00:00:00+00:00

A while ago I created a simple lockfile implementation in Ruby. Just recently I received a pull request from Greg Fitzgerald who converted the project into a gem.

Wanting to have a quick play with Gemcutter I decided to build and push the gem for others to use. I have to say, Gemcutter is slick. I had the gem pushed and available within a minute. Now its just a simple case of:

  gem install simple_lockfile

I also took the time to add some quick specs and a bit more information to the readme (where you can find usage examples). I find myself using it here and there for the odd task, so let’s hope someone else finds it useful too.

Writing contingent Ruby code with #retryable

2010-02-08T00:00:00+00:00

Sometimes when writing a pivotal piece of code that’s prone to raising errors, you want to be able to rescue and recover easily. I needed just that when I was programming some code that needed to be reliable when communicating through sockets.

I found this really useful post on creating a retryable block of code. I decided to better it by adding the ability to use multiple exception classes. Here’s a really nice way of retrying a chunk of code n times:

  def retryable(options={}, &block)
    attempts          = options[:attempts] || 1
    exception_classes = [*options[:on] || StandardError]
  
    begin
      return yield
    rescue *exception_classes
      retry if (attempts -= 1) > 0
    end
  
    yield
  end

Use it like this:

  retryable(:on => [MyExceptionClass, StandardError], :attempts => 5) do
    # .. my risky code ..
  end

This is great but it didn’t feel as elegant as it could have been. I then stumbled across this post which presented the style I was looking for. Refactored, the above snippet now looks like this:

  class Integer
    def tries(options={}, &block)
      attempts          = self
      exception_classes = [*options[:on] || StandardError]

      begin
        return yield
      rescue *exception_classes
        retry if (attempts -= 1) > 0
      end

      yield
    end
  end

Which provides a cleaner call to the previous method:

  3.tries :on => NoMethodError do
    # ... my risky code ...
  end

This provided me a really clean way of catching socket timeout errors and service unavailable calls without repeated begin rescue blocks. Of course you should always provide an error class to rescue from as rescuing from StandardError will just catch any old error.

The state of the internet

2010-02-06T00:00:00+00:00

Mashable recently published an article about current usage statistics on the internet. The study was drawn up by focus.com and accompanied by a really nice graphic (below).

What i find most interesting is the share of people who use the internet over the share of people who are actively blogging. The most active bloggers are found in the 35-44 age range, however the 55-64 age range are more active than 18-24 year olds. On the other hand the amount of younger people using the internet is a lot larger than older age groups.

I’m wondering, what’s it going to take for more people from younger generations to start blogging? Is there something so unappealing about signing up and creating a blog? Services such as Blogger and Tumblr have made the barriers to entry as low as possible, some would even say inviting.

Maybe it’s the fact that you can’t hide your opinions behind a blanket of other comments under the security of the latest and greatest social network. People enjoy that security, it could be that people are scared to display their opinion outside of their own friendship circles, as blogging is considerably more public and open to criticism.

Maybe it’s none of that. It could just be that the amount of people blogging about business, hobbies and the like are just a bit more into the whole blogging notion, as it will inevitably be more beneficial to their career or business. It’ll be interesting to know if in a years time these figures still sing the same song that’s for sure.

Credits for the crisp graphic go to focus.com.

Managing CRON jobs the Ruby way

2009-11-30T00:00:00+00:00

CRON jobs have always been a bit of a pain to manage. Whilst the syntax isn’t the most difficult to calculate it would be nice to have an easier way to describe the intervals at which tasks should run. Even more so the schedules should be managed alongside your other project code, as one undoubtably requires the other to run (such as Rake tasks). Lots of people have written gems and plugins to scratch this itch, one of them being whenever.

Whenever provides a really tidy DSL to declare your CRON jobs in natural Ruby syntax. A basic example looks like this:

  set :output, File.join("log", "cron.log")
  
  every 1.day do
    rake "mailer:enqueue_daily"
  end

Running the whenever command from your Rails project directory then produces the desired CRON syntax:

  # => 0 0 * * * cd /path/to/app && RAILS_ENV=production /usr/bin/env rake mailer:enqueue_daily >> log/cron.log 2>&1

This worked really well for me however I prefer to have my environment specific configurations in separate files. Although the wiki provided a way to do this it didn’t feel very clean. A simple fix to this would be to load an environment specific file if it exists:

  set :output,   File.join("log", "#{@environment}_cron.log")
  set :env_file, File.join("config", "schedule", "#{@environment}.rb")

  # Global jobs
  every 1.day do
     rake "mailer:enqueue_daily"
  end

  # Load environment specific jobs
  load(@env_file) if File.exists?(@env_file)

Therefore running whenever --set environment=staging would load RAILS_ROOT/config/schedule/staging.rb with your staging environment specific jobs and output the result to log/staging_cron.log. You can also easily integrate this into a Capistrano task when you deploy new code:

  desc "Update the crontab file from the rules within config/schedule.rb"
  task :update_crontab, :roles => :db do
    run "cd #{release_path} && whenever --update-crontab #{application} --set environment=#{rails_env}"
  end

It’s tools like these that make maintaining larger applications that little bit easier.

Go Configure...

2009-11-11T00:00:00+00:00

Its handy to keep common configuration values inside a single file. This is nothing new and people have been doing something like this in Rails apps for quite some time:

  CONFIG = YAML.load_file(RAILS_ROOT + "config/settings.yml")

Whilst this is perfectly fine its not the nicest to use across your app. To me it doesn’t look very pretty when accessing the values.

Given the following YAML file:

  meta:
    keywords: "my, app, keywords"
    description: "My apps description."

Accessing these values using the above method would look like this:

  CONFIG["meta"]["description"]

I wanted to scratch this itch, so I made a quick and dirty plugin to solve the issue. I wanted to be able to access the values easier and have them available within all of my models, mailers, views and controllers. I also wanted the plugin to auto load once dropped into a Rails project.

To make things even easier I also wanted a Rake task to copy the YAML files into the config/ directory. Although this is a trivial job, this would help to automate the installation process when using Rails templates.

Using the plugin is simple. Once installed, enter some configuration values into a YAML file as normal. Then access the values from within this file within your app like this:

  c(:meta, :keywords)

The plugin will try to load config/environment.yml and config/environments/RAILS_ENV.yml if they exist. To install these files automatically simply enter rake configurable:install to copy over the default files:

  >> Added configuration files:
  + config/environment.yml
  + config/environments/development.yml
  + config/environments/staging.yml
  + config/environments/production.yml

If you want to extend the plugin to be accessible in other areas of your application, you can add another hook into the self.setup method located in lib/configurable/configuration.rb:

  # lib/configurable/configuration.rb
  
  def self.setup(files)
    # ... code omitted
    
    # Add any other classes you may want the plugin to be available in below:
    MyClass::Base.send :include, Helpers
  end

The codes nothing impressive and its a tiny plugin but its one of those things thats been annoying me for a while.

Get the code:

» Directly from GIT git://github.com/joshnesbitt/configurable.git.
- » From the Github repository http://github.com/joshnesbitt/configurable.

Creating simple unique tokens in Ruby with Hasher

2009-10-12T00:00:00+01:00

Randomly hashed tokens have thousands of uses. One of them is creating a random secure hash as a handle on a user account:

http://awesomewebapp.com/confirm/843e9d74d716b41517addde515f4e6db940be21d

A random unique hash is created and assigned to your account, allowing you to confirm your identity through a link without disclosing any personal information about who you are. Hashes like this can easily be created by digesting a single string or a string with an additional salt injected into it to ensure the original string is even more indecipherable.

I was seeing quite a lot of duplication in areas of my code where I was constructing hashes based on certain attributes of a particular class. This required me to build up a hash-able string and then pass it through SHA1 to digest it. Although this is only a few lines of code, its nice to bundle this up into a reusable module so you can just do something like this:

  generate_hash "somevalueiwanttohash" # => 1a09a8b3dd4b24d5284ac13705523d1b15b55e64

Thats where Hasher comes in, a simple module you can mix into any Ruby project. Heres the code:

  # hasher.rb
  # git://github.com/joshnesbitt/hasher.git
  
  module Hasher
    require 'digest/sha1'
    SECRET = "ilikebigbuttsandicannotlie"

    def generate_hash(string)
      encrypt string
    end

    protected
    def encrypt(string)
      Digest::SHA1.hexdigest(salt + string)
    end

    def salt
      Digest::SHA1.hexdigest(SECRET + time_stamp + random)
    end

    def time_stamp(time = Time.now)
      time.strftime("%d%Y%l%M").to_s
    end

    def random(size = 10)
      (0...size).map { character_set[ rand(character_set.size) ] }.join
    end

    def character_set
      [('0'..'9'),('A'..'Z')].map { |range| range.to_a }.flatten
    end
  end

So say you want to create a confirmation token for a user before_create, you would use it in an ActiveRecord model like this:

  # app/models/user.rb
  
  class User < ActiveRecord::Base
    include Hasher
    
    # ... other ruby code ...
    
    before_create :generate_confirmation_token
    
    protected
    def generate_confirmation_token
      self.confirmation_token = generate_hash("#{self.id}-#{self.email}")
    end
  
  end

Its not much, but if you end up having more models needing similar functionality it makes sense to package it up into a nice mixin.

Using ActionMailer layouts with Clearance

2009-09-11T00:00:00+01:00

If you’re building a Rails application thats going to be sending a lot of emails, chances are you’re going to want to have a consistent layout between each. You use layouts within your application so why not use them in your mailers? Since commit e9a8e0053be3b293ab89fb584f1d660063f107aa you’ve been able to use layouts within your mailers, making it easy to implement a consistent email format across all emails.

You use them just like normal rails layouts, i tend to explicitly declare the mailer layout at the top of my mailer class. More specifically i usually declare a base mailer class to use as the parent class for all my other mailers, making setting up and configuring each mailer really easy, heres what mine looks like:

# app/models/base_mailer.rb

class BaseMailer < ActionMailer::Base
  layout "mailer"
  default_url_options[:host] = HOST # set in development/staging/production.rb

  protected
  def setup_mail(email)
    from          DO_NOT_REPLY # set in development/staging/production.rb
    recipients    email
    subject      "[My Mailer] "
    sent_on       Time.now
  end

end

Then my object specific mailer:

# app/models/enquiry_mailer.rb

class EnquiryMailer < BaseMailer

  def after_create(enquiry)
    setup_mail(enquiry.email)
    @subject += "Thanks for contacting us."
    @body = { :name => enquiry.name }
  end

end

I, like a lot of people use Clearance, an authentication engine written by the guys from Thoughtbot. Obviously you want to take advantage of the same mailer layouts within the Clearance mailers too, so lets tweak the ClearanceMailer class to tell it about the layout:

# config/initializers/apply_clearance_mailer_layout.rb

ClearanceMailer.class_eval { layout "mailer" }

This performs a class_eval, modifying the class when the application initialises to use the mailer layout we’re using in the rest of our application. If you’re sending a lot of mails it pays off to keep them all consistent, as sometimes this is the first port of call for your client/customer.

Implementing a simple lock file system in Ruby

2009-09-08T00:00:00+01:00

Lock Files are great for monitoring when a resource is in use, Wikipedia explains it simply:

File locking is often used by shell scripts and other programs: creation of lock files, which are files whose contents are irrelevant (although often one will find the Process identifier of the holder of the lock in the file) and whose only purpose is to signal by their presence that some resource is locked. A lock file is often the best approach if the resource to be controlled is not a regular file at all, so using methods for locking files does not apply. - Wikipedia

Recently whilst working on a project i needed to implement some way of monitoring when a particular task was being executed, ensuring that a duplicate task wasn’t invoked (consuming more resources and possibly hindering the execution of the current task). Ruby (and pretty much any other programming language) provides a whole host of file manipulation classes to easily implement a simple lock file system.

With Ruby we can use the File class to manipulate files, ensuring a platform independent way of reading/writing to/from the filesystem. The actual class methods that we’ll be using within the File class come from the mixin of the ftools module (which adds a group of methods for copying, moving, deleting, installing, and comparing files).

Heres the code:

# lock_file.rb
# git://github.com/joshnesbitt/lockfile.git

class LockFile
  attr_accessor :path, :filename, :qualified_path

  def initialize(path="/tmp", filename="lockfile.lock")
    @path, @filename = path, filename
  end

  def path
    @path
  end

  def filename
    @filename
  end

  def qualified_path
    "#{@path}/#{@filename}"
  end

  def process_id
    locked? ? read_lockfile(self.qualified_path) : nil
  end

  def lock!
    locked? ? false : create_lockfile(self.qualified_path)
  end

  def unlock!
    unlocked? ? false : destroy_lockfile(self.qualified_path)
  end

  def locked?
    lockfile_exists?(self.qualified_path)
  end

  def unlocked?
    !lockfile_exists?(self.qualified_path)
  end

  protected
  def lockfile_exists?(file)
    File.exists?(file)
  end

  def create_lockfile(lockfile)
    begin
      File.open(lockfile, "w") { |f| f.write(Process.pid) }
    rescue
      raise LockFileExists
    end
  end

  def read_lockfile(lockfile)
    begin
      File.open(lockfile, "r").gets { |l| puts l }
    rescue
      raise LockFileMissing
    end
  end

  def destroy_lockfile(lockfile)
    begin
      File.delete(lockfile)
    rescue
      raise LockFileMissing
    end
  end

  class FileError < StandardError #:nodoc:
  end
  class LockFileExists < FileError #:nodoc:
  end
  class LockFileMissing < FileError #:nodoc:
  end  

end

This is a very basic implementation as i didn’t need anything uber fancy. Use it like this:

l = LockFile.new("/path/to/lockfile", "test.lock")
# => "/path/to/lockfile/test.lock" containing the PID

l.locked?
# => false

l.lock!
# => Integer value

l.locked?
# => true

l.process_id
# => PID

l.unlock!
# => Integer value

l.locked?
# => false

To put it into context a bit more, imagine you’re doing something like this within your application:

def do_something
    call_rake_task
end

When hit this method will call a Rake task from within your application, lets not worry about why you might be doing this for now but focus on how we ensure this task isn’t invoked again if it’s already being executed:

def do_something
    call_rake_task unless @lockfile.locked?
end

Where you initialize the lock file instance is up to you, i ended up creating it in the initializer of the class i was using it in to make sure it was available as soon as the class was instantiated.

There are lots of uses for lock files, for me this was just one quick solution to a problem. Also its important to note that there are other solutions to performing expensive tasks outside of a typical request (most of which are better than using plain old lock files), you might want to look at these too:

Bonza!

Deployment tip when using Passenger and Capistrano

2009-09-06T00:00:00+01:00

A quick tip i picked up from a Gist off Github. Every time you deploy new changes to a rails application powered by Passenger using Capistrano, chances are you’re going to be restarting the application by hitting tmp/restart.txt. The first time you access the site after every restart you’ll experience a short hang whilst the passenger process spawns. A good way to stop this is by pinging the applications URL after you’ve touched the restart file:

desc "Restarts this application when running passenger"
task :restart, :roles => :app do
  run "touch #{current_path}/tmp/restart.txt" 
  `curl -s http://myhost.com $2 > /dev/null`
end

This hits the URL specified and throws the output to /dev/null. It might seem like nothing but if it saves you 10 seconds on every deploy it can’t be a bad thing…