Copying schema to a new, non-test database

As far as I can tell, there's no easy way to copy the schema for one of our databases to a new database config entry. So, here's a quickie script that schleps schemata for you. (I would love someone to tell me "Oh, to do that, you just do X" and then demonstrate it: this is kind of ugly, though it's based on how the railties Rakefile does it...)

require 'stringio'

# We want to copy the schema from class h[:db] to the database entry
# named "#{h[:prefix]}#{RAILS_ENV}"
#
to_copy = [
          { :db => SomeUseDbAbstractClass,
            :prefix => "evaldb_",
          }
         ]

to_copy.each { |h|
  db = h[:db]
  prefix = h[:prefix]

  # Place to stick the schema
  s = StringIO.new()

  # Dump the schema
  ActiveRecord::SchemaDumper.dump(db.connection, s)

  # Extract the schema
  s.rewind
  schema = s.string


  # Grab the original connection... we'll need this in a bit.
  conn_spec_original = UseDbPluginClass.get_use_db_conn_spec({})

  # Grab the destination connection information
  conn_spec =  UseDbPluginClass.get_use_db_conn_spec(:prefix => prefix)


  # SQLite3 doesn't like the ENGINEs we emit... scrubadub.
  if conn_spec["adapter"] == "sqlite3"
    schema.gsub!(/ENGINE=(MyISAM|InnoDB)/, '')
  end


  # Move the default AR connection over to our destination database
  ActiveRecord::Base.establish_connection(conn_spec)

  # Play back the "migration" schema dump
  eval schema

  # Restore connection to its proper home
  ActiveRecord::Base.establish_connection(conn_spec_original)
}

edge rails non-migration support for populating new tables

http://ryandaigle.com/articles/2009/5/13/what-s-new-in-edge-rails-database-seeding

We could use to (re)populate our indexes (QuestionIndex, etc.).

an network aggregation utility/

http://googleblog.blogspot.com/2009/05/netlog-integrates-with-google-friend.html

S3 and retries

As an FYI, reading from S3 isn't necessarily reliable. In the past couple of days, I've seen a lot of ETIMEDOUT errors coming back from the connection. The first step was to add 10x retry, which took a while, but failed again. Then, I added 10x retry with 1 second sleeps between retries, which, again, worked for a while, then failed. Finally, I wound up at 25x retries with 1s sleeps; this seems to work. (Silly me, I thought s3sync's 100x retry policy was just paranoia or overly cautious...)

Except that now I'm getting occasional 'getaddrinfo: nodename nor servname provided, or not known (SocketError)'.

So, if you're reading in data stored in S3 and want it to be reliable, you should wrap it with a generic exception handler and a lot of backing-off retries. With luck, it'll work out for you. Without luck, well, what sort of reliability do you expect from a "cloud"?

  max_attempts = 25
   attempts = 0
 
   begin
     my_stuff = s3object.load
   rescue Exception => e
     attempts += 1
     sleep 1
     retry if attempts < max_attempts
   end

redis twitter-clone example

http://code.google.com/p/redis/wiki/TwitterAlikeExample

hmm...this looks to be a useful technology bridge between memcached and mysql...plus there are ruby bindings.

i'll spend some time playing with it and report back to the team.

-N.

really nice activerecord feature i was unaware of

an easy way to calc stats:

http://api.rubyonrails.org/classes/ActiveRecord/Calculations/ClassMethods.html

we use it a few places, but i was unaware

--
Love,
Fritz

ruby proxies for testing staging clusters and much more

Intriguing use of proxies for testing a staging cluster:

http://www.igvita.com/2009/04/20/ruby-proxies-for-scale-and-monitoring/