URLs and domain names in international characters

Looks like ICANN has finally approved domain names with international characters. I'm a huge fan of this change away from the english centered web. Regexes that validate domain names aren't going to cut it much longer.

for when deploying from svn stops scaling...

http://dev.bizo.com/2008/09/capistrano-deployment-with-s3.html

improving mysql performance: some notes and links

• NDB mysql clusters. Handles transactions, is about as fast as memory tables, can do replica / master-master clusters, but because it's multi-server it's fundamentally more scalable as a transaction engine.
  

• Patches from Google to improve replication stability, mysql/innodb performance with > 4 cores

• Caching InnoDB memory pages into a Memcached server as a way to dramatically improve innodb performance and expand the memory available for it to use. It's a very small patch to mysql.

• Mysql + Hadoop: recently there has been a lot of noise about using them in conjunction as a way to dramatically scale the types of read operations you can do in a replicated relational database environment. E.g., either the data in the tables is enormous, and you want to do complex operations over it all at once that would typically cause a single database server (even though the data might all fit) to blow up. Not sure if someone has a setup working out of the box, but the idea is that you load a "snapshot" into hadoop as you would a new replica, then process binlogs (row replication updates are ideal for this) to update the dataset in the hadoop cluster. Would work pretty smoothly with, say, our ML database. Wonder if the INVERSE could be done too (take a dataset in a hadoop cluster, compute some set of updates, and then generate the appropriate row updates to send to your live DB cluster.

ec2 security groups restrictions within the cluster

According to the Ec2 security groups documentation, within your ec2 cluster of servers, it is not possible to setup access rules within the cluster that reference both multiple groups of servers (by aws account or security groups) and ports/protocols. You authorize between multiple groups of servers, but it must be for all ports and protocols. If you are restricting by CIDR (ip addres) you can restrict by port and protocol, but not within your own cluster.

When authorizing a user/group pair permission, GroupName, SourceSecurityGroupName and SourceSecurityGroupOwnerId must be specified. When authorizing a CIDR IP permission, GroupName, IpProtocol, FromPort, ToPort and CidrIp must be specified. Mixing these two types of parameters is not allowed.

http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/index.html?ApiReference-soap-AuthorizeSecurityGroupIngress.html

http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-AuthorizeSecurityGroupIngress.html

However, this documentation is straight up WRONG. It is indeed possible, although completely undocumented, to add access restrictions that reference amazon accounts, security groups, protocols, and ports all together. You can only use the SOAP API (as well as the command line tool ec2-authorize, which uses the SOAP api) to do this, not the Query API (what the amazon-ec2 and right-aws gems both use):

~> ec2-authorize backendservers -P tcp -p 8080 -u $AWS_USER_ID -o frontendservers
GROUP backendservers PERMISSION frontendservers ALLOWS tcp 8080 8080 FROM USER (redacted) GRPNAME frontendservers

Someday this will likely showup in the Query API, but until then, we're stuck coding for SOAP or referencing the command line tools to make use of this lovely feature that you'd expect would be standard. This feature has been live for well over a year (I started using it in spring of 2008), so it's really surprising that it's not available in all of the apis.

collectd versus munin

http://collectd.org/

* more efficient C interface than munin's cron/perl-based polling mechanism
* cleaner C interface versus munin's perl
* nice interfaces for ruby, java, C, erlang http://collectd.org/related.shtml
* lots of options for graphing, data manipulation/utilization
* similar default out-of-the-box plugins for all the major stuff; all the other ones had to be customized for munin anyhow.

Vidar tweeted this a little bit ago: http://github.com/ileitch/hijack/tree/master

Hijack: Provides an irb session to an existing ruby process.

Intro

Hijack allows you to connect to any ruby process and execute code as if it were a normal Irb session. Hijack does not require your target process to require any hijack code, Hijack is able to connect to any ruby process. It achieves this by using gdb to inject a payload into the process which starts up a DRb server, Hijack then detaches gdb and reconnects via DRb. Please note that gdb will halt your target process while it is attached, though the injection process is very quick and your process should only be halted for a few milliseconds.

Hijack uses DRb over a unix socket file, so you need to be on the same machine as the process you want to hijack. This is by design for security reasons. You also need to run the hijack client as the same user as the remote process.

Design Patterns for Social Experiences

A good article in ASIS&T's Bulletin about the information architecture of social experiences/interfaces.

http://asis.org/Bulletin/Aug-09/AugSep09_Crumlish.html

The associated wiki seems pretty good, too. It's a set of design patterns for these things:

http://designingsocialinterfaces.com/patterns.wiki/index.php?title=Main_Page

"The Flawed Theory Behind Unit Testing"

A nice, quick piece pointing out the real value in testing: they make you think harder about what you're doing. He omits it, but it seems like refactoring gets its benefits from much the same reasons.

My point is that we can't look at testing mechanistically. Unit testing does not improve quality just by catching errors at the unit level. And, integration testing does not improve quality just by catching errors at the integration level. The truth is more subtle than that. Quality is a function of thought and reflection - precise thought and reflection. That’s the magic. Techniques which reinforce that discipline invariably increase quality.

http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

HTTP PubSub: Webhooks & PubSubHubbub

http://www.igvita.com/2009/06/29/http-pubsub-webhooks-pubsubhubbub/

zero-downtime rolling restarts behind a proxy balancer

This very good example configuration is with HAProxy, but it will work with any proxy service that has health checking.

http://www.igvita.com/2008/12/02/zero-downtime-restarts-with-haproxy/

website speed and effects on users

http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-pag.html

Another Enterprise Ruby

(Another) Enterprise Ruby

DESCRIPTION:

Wish you could write your Ruby in XML? Has the fact that Ruby is not "enterprise" got you down? Do you feel like your Ruby code could be made to be more "scalable"? Well look no further my friend. You’ve found the enterprise gem. Once you install this gem, you too can make Rails scale, Ruby faster, your code more attractive, and have more XML in your life.

I’m sure you’re asking yourself, "how can the enterprise gem promise so much?". Well the answer is easy, through the magic of XML! The enterprise gem allows you to write your Ruby code in XML, therefore making your Ruby and Rails code scale. Benefits of writing your code in XML include:

* It’s easy to read!
* It scales!
* Cross platform
* TRANSFORM! your code using XSLT!
* Search your AST with XPath or CSS!

The enterprise gem even comes with a handy "enterprise" binary to help you start converting your existing legacy Ruby code in to scaleable, easy to read XML files. Let’s start getting rid of that nasty Ruby code and replacing it with XML today!

master-master database replication

http://www.rubyrep.org/

Profiling Ruby with Google's perftools

http://www.igvita.com/2009/06/13/profiling-ruby-with-googles-perftools/

filed a Rails Enhancement

https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/2763-question-and-serialized-attribute-methods-should-raise-activerecordmissingattributeerror

This is the patch and test case already applied to our local Rails 2.3.2 tree. It ensures we can catch an exception and reload the object when we access attributes that aren't in the cached data.

developer build of OS X chrome available

http://blog.chromium.org/

If you're as sick of FFox & Safari as I am, this is good news!

mysql throws "Server shutdown in progress" error when a thread is killed

This rather serious sounding error will probably be revised, but if
you ever see it, know that it might not be the actual server shutting
down, but rather just the local client thread to the database being
terminated, possibly by a maintenance operation of some sort.

http://bugs.mysql.com/bug.php?id=18256

Copying schema to a new, non-test database

As far as I can tell, there's no easy way to copy the schema for one of our databases to a new database config entry. So, here's a quickie script that schleps schemata for you. (I would love someone to tell me "Oh, to do that, you just do X" and then demonstrate it: this is kind of ugly, though it's based on how the railties Rakefile does it...)

require 'stringio'

# We want to copy the schema from class h[:db] to the database entry
# named "#{h[:prefix]}#{RAILS_ENV}"
#
to_copy = [
          { :db => SomeUseDbAbstractClass,
            :prefix => "evaldb_",
          }
         ]

to_copy.each { |h|
  db = h[:db]
  prefix = h[:prefix]

  # Place to stick the schema
  s = StringIO.new()

  # Dump the schema
  ActiveRecord::SchemaDumper.dump(db.connection, s)

  # Extract the schema
  s.rewind
  schema = s.string


  # Grab the original connection... we'll need this in a bit.
  conn_spec_original = UseDbPluginClass.get_use_db_conn_spec({})

  # Grab the destination connection information
  conn_spec =  UseDbPluginClass.get_use_db_conn_spec(:prefix => prefix)


  # SQLite3 doesn't like the ENGINEs we emit... scrubadub.
  if conn_spec["adapter"] == "sqlite3"
    schema.gsub!(/ENGINE=(MyISAM|InnoDB)/, '')
  end


  # Move the default AR connection over to our destination database
  ActiveRecord::Base.establish_connection(conn_spec)

  # Play back the "migration" schema dump
  eval schema

  # Restore connection to its proper home
  ActiveRecord::Base.establish_connection(conn_spec_original)
}

edge rails non-migration support for populating new tables

http://ryandaigle.com/articles/2009/5/13/what-s-new-in-edge-rails-database-seeding

We could use to (re)populate our indexes (QuestionIndex, etc.).

an network aggregation utility/

http://googleblog.blogspot.com/2009/05/netlog-integrates-with-google-friend.html