in web we trust

The last month I have been working in Query Memcached, a plugin that replaces the Query Cache that comes with Rails, adding a Memcache layer for persisting the query's cache between requests. That means that the cache does not expire at the end of the request, because it is stored in Memcached.

If you are not familiar with Query Cache, you only have to know that it's a very simple caching system from ActiveRecord that stores in memory all the queries performed during a request. If a query is fired more than one time, it will be in cache, so the database won't be affected.

The adventages of my approach are evident: you only use the database when some table has been modified or when a new query is executed. So you can get a notice an important speed improvement.

It is ideal for pages which have a lot of customization for the user logged in the application: in that cases it is very difficult to cache fragments or pages, because of the expiring of all that cache, and so many things.

Cache expiring

For expiring this cache, each Memcached key contains a sum of all the version numbers of the tables involved in the query. If one of that tables is modified, then the version number for that table is increased.

For example, the query below involves the table items and the table places:

So, the version for the cache of that query will be the sum of the cache version of the table items and the cache version of the table places.

Disclaimer

We are using this plugin in two on-line web projects: unvlog.com and iwannagothere, and everything seems to work fine (and faster than before), but these sites don't have so much traffic (about 30.000 req/day), so, there are stille possibilities that appear bugs and "unexpected behaviours".

In spite of all that, if you still are encouraged to try feel free to ask me anythin.

More information at README.

At least this week I had some time to spent in ending a litte small project: adapt and improve the cache_test plugin to Rails 2.x. You can find it at GitHub: cache_test.

Cache test is a very useful plugin that allows you to test if action/fragment/page cache is generated or expired during a request. For me is very surprisingly that Rails hasn't asserts for testing the cache, specially when, if your application uses cache, it can be the cause of its malfunctioning. Apart of that cache is so tedious to debug.

The way it works is very easy:

     assert_cache_fragment({:fragment => 'total_posts'}) do
       get :index, :user_id => users(:first)
       assert_response :success
     end
 

In my plugin I have adapted the cache storage definition to Rails 2.x (moved to ActiveSupport, etc), added more asserts and some tests.

Enjoy it!

More info in the README.

These days I have been working in a small parser to convert BBCode to HTML in Ruby.

The motivation was that I need it (of course), and there wasn't anything complete enough. So I found bb-ruby at GitHub and I decided to complete it with more tags, tests, and some stuff for doing it more flexible.

The result is a fork that you can find here:

http://github.com/ferblape/bb-ruby/tree/master

You can read the README file for getting more information.

A small snippet for testing.

If you want to add some special actions before and after the execution of each test, and don't want to add it to setup and teardown because you will be repeating your code for each unit test and functional test, or because it is a block, you can do an alias_method_chain for the run method:

 module Test
   module Unit
     class TestCase
       
 
       def run_with_my_block(*args, &block)
         Cache.clean # code before 
        
         a_block_that_you_want_to_execute do 
           run_without_my_block(*args, &block)
         end
         
         # Code after
       end
 
       alias_method_chain :run, :my_block
 
     end
   end
 end
 

You can redefine it at the beginning of your test_helper.rb for example.

Note that this code is only for Rails, becaus of the use of alias_method_chain function.

Google Maps is, may be, the coolest service for integrating maps in your web applications: it offers a powerful and easy API with no limit usage.

The problem is that it loads quite Javascript that has to connect to the main Google Maps page in order to draw our map. In order to avoid that we can use static Google Maps, it is, a picture (generated in real time) of the map we want to visualize. The problem with this is that you can't interact with the map (because it is an image) and that there is a limitation per day. But may be you find it useful in a particular moment.

For example, in iwannagothere.net we have a small map in every place that shows the location of that city:

In that situation we can change the map because the map is not for navigating along the city. For that you have a map in each item.

In Ruby on Rails, the programming framework of iwannagothere.net there is a gem called static-gmaps that does the hard work for you. What's more, we have found the way to integrate this maps with attachment_fu, in order you save a copy of the map in your filesystem and then avoid the day limitation usage.

Basically we have an attachment_fu kind of model (nothing special here):

class PlaceMap < ActiveRecord::Base
   belongs_to :place
 
   has_attachment  :content_type => :image, 
                   :storage => :file_system,
                   :processor=> :rmagick,
                   :path_prefix => 'public/userfiles/place_maps', 
                   :max_size => 1.megabyte
 
   validates_as_attachment  
 end
 

The we generate the static map instance:

	
 map = StaticGmaps::Map.new :center   => [ place.lat, place.long ],
                             :zoom     => 12,
                             :size     => [ 334, 144 ],
                             :key      => APP['gmkey']  
 

And save it in our PlaceMap model:

 PlaceMap.create(:uploaded_data => (open(map.url)), :place_id => place.id, :content_type => 'image/gif')
 

The problem here is that we are not uploading a file in the usual way: we are trying to save a raw of data. Attachment_fu doesn't know how to do that, so we have to apply a little patch:

 module Technoweenie
   module AttachmentFu
     module InstanceMethods
 
       def uploaded_data=(file_data)
         return nil if file_data.nil? || file_data.size == 0
         self.content_type = file_data.content_type
         self.filename     = Time.now.to_i.to_s+'.gif'
         if file_data.is_a?(StringIO)
           file_data.rewind
           self.temp_data = file_data.read
         else
           self.temp_path = file_data
         end
       end          
 
     end
   end
 end
 

Here we are giving a random filename, generated from the current time.

And that's all, really easy :)

Muchas veces cuando trabajamos con ActiveRecord nos olvidamos / ignoramos qué hace por debajo.

Un claro ejemplo lo encontramos cuando tenemos una relación :has_many y queremos comprobar si un elemento está incluída en dicha relación.

Por ejemplo:

 class Place < ActiveRecord::Base
   has_many :items
 end
 

 class Item < ActiveRecord::Base
   belongs_to :place
 end
 

Seguro que más de una vez, para comprobar que un place contiene un ítem hemos utilizado el "rubysta" include?:

 place.items.include?(item)
 

Pensándolo fríamente 0.2 segundos nos damos cuenta de que estamos cargando un array con todos los elementos de una relación, que pueden ser 10 o pueden ser 1.000.000. Y ya sabemos lo generoso que es Ruby con la memoria que toma y que nunca deja ir.

Un truco para evitar que esto suceda sin dejar de utilizar include? es definir dicho método dentro de la relación tal que así:

 class Place < ActiveRecord::Base
   has_many :items do 
     def include?(item)
       count(:conditions => ["item_id = ?", item.id]) > 0
     end
   end
 end
 

Es decir, resolvemos el problema con un simple COUNT de Mysql.

¿Alguien se anima a hacer un plugin? Creo que sería bastante inmediato para cuando se cumplen las convenciones y sabemos que la clave foránea de la relación es singular del nombre de la relación_id. O quizá ni eso, si encontramos la forma de preguntarle a ActiveRecord cuál es la clave foránea de una relación.

Cuando tienes una aplicación en producción es más que necesario controlar los tiempos de las peticiones más lentas.

Sin embargo muchas veces con el log de Rails no es suficiente, y hay que alcanzar un mayor nivel de detalle.

El log en modo debug nos da todo el detalle que necesitamos, pero mostrando demasiada información (¿de qué nos sirve saber que un SELECT sencillo tarda 0.001 segundos?).

Pero si en ese log pudiéramos filtrar las consultas SQL más lentas sí que tendríamos información de consultas lentas asociadas a cada petición al servidor.

Gracias al plugin query trace y a una pequeña modificación podemos mostrar qué consultas tardan más de 1 segundo:

 module ActiveRecord
   module ConnectionAdapters
     class AbstractAdapter
       def log_info(sql, name, runtime)
         return unless @logger
         @logger.info(
           format_log_entry(
             "#{name.nil? ? "SQL" : name} (#{sprintf("%f", runtime)})",
             sql.gsub(/ +/, " ")
           )
         ) if (RAILS_ENV == 'production' && runtime > 1.0) || RAILS_ENV != 'production'
       end
     end
   end
 end
 

Este código lo he probado en Rails 1.2.x, no sé si funcionará en otras versiones.

Una cuestión que hay que tener presente a la hora de trabajar con Memcached y que no se suele tener en cuenta es el tamaño de las claves y los valores:

• claves: máximo de 250 caracteres
• valores alojados: máximo 1 Megabyte

Es muy difícil que una aplicación Rails las claves generadas a partir de las URL's tengan tanto tamaño, pero puede darse el caso y hay que tenerlo presente.

En el grupo de discusión acts_as_cached esta semana han propuesto hashear (con MD5, por ejemplo) las claves antes de guardarlas para curarnos en salud.