Cassandra source on Google Code

As predicted, Facebook has open sourced Cassandra. It's available on Google Code with very little fanfare.

Some notes after my browsing through the source:

• It uses hinted handoff and bootstrapping just like Amazon's Dynamo


• Its consistency model doesn't seem to be quite the same - Dynamo uses vector clocks to determine causal relationships, whereas Cassandra seems to be just based on timestamps and "majority rules" semantics when timestamps are tied.


• Membership is communicated by a gossip protocol as described in the Dynamo paper.


• Requests are made to the system by sending thrift calls to any node. The thrift interface is included in the source.

Some further thoughts:

• It doesn't seem like there's a lock on the table during bootstrapping. What happens to mutations made on the source node while it is bootstrapping the destination? Are they marked for later hinted handoff?


• Would system performance be improved by using the new Thrift TNonblockingServer (see THRIFT-5 on JIRA)? It should be more scalable than the TThreadPoolServer they're using now.


• Cassandra is around 40K lines of Java. How many lines would an equivalent Erlang program be, and what would be the performance difference?

All in all, it's a very interesting project sure to attract much attention. Now that Powerset has been acquired by Microsoft, I'm a little worried for Hbase's future -- two of the three main developers are Powerset employees. Maybe Cassandra can help fill the open source scalable database niche.

Facebook's next open source projects: Hive and Cassandra

A couple weeks ago, Jeff Hammerbacher from Facebook presented some details on Cassandra (see later slides), a structured p2p storage system similar to Google's Bigtable or Amazon's Dynamo. What is most interesting about Cassandra is that they seem to be preparing to open source it imminently. Jeff bookmarked two things on delicious last night:

1. Cassandra: Welcome to your new Wikidot site


2. Cassandra: A Structured Storage System on a P2P Network in Launchpad
   

Both sites are empty as of now, but it looks like they're planning on releasing the source some time soon using bzr for version control.

Another interesting Facebook project is Hive, a sort of data warehousing solution built on Hadoop. They've been discussing open sourcing this for several months now, but it looks like things are starting to happen with HADOOP-3601: Hive as contrib project.

On the non-Facebook open source front, we've got some news coming soon as well. We've made the decision to open source several of our internal tools under an MIT license - hold tight for more info.

How to work around an MSIE bug with Scriptaculous

For several months we've noticed a lot of 404s in our access logs that look like this:

x.x.x.x - - [2008-06-23 13:14:30] "GET /static/r/toLRhQ/js/'+libraryName+' HTTP/1.1" 404 549 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
x.x.x.x - - [2008-06-23 13:14:30] "GET /static/r/toLRhQ/js/'+libraryName+' HTTP/1.1" 404 568 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
x.x.x.x - - [2008-06-23 13:17:06] "GET /static/r/toLRhQ/js/'+libraryName+' HTTP/1.1" 404 568 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
x.x.x.x - - [2008-06-23 13:17:06] "GET /static/r/toLRhQ/js/'+libraryName+' HTTP/1.1" 404 549 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
x.x.x.x - - [2008-06-23 13:18:50] "GET /static/r/toLRhQ/js/'+libraryName+' HTTP/1.1" 404 549 - "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"

For a long time we just ignored them, but I finally decided to track the issue down yesterday, and found the following line in scriptaculous.js:

    document.write('<script type="text/javascript" src="'+libraryName+'"></script>');

For whatever reason, this particular version of MSIE decides to interpret this script tag in the middle of the javascript and go trying to find a JS named "'+libraryName+'". This is clearly incorrect behavior, and it seems like the bug was fixed in a later version of IE.

Eric came up with the following clever fix:

    document.write('<' + 'script type="text/javascript" src="'+libraryName+'"></script>');

Splitting the '<' and the 'script' tricks the broken version of IE into not seeing the script tag and solves the issue.

Hope this is helpful to others who find these mysterious requests in their logs.

Automatically parallelizing list comprehensions in Erlang

This evening I got to chatting with some people on IRC about how it would be neat if Erlang had syntax to automatically parallelize a list comprehension. For those who aren't familiar with Erlang yet, here's a brief introduction:

A list comprehension is a quick way of performing an operation over a list of values. For example, at the erlang prompt:

1> MyList = lists:seq(1,10).
[1,2,3,4,5,6,7,8,9,10]
2> [X * 2 || X <- MyList].
[2,4,6,8,10,12,14,16,18,20]

For those familiar with Python list comprehensions, this is equivalent to [x * 2 for x in range(1,11)]. It's also just like the map operator common in pretty much every functional language. In both Python and Erlang you can get a bit more complicated:

3> [X * 2 || X <- MyList, X rem 2 == 1]. %% Filter out the odd numbers from the list
[2,6,10,14,18]

or even take the cartesian product of several lists in the same list comprehension:

4> [{X, Y, X * Y} || X <- lists:seq(1,3), Y <- lists:seq(1,3)].
[{1,1,1},
 {1,2,2},
 {1,3,3},
 {2,1,2},
 {2,2,4},
 {2,3,6},
 {3,1,3},
 {3,2,6},
 {3,3,9}]

Now, back to the interesting part. Everyone talks about how Erlang is so easy to make concurrent. But if we add the self() function, which returns the current running process id, to the output of the list comprehension we see that it is not parallelized:

5> [{self(), X, Y, X * Y} || X <- lists:seq(1,3), Y <- lists:seq(1,3)].
[{<0.31.0>,1,1,1},
 {<0.31.0>,1,2,2},
 {<0.31.0>,1,3,3},
 {<0.31.0>,2,1,2},
 {<0.31.0>,2,2,4},
 {<0.31.0>,2,3,6},
 {<0.31.0>,3,1,3},
 {<0.31.0>,3,2,6},
 {<0.31.0>,3,3,9}]

All of the results were computed in the same process.

Handily, Erlang provides a way of inserting user code in between the parser and the compiler in the form of parse transforms. A parse transform is specified in the compile options for a module and can make arbitrary modifications to the abstract syntax tree of an Erlang program before the compiler gets to it, with the important caveat: "programmers are strongly advised not to engage in parse transformations and no support is offered for problems encountered." Everyone knows caveats are lame, so I went ahead and built a parse transform. Here's example usage:

-module(test).
-compile({parse_transform, plc}).
-export([test/0]).

test() ->
    Result = plc:lc([{self(), A, B, C} || A <- lists:seq(1,2),
                                  B <- lists:seq(3,4),
                                  C <- lists:seq(5,6),
                                  A * B < 5
                                 ]),
    io:format("result: ~p~n", [Result]).

Output:

result: [{<0.3145.0>,1,3,5},
         {<0.3146.0>,1,4,5},
         {<0.3147.0>,1,3,6},
         {<0.3148.0>,1,4,6}]

We can see that the list comprehension has been performed in parallel this time - each result element has a different value of self().

Want to check it out? Clone the git repo on github.

This probably doesn't work that well. Don't use it on production software. Unless you want to test it thoroughly first.

Confidence intervals for Jaccard Similarity?

Hoping that someone is googling for the right terms here:

Anyone out there know how to calculate a confidence interval around an estimate of the Jaccard similarity coefficient?

For Pearson correlation you can use Fisher's Z-prime Transformation, but I can't quite figure a principled way of doing the same for Jaccard similarity.

FB Engineering blog post on Facebook Chat

Eugene at Facebook posted an interesting article about the technology behind the new Facebook Chat. This new service has large parts written in Erlang and communicates with the rest of the system using the Thrift bindings Amie Street and Facebook have been collaborating on for the last couple of months.

The good news for us: our thrift bindings are pretty much guaranteed to be stable and leak/bug free now that they're used for millions of messages/second over at FB.

If you're interested, check out over at the thrift git repository

Forcing a process to garbage collect in Erlang

We upgraded our dynamic pricing service tonight with a new version of thrift, so I was checking top to make sure everything was cool a few hours later. I noticed that one of the pricers was using 1.1G of RAM - significantly more than I'd ever seen it using before. Figuring it was a memory leak, I started a console node and connected it to the erlang cluster:

amiest@app2:~$ erl -name console
Erlang (BEAM) emulator version 5.5.2 [source] [64-bit] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
(console@app2.prod.amiestreet.com)1> P = pricer@app2.prod.amiestreet.com. 
'pricer@app2.prod.amiestreet.com'
(console@app2.prod.amiestreet.com)2> net_adm:ping(P).
pong

First step of diagnostics was to fire up etop with etop:start([{node, P}]). This showed that the process count was remaining stable at an appropriate number -- we'd had a bug with a process leak once before but it didn't seem to be the cause this time. The amount of RAM used by processes seemed pretty high though. Next step:

(console@app2.prod.amiestreet.com)4> rpc:call(P, shell_default, i, []).

This is equivalent to running the i() command on the remote node, and shows all the running processes along with some info.

This printed out something useful - the rex process was using almost 900MB of RAM for no apparent reason. I'd never heard of rex, but evidently it handles remote execution from other languages, and possibly RPC as well. Checking on some other erlang nodes I saw that rex usually only used a few hundred KB.

After much googling, I came across this article which is my only plausible explanation for how rex got so big -- the Erlang GC doesn't run on a process if the process isn't doing any work.

The solution? rpc:call(P, erlang, garbage_collect, [pid(5038,10,0)]). (5038.10.0 was the pid shown by i()). This kicked the memory usage back down where it should be.

io_lib_pretty - a nice secret module

There are some modules in the erlang stdlib that aren't exactly advertised, but are quite useful. My newest discovery is io_lib_pretty. It hasn't got a manpage, but there are some docs if you less `locate io_lib_pretty.erl`.

io_lib_pretty is the module used by the shell to print records in a nicely formatted way. This isn't possible using plain io:format but can make program output a lot nicer.

Take for example a logging program that deals with records that look like this:

5> L = #logMessage{actor=23507, server_ip = <<123,234,123,234>>}.
#logMessage{actor = 23507,
            server_ip = <<"{\352{\352">>,
            timestamp = undefined,
            level = undefined,
            log_filename = undefined,
            message = undefined}

If you just try to print it out, you get:

7> io:format("Logged: ~p", [L]).
Logged: {logMessage,23507,<<"{\352{\352">>,undefined,undefined,undefined,undefined}ok

Pretty useless output.

Using io_lib_pretty you can get:

9> io:format(io_lib_pretty:print(L, fun(logMessage, 6) -> [actor, server_ip, timestamp, level, log_filename, message] end)).
#logMessage{actor = 23507,
            server_ip = <<"{\352{\352">>,
            timestamp = undefined,
            level = undefined,
            log_filename = undefined,
            message = undefined}ok

Just like the shell. I listed the record information manually in the function above, but you can easily use the record_info macro to accomplish the same without code duplication. Or even easier, use the exprecs parse transform (pretty printing example available there).

Next time: how to load record definitions dynamically at runtime.

Now using a CDN

I just pushed a change that enables CDN delivery of most of our static content (except for the music itself... for now). In theory it should make browsing the site a bit quicker, especially for those users not in the northeastern US.

If you happen to read this and notice any difference, comment below!

EXPLAIN everything, part 2

I previously wrote a post urging people trying to optimize their database usage to run the "EXPLAIN" command on everything. This is easy advice to ignore in the case of simple queries, but even the simple queries can be big performance problems if you miss an important detail.

The other day, a friend was complaining of slow queries and was looking for some help fixing them. The first step we took was to take the "problem page" and enable debug output that printed each query that was run along with the number of milliseconds each took. If you don't have a facility for easily doing this (at the very least on a staging or development server) then you are essentially wandering in the dark for optimization.

Looking at the debug output, we found something surprising. The slowest query on the page was the seemingly innocuous:

SELECT `User`.`id` FROM `users` AS `User` WHERE `opensocial_id` = 219771253 LIMIT 1;

My friend was sure there was an index on User.opensocial_id yet the debug output showed that it took more than a second. With an index, this should take less than a millisecond. I asked him to run an EXPLAIN on it, and we saw:

mysql> EXPLAIN SELECT `User`.`id` FROM `users` AS `User` WHERE `opensocial_id` = 2778153 LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows   | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
|  1 | SIMPLE      | User  | ALL  | opensocial_id | NULL | NULL    | NULL | 272877 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
1 row in set (0.00 sec)

The entry in the possible_keys column confirmed that there indeed was an index on opensocial_id. However, NULL in the key column meant that mysql was deciding the opensocial_id key was either unusable or didn't have good enough selectivity to choose over a full table scan. The high value in the rows column along with Using where under Extra indicated that mysql was reading through every row in the table, running a comparison, and outputting those rows for which the condition matched.

So, the next question was why mysql was deciding not to use the index for this query. Doing a SHOW CREATE TABLE User revealed that the opensocial_id column was of type varchar(30). Because the column was a varchar, mysql couldn't use an index on it to do a comparison against a numeric query. You can see the reason for
this:

mysql> select "0234" = 234\G
*************************** 1. row ***************************
"0234" = 234: 1

mysql> select "234.0" = 234\G
*************************** 1. row ***************************
"234.0" = 234: 1
1 row in set (0.00 sec)

For any given numeric value in the query, there are several varchar representations that are "equal" to it (shown as value 1 above). As such, there is no way for mysql to do an index lookup on a varchar column to satisfy an equality condition against a numeric constant.

One easy solution would be to change the varchar(30) to a numeric type. Unfortunately, it turns out that opensocial user identifiers are not simple 64-bit numeric values -- depending on the opensocial platform provider they are sometimes zero-padded, sometimes not, etc. So, using a bigint unsigned was not an option.

What was the solution, then? We simply added quotes to the query:

SELECT `User`.`id` from `users` as `User` WHERE `opensocial_id` = '23424234' LIMIT 1;

This made the equality comparison between two varchars, and the index was used. The query now runs consistently in less than a few milliseconds.

In summary, EXPLAIN everything! Even the queries that look incredibly simple can be incredibly slow if they're not using the index you think they are.

PHP Pop Quiz

What would you expect the following code to do?

$foo = 0;
if ($foo == "DB") {
  die("hello world");
}
die("goodbye world");
?>

I think most rational people would expect the program to exit with the message "goodbye world."

It turns out that PHP is not so rational. Because $foo is a numeric value, it's compared using a numeric comparator with "DB". This typecast coerces "DB" into an integer value 0, so the equality matches.

The solution, of course, is to use the odd === operator which is unique to PHP. This means something along the lines of "I really mean it. No, seriously!" and disables the coercion of the dissimilar types.

The problem with that method, though, is that "0" === 0 evaluates to false.

Is there a reasonable solution to this?

Optimizing JPEGs with jpegoptim

Over the weekend I deployed a new plugin I wrote for Perlbal that tracks bandwidth usage of certain features on Amie Street. The plugin is only running on 25% of our traffic for now, but after a couple of days I had collected plenty of data to start analyzing it this morning.

The first thing I noticed is that a single one of our images was accounting for 10x as much bandwidth use as the next most transferred image on the site. I loaded it up to find that it was only a 128x128 album thumbnail, yet was a 560KB file. I downloaded it and tried to figure out why it was so big, but without much luck. I soon figured out that I could use convert -scale 1x1 to scale it to a single pixel and it still took over 500KB.

After a bit of googling I came upon jpegoptim, a utility that performs lossless compression on JPEGs by reconstructing the huffman encoding without changing the image contents. Simply running jpegoptim image.jpg I cut the image size down to 28KB and reuploaded it to production.

After scanning the top 1000 largest thumbnail images on the site, I found that the majority of them could be decreased by a good 50% or more by simply using this tool. So, if you've got a site that has a lot of JPEGs (particularly user-uploaded content that's automatically scaled using ImageMagick) give jpegoptim a try. The worst that happens is that it will report that the file is already optimal; the best case is a 95% reduction in load time for users with slow connections.

Always EXPLAIN your queries

One of the first tips if you're working on improving your database speed is to always run EXPLAIN on your queries. This lets you see the plan of execution that MySQL will take, and evaluate whether and how your indexes will be used.

Usually I only bother to take my own advice for complicated queries involving GROUP BY, subselects, etc. Tonight, though, I was working on optimizing some of our pages by moving data into summary tables, and a query took a lot longer than I expected:

mysql> select SQL_NO_CACHE s.title, ar.name from SongBuzz sb JOIN Song s ON sb.song_id = s.id JOIN Album al ON s.album_id = al.id JOIN Artist ar ON al.artist_id = ar.id ORDER BY sb.recs_1w desc limit 3;
+----------------------+------------------+
| title                | name             |
+----------------------+------------------+
| Deep Sea Green       | Mandyleigh Storm | 
|  Suffer For Fashion  | of Montreal      | 
| Just Enough          | Dan Tharp        | 
+----------------------+------------------+
3 rows in set (1.17 sec)

All of the foreign key fields in the JOIN were indexed, as was sb.recs_1w. There's no reason this query should take 1.2 seconds when the entire data fits easily in RAM. So I ran EXPLAIN:

mysql> EXPLAIN select SQL_NO_CACHE s.title, ar.name from SongBuzz sb JOIN Song s ON sb.song_id = s.id JOIN Album al ON s.album_id = al.id JOIN Artist ar ON al.artist_id = ar.id ORDER BY sb.recs_1w desc limit 3;
+----+-------------+-------+--------+-------------------+-----------+---------+-------------------------+-------+---------------------------------+
| id | select_type | table | type   | possible_keys     | key       | key_len | ref                     | rows  | Extra                           |
+----+-------------+-------+--------+-------------------+-----------+---------+-------------------------+-------+---------------------------------+
|  1 | SIMPLE      | ar    | ALL    | PRIMARY           | NULL      | NULL    | NULL                    | [redacted big number] | Using temporary; Using filesort | 
|  1 | SIMPLE      | al    | ref    | PRIMARY,artist_id | artist_id | 4       | amiest_production.ar.id |     1 | Using index                     | 
|  1 | SIMPLE      | s     | ref    | PRIMARY,album_id  | album_id  | 4       | amiest_production.al.id |     4 |                                 | 
|  1 | SIMPLE      | sb    | eq_ref | PRIMARY           | PRIMARY   | 4       | amiest_production.s.id  |     1 |                                 | 
+----+-------------+-------+--------+-------------------+-----------+---------+-------------------------+-------+---------------------------------+
4 rows in set (0.02 sec)

From this query plan, it was clear that MySQL was first reading the artist table, then joining to find all the albums, then joining to find all the songs, then joining against the SongBuzz summary table. After getting all these rows, it was sorting the whole thing in a temporary table. The more logical plan would be to sort the SongBuzz table by recs_1w, since that column is already indexed, grab the top 3 rows, and then join through to find the song and artist names. Using the STRAIGHT_JOIN flag to the SELECT statement, we can force this:

mysql> explain select SQL_NO_CACHE STRAIGHT_JOIN s.title, ar.name from SongBuzz sb JOIN Song s ON sb.song_id = s.id JOIN Album al ON s.album_id = al.id JOIN Artist ar ON al.artist_id = ar.id ORDER BY sb.recs_1w desc limit 10;
+----+-------------+-------+--------+-------------------+---------+---------+--------------------------------+--------+-------------+
| id | select_type | table | type   | possible_keys     | key     | key_len | ref                            | rows   | Extra       |
+----+-------------+-------+--------+-------------------+---------+---------+--------------------------------+--------+-------------+
|  1 | SIMPLE      | sb    | index  | PRIMARY           | recs_1w | 2       | NULL                           | [redacted big number] | Using index | 
|  1 | SIMPLE      | s     | eq_ref | PRIMARY,album_id  | PRIMARY | 4       | amiest_production.sb.song_id   |      1 |             | 
|  1 | SIMPLE      | al    | eq_ref | PRIMARY,artist_id | PRIMARY | 4       | amiest_production.s.album_id   |      1 |             | 
|  1 | SIMPLE      | ar    | eq_ref | PRIMARY           | PRIMARY | 4       | amiest_production.al.artist_id |      1 |             | 
+----+-------------+-------+--------+-------------------+---------+---------+--------------------------------+--------+-------------+

and the resulting query executes somewhere around 60x faster:

mysql> select SQL_NO_CACHE STRAIGHT_JOIN s.title, ar.name from SongBuzz sb JOIN Song s ON sb.song_id = s.id JOIN Album al ON s.album_id = al.id JOIN Artist ar ON al.artist_id = ar.id ORDER BY sb.recs_1w desc limit 3;
+----------------------+------------------+
| title                | name             |
+----------------------+------------------+
| Deep Sea Green       | Mandyleigh Storm | 
|  Suffer For Fashion  | of Montreal      | 
| Just Enough          | Dan Tharp        | 
+----------------------+------------------+
3 rows in set (0.02 sec)

More useful code for user_defaults.erl

xd on freenode #erlang posted a useful code snippet today (pasted below). This code adds several useful functions to the shell. If you're the original author of this, let me know so I can give you credit!

-module(user_default). 
-author('serge  hq.idt.net'). 

-export([help/0,dbgtc/1, dbgon/1, dbgon/2, 
dbgadd/1, dbgadd/2, dbgdel/1, dbgdel/2, dbgoff/0, lm/0, mm/0]).

-export([time/1]).

-import(io, [format/1]). 

help() -> 
    shell_default:help(), 
    format("** user extended commands **~n"), 
    format("dbgtc(File) -- use dbg:trace_client() to read data from File\n"), 
    format("dbgon(M) -- enable dbg tracer on all funs in module M\n"), 
    format("dbgon(M,Fun) -- enable dbg tracer for module M and function F\n"), 
    format("dbgon(M,File) -- enable dbg tracer for module M and log to File\n"), 
    format("dbgadd(M) -- enable call tracer for module M\n"), 
    format("dbgadd(M,F) -- enable call tracer for function M:F\n"), 
    format("dbgdel(M) -- disable call tracer for module M\n"), 
    format("dbgdel(M,F) -- disable call tracer for function M:F\n"), 
    format("dbgoff() -- disable dbg tracer (calls dbg:stop/0)\n"), 
    format("l() -- load all changed modules\n"), 
    format("nl() -- load all changed modules on all known nodes\n"), 
    format("mm() -- list modified modules\n"), 
    true. 

dbgtc(File) -> 
    Fun = fun({trace,_,call,{M,F,A}}, _) -> 
    io:format("call: ~w:~w~w~n", [M,F,A]); 
      ({trace,_,return_from,{M,F,A},R}, _) -> 
    io:format("retn: ~w:~w/~w -> ~w~n", [M,F,A,R]); 
      (A,B) -> 
    io:format("~w: ~w~n", [A,B]) 
   end, 
    dbg:trace_client(file, File, {Fun, []}). 

dbgon(Module) -> 
    case dbg:tracer() of 
        {ok,_} -> 
     dbg:p(all,call), 
     dbg:tpl(Module, [{'_',[],[{return_trace}]}]), 
     ok; 
 Else -> 
     Else 
    end. 

dbgon(Module, Fun) when is_atom(Fun) -> 
    {ok,_} = dbg:tracer(), 
    dbg:p(all,call), 
    dbg:tpl(Module, Fun, [{'_',[],[{return_trace}]}]), 
    ok; 

dbgon(Module, File) when is_list(File) -> 
    {ok,_} = dbg:tracer(file, dbg:trace_port(file, File)), 
    dbg:p(all,call), 
    dbg:tpl(Module, [{'_',[],[{return_trace}]}]), 
    ok. 

dbgadd(Module) -> 
    dbg:tpl(Module, [{'_',[],[{return_trace}]}]), 
    ok. 

dbgadd(Module, Fun) -> 
    dbg:tpl(Module, Fun, [{'_',[],[{return_trace}]}]), 
    ok. 

dbgdel(Module) -> 
    dbg:ctpl(Module), 
    ok. 

dbgdel(Module, Fun) -> 
    dbg:ctpl(Module, Fun), 
    ok. 

dbgoff() -> 
    dbg:stop(). 


lm() ->
    [c:l(M) || M <- mm()].

mm() ->
    modified_modules().

modified_modules() ->
    [M || {M, _} <- code:all_loaded(), module_modified(M) == true].

module_modified(Module) ->
    case code:is_loaded(Module) of
 {file, preloaded} ->
     false;
 {file, Path} ->
     CompileOpts = proplists:get_value(compile, Module:module_info()),
     CompileTime = proplists:get_value(time, CompileOpts),
     Src = proplists:get_value(source, CompileOpts),
     module_modified(Path, CompileTime, Src);
 _ ->
     false
    end.

module_modified(Path, PrevCompileTime, PrevSrc) ->
    case find_module_file(Path) of
 false ->
     false;
 ModPath ->
     case beam_lib:chunks(ModPath, ["CInf"]) of
  {ok, {_, [{_, CB}]}} ->
      CompileOpts = binary_to_term(CB),
      CompileTime = proplists:get_value(time, CompileOpts),
      Src = proplists:get_value(source, CompileOpts),
      not (CompileTime == PrevCompileTime) and (Src == PrevSrc);
  _ ->
      false
     end
    end.

find_module_file(Path) ->
    case file:read_file_info(Path) of
 {ok, _} ->
     Path;
 _ ->
%% may be the path was changed?
     case code:where_is_file(filename:basename(Path)) of
  non_existing ->
      false;
  NewPath ->
      NewPath
     end
 end.


time(F) when is_function(F) ->
    S = now(), Res = F(), E = now(), {timer:now_diff(E,S), Res}.

Weird Java exception XML parsing

For the last couple of weeks, I've been unable to start any of our Java-based Thrift services that require XML configuration on my development workstation. They've been dying with the following informative exception:


[java] Caused by: java.lang.UnsupportedOperationException: This parser does not support specification "null" version "null"
[java] at javax.xml.parsers.DocumentBuilderFactory.setSchema(DocumentBuilderFactory.java:489)
[java] at com.amiestreet.service.facebook.FacebookConfig.createDocumentBuilder(FacebookConfig.java:82)


The same code from the same tree and the same classpath works fine on our VM development systems and our production servers.

After a bit of a witch-hunt, with innocent bystanders such as OpenOffice.org killed along the way, I finally traced the problem to a file /usr/lib/jvm/java-1.5.0-sun/jre/lib/ext/xerces.jar. No idea how it got there, but figured I'd contribute to the google results for this problem and save some poor soul some aggravation down the road.

Alternative Erlang bindings for Thrift

At Amie Street we use Thrift, an open source project from Facebook, to offload various services from our PHP frontend to backends written in various other languages. Last week we released our first backend service written in Erlang, using the facebook-provided bindings.

At the beginning of this week we began to work on our backlog of improvements for this service. In working on these improvements, I realized that the Thrift bindings for Erlang were written as a fairly close translation of the bindings in object oriented languages, to the point of an OOP emulation layer (thrift_oop_server). This resulted in a lot of code that was both confusing for developers to read through and inefficient for the Erlang VM to execute.

Since Erlang is quick to write, I spent the last couple of days rewriting the bindings, and have the new version online in a git repository. Here's the list of notable new features:

• Improved idiomatic style - the whole thing is more "Erlangy" -- there are less gratuitous processes, and things are named thrift_binary_protocol rather than tBinaryProtocol


• Improved performance - as a result of ditching the OOP layer, the new bindings are at least 5x faster than the old.


• More robust code generation - the old version generated broken code from the ThriftTest.thrift file in the repository. The new one passes the tests.


• Bug fixes - for example, the old version did not properly serialize/unserialize doubles


• No unexpected crashing processes - the old thrift bindings crashed at the end of every connection when the client disconnected. The new version only crashes when something goes wrong
  

Hopefully we'll be able to get these new bindings merged upstream in the next couple of weeks. But if you use Erlang and Thrift, please check it out and try it on your project. And if you don't use Erlang and Thrift, it may be worth writing a quick server to see how useful it really is!

Reloading all code from the Erlang shell (take two)

So, it turns out that the snippet from earlier doesn't quite work right.

Thanks to #erlang on freenode, I found out that you can add commands to the shell by putting them in the user_default module. So here's the improved version:

In ~/erl/user_default.erl

-module(user_default).

-export([la/0]).

la() ->
    Modules = [M || {M, P} <- code:all_loaded(), is_list(P) andalso string:str(P, "amiest") > 0],
    [shell_default:l(M) || M <- Modules].

Then compile that module using c(user_default) from within ~/erl.

In your ~/.erlang:

code:load_abs("/home/amiest/erl/user_default").

Reloading all code from the Erlang shell

One of the reasons Erlang is pretty cool is that it supports hot code reloading. With OTP (Open Telecom Platform) applications, this can be done through a complicated mechanism involving release files, code_change callbacks to update state, and special upgrade/downgrade scripts.

In development, however, that's a bit of a nuisance. So, how do you simply reload all of the loaded modules after you've recompiled them? Here's a quick solution to paste into your ~/.erlang:

Reload = fun(M) ->
                 code:purge(M),
                 code:soft_purge(M),
                 {module, M} = code:load_file(M),
                 {ok, M}
         end.
 
ReloadAll =
 fun() ->
         Modules = [M || {M, P} <- code:all_loaded(), is_list(P) andalso string:str(P, "/home/amiest") > 0],
         [Reload(M) || M <- Modules]
 end.

Obviously, replace /home/amiest with a directory within which your modules reside. Then, simply run ReloadAll(). from the shell to load the newest versions of your modules.

Launching ASDoc with External Tools

While redoing the player, I have been writing a lot of actionscript lately. And by a lot I mean a ton. After I had everything working properly, the time came to go back through and clean it all up. In that process, I came across ASDoc, the documentation generator for actionscript. Like JavaDoc or PHPDocumentor, you mark up your code with multi-line comments, and after running the tool, you get a nice pretty set of HTML pages that make it understandable.

As usual, I dove right in. Marked up all of my methods and member variables and was ready to go. But, ASDoc has quite a few arguments to customize the output and using the shell to run this every time I made a change got old real fast.

Now enter External Tools from the Eclipse framework. If you haven't used external tools in the past, basically it allows you to configure and run mind numbing tasks click and go style. After digging through the flex mailing lists for a bit, I came across this. Problem solved.

But.... I had several projects I wanted to document and I didn't want a million of these external tool configurations all over the place. Another nice feature of External Tools is variable substitution on workspace and project levels. So, here is how to add the external tool to run asdoc on the currently open project:

1. Click Run-> External Tools -> Open External Tools Dialog
2. Click on the New Configuration button
3. Name it Generate ASDoc
4. For location, enter the path to asdoc. On windows, the default is C:\Program Files\Adobe\Flex Builder 3\sdks\3.0.0\bin\asdoc.exe
5. In the working directory, enter ${project_loc}
6. In the arguments input, -source-path . -doc-sources ./com -window-title "${project_name}" -main-title "${project_name}"
7. Click Apply

After you run "Generate ASDOC", you will get a new folder in your project called asdoc-oputput that contains the fruit of our labor. If you open up index.html, it will look something like this:




Pretty sexy, no?

About the Arguments

Here is a quick explanation of what the arguments in the above instructions do. You can view all of the available arguments here.

-source-path . Sets the working path for asdoc. Why not use ${project_loc} instead of setting the working directory? For some reason, external tools doesn't like spaces in its substitutions (even with proper escaping). Since the default workspace in flex builder contains spaces, this can be quite a pickle. Setting the working directory gets around this (thanks Jimmy!).

-doc-sources ./com Tells ASDoc to generate docs for everything inside ${project_loc}/com/. If you are documenting a Flex Library Project, you can keep the code as is. Note: If you are trying to document a normal Flex Project (ie your code is in ${project_loc}/src/), change the arguments appropriately.

-window-title "${project_name}" -main-title "${project_name}" Sets the HTML title and any other title references to the name of your project.

fcsh-mode for flex work from (x)emacs

A week or so ago I finally bit the bullet and decided I was going to have to learn some Flex/ActionScript so I could be self-sufficient on my latest project, a music game that integrates a Flash frontend with an Erlang backend. Being a Linux user (at work, at least) I downloaded the Flex SDK and was pleased to find that I could compile my Flex application using mxmlc from the command line. However, it was ridiculously slow -- we're talking 25+ seconds to compile a 500 line mxml file.

It turns out the reason for this is that mxmlc is written in Java and has significant startup cost. The solution is to use fcsh, the Flex Compiler Shell, which runs continuously and can incrementally compile your Flex application as you make changes. Using this from my shell reduced compilation time to 1-2 seconds, which is perfectly acceptable.

However, I quickly noticed that fcsh sucks. Most notably, it has no command history. I was typing "compile 1" every 15 seconds and getting annoyed. So I buckled down and wrote an fcsh-mode for Xemacs, reproduced below.

If you want to try it, paste it into a file like ~/.xemacs/fcsh-mode.el and load it using M-x load-file or adding (load-file "~/.xemacs/fcsh-mode.el") to your emacs init file. Then M-x fcsh will launch an fcsh with command history (even cross-session) in a new buffer. I also added an fcsh-repeat-last command which I bind to C-Enter using M-x local-set-key in my editing buffer. I can then just hit control-enter to recompile my flex app in less than a second.

Hope someone finds this useful!


(require 'comint)

(defvar fcsh-mode-map
(let ((fcsh-mode-map (copy-keymap comint-mode-map)))
(define-key fcsh-mode-map [(up)] 'comint-previous-input)
(define-key fcsh-mode-map [(down)] 'comint-next-input)
fcsh-mode-map
)
"Keymap for fcsh major mode")

(defvar fcsh-mode-hook nil
"Functions to run when fcsh mode is actived.")

(defvar fcsh-input-ring-file-name "~/.fcsh_history"
"*When non-nil, file name used to store Fcsh shell history information.")


(defun fcsh-mode ()
"Major mode for running the fcsh, flex compiler shell"
(interactive)
(comint-mode)
(setq major-mode 'fcsh-mode)
(setq mode-name "FCSH")
(use-local-map fcsh-mode-map)

(setq comint-prompt-regexp "^\\(fcsh\\) ")
(setq comint-eol-on-send t)
(setq comint-input-ignoredups t)
(setq comint-scroll-show-maximum-output t)
(setq comint-scroll-to-bottom-on-output t)

;; Some older versions of comint don't have an input ring.
(if (fboundp 'comint-read-input-ring)
(progn
(setq comint-input-ring-file-name fcsh-input-ring-file-name)
(comint-read-input-ring t)
(make-local-variable 'kill-buffer-hook)
(add-hook 'kill-buffer-hook 'comint-write-input-ring)))

(run-hooks 'fcsh-mode-hook))

(defun fcsh ()
"Run an inferior fcsh, with I/O through buffer *fcsh*.
If buffer exists but fcsh process is not running, make new process.
If buffer exists and fcsh process is running, just switch to *fcsh*.
The buffer is put in fcsh-mode.

\(Type \\[describe-mode] in the fcsh buffer for a list of commands.)"
(interactive)
(cond ((not (comint-check-proc "*fcsh*"))
(set-buffer (make-comint "fcsh" "fcsh"))
(fcsh-mode)))
(pop-to-buffer "*fcsh*"))


(defun fcsh-command (cmd)
"Run a command in the fcsh shell"
(interactive "scommand?")
(let ((old-buffer (current-buffer)))
(cond ((not (comint-check-proc "*fcsh*"))
(set-buffer (make-comint "fcsh" "fcsh"))
(fcsh-mode)))
(set-buffer "*fcsh*")
(goto-char (marker-position (process-mark (get-buffer-process (current-buffer)))))
(insert cmd)
(comint-send-input)
(switch-to-buffer old-buffer)))

(defun fcsh-repeat-last ()
"Repeat last command in fcsh"
(interactive)
(let ((old-buffer (current-buffer)))
(cond ((not (comint-check-proc "*fcsh*"))
(set-buffer (make-comint "fcsh" "fcsh"))
(fcsh-mode)))
(set-buffer "*fcsh*")
(goto-char (marker-position (process-mark (get-buffer-process (current-buffer)))))
(call-interactively 'comint-previous-input)
(comint-send-input)
(switch-to-buffer old-buffer)))