Alex Le’s Blog

I just checked out the 280Slides.com, a YCombinator’s funded company. They are developing an online Slideshow/Powerpoint site (yes, another one). The application is pretty slick and it has the Apple’s look and feel to it (both the 2 founds were from Apple). Paul Graham used their app to create his slide at StartupSchool 2008, which at the time, took him 10 minutes to start his slides due to “technical issues”.

What really got me amazed is that they developed a JavaScript UI framework called Objective-J. Apple do a lot of their apps and gadgets in Objective-C (iPhone for one), and since the guys behind 280Slides were from Apple, probably they took the concepts from Objective C and bring it over to JavaScript. Having coded TubeCaption’s caption editor, the Captionizer, from scratch, I understand how much work it takes to do something non-trivial in JavaScript. Programming an online application is totally different than programming a simple “Ajaxy, Web2.0″ page because the amount of work involved. We have hundreds of objects potentially interact with one another and trying to compete for CPU. Without a solid foundation, the application won’t be able to bear the performance and complexity weight.

I’m pretty excited to hear that 280Slides is planning to open-source their framework (probably the Objective J) in the near future. It will be a fresh idea besides the currently established frameworks such as YUI, Ext, Prototype.

Increasingly I see the trend of JavaScript being used as the underlining cross-platform language to build other frameworks and programming languages on top of it. John Resig (from jQuery) recently ported the processing visualizing language to JavaScript. His JS implementation looks AMAZING (check out the parser’s code, what a work of art) and performance-wise the library kicks major ass. Then somebody wrote a Ruby VM in JavaScript (HotRuby) and Ruby code can get executed natively in the browser. A VM written in JavaScript? WOW! Just the thought of Rails *may* work in the browser (hopefully not IE6) makes me feel dreamy. These stuffs are truly innovative and that’s what really push the web technologies forward. And with the new JavaScript engines that promise excellent performance (Webkit’s Squirrelfish, Mozilla’s SpiderMonkey), for sure we will even see MORE of the creative innovations.

PS: don’t forget to check out www.tubecaption.com’s Captionizer, the first timeline-based caption editor.

Recently I had to configure my VPS at Slicehost.com to send out emails for my TubeCaption application. Due to the time constraint of the launch, we decided to outsource the mail delivery to Gmail SMTP server.

Using Gmail SMTP server with ActionMailer

When I first deployed the application, I didn’t configure sendmail correctly (I’m a total stranger to sendmail or postfix) and went ahead with Google’s SMPT server.  Since I am using Google Apps to host the mailboxes, I created an extra account specifically to use for mail delivery. In my development.rb and production.rb file, I have ActionMailer configured as follow:

ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
:tls => true,
:address => “smtp.gmail.com”,
:port => 587,
:domain => “tubecaption.com”,
:authentication => :plain,
:user_name => “sample-mailer-account@tubecaption.com”,
:password => “the-very-secured-password”
}

Within my ActionMailer handler, I have a function to setup the default params as follow

class Postoffice < ActionMailer::Base
def setup_default_params
@from = “my-email@tubecaption.com”
@headers = “Reply-to my-email@tubecaption.com”
@subject = “TubeCaption.com”
@sent_on = Time.now
@content_type = “text/html”
end
end

Then for each email, before sending, I invoke setup_default_params() to have all the params setup. The site could now sending out emails, albeit slow. ActionMailer had to open a connection to GMail, authenticated, and then started writing the content as a stream of text. The whole process took from 1 to 3, or even 4 seconds some time on my machines, which means the mongrel process was also stuck there waiting.  But the mailing piece worked. However, I came to know the real issue:  all the emails were marked as SPAM or JUNK if they are sent to GMail or Yahoo.

I talked to a friend of mine, who has more knowledge in the emailing world. He suggested the cause of this is because emails sent from my VPS failed the reversed DNS lookup verification. Yahoo mail servers perform a reverse DNS look up to check my domain’s DNS records, while the SMTP server I used was from Gmail. Hence the emails were marked as spam in Yahoo and junk in Gmail due to the DNS check mis-match.

So I decided that it was about time for me to configure sendmail for the box and switch ActionMailer to use sendmail instead.

ActionMailer with local sendmail

Truthfully speaking, I would love to get away as far as I can from configuring sendmail. The 4th edition O’Reilly book on Sendmail is more than 1,000 pages! I only want something that can send out emails… Luckily, there is yum.

# sudo yum install sendmail

Now I switch my ActionMailer config back to use sendmail. Since my sendmail is installed in the default folder, I don’t need to configure ActionMailer any further. In production.rb, I have

ActionMailer::Base.delivery_method = :sendmail

ActionMailer now delivered mails using the local sendmail program.

Using the console, I sent out a test email to my yahoo acount.   Everything works.  Then, I received a notice from my server, telling me I have new mails at /var/spool/mail. Great! I checked the mail log, and saw …

—– The following addresses had permanent fatal errors —–
<nworld3d@yahoo.com>
(reason: 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550])

—– Transcript of session follows —–
… while talking to b.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
… while talking to d.mx.mail.yahoo.com.:
<<< 553 Mail from XX.XXX.XXX.XXXnot allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
…….
<<< 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
554 5.0.0 Service unavailable

…….

Final-Recipient: RFC822; nworld3d@yahoo.com
Action: failed
Status: 5.5.0
Diagnostic-Code: SMTP; 553 Mail from XX.XXX.XXX.XXX not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/550-bl23.html [550]
Last-Attempt-Date: Tue, 3 Jun 2008 14:02:54 -0400

OUCH! My VPS got black-listed! Somehow the Spamhaus XBL list decided to black list the IP address of the server. Probably because I had port 25 opened in the firewall and somebody has taken advantage of my lack-of-linux-sysadmin-skills to start relaying spams.  CRAP! Yahoo outrightly refused any SMTP connection from my server because it was marked as a spammer-wannabe .  Good thing Yahoo included the  link to their help page and references to Spamhaus XBL.

I went to Spamhaus website and looked up my server’s IP:

The CBL (composite blocking list) was blocking my server.  I went ahead and request a removal from the list.  Afterwards, the CBL list showed

A few minutes later, I checked Spamhaus and it’s not showing the IP listed anymore.  Phew!

It would take a while before this list is propagated through out the internet.  At the moment, I still can’t connect to Yahoo’s SMTP server and my mails is still delivered right into Google’s Junk mailbox.  Hopefully a few hours more and my server would be good to go.

I don’t know if Spamhaus is my friend or foe.  I have a hard time deciding it.  I don’t know what people’s experiences with emailing services in general, but I can tell it’s a pain in the butt, and everything will only get worse from here.  I wonder what it will be like when ipv6 is used…

I was implementing cache_fu for my upcoming application based on YouTube (will be premiered very soon here on my blog). I wanted to cache certain calls to the YouTube API to reduce the latency and the time mongrels got stuck waiting for the response. The first candidate to get cached was the “Top Rated Video” from YouTube. Since this section is a part of the homepage, I used fragment caching for it. Everything went fine, Memcached just ran so beautifully and the speed gain was exceptional. I added the :ttl option to the cache command to expire the contents, then suddenly nothing worked. Rails kept on whining for the cache helper call:

wrong number of arguments (2 for 1)

I checked and double-checked cache_fu source, and the fragment_cache.rb file in particular. I saw cache_fu extending the fragment cache helper with an extra hash argument to support the :ttl option. Why the hell I could not use it in my view?

It turned out that the fragment_cache overwrite in cache_fu was not called automatically. I had to manually added the call in environment.rb file to invoke the setup.  Strange! It took me a while to figure out since I traced all source files in cache_fu to find the bug. I have to admit, cache_fu (previously known as acts_as_cached) has some kick-ass Ruby code written. I still need to learn a lot more about Ruby to get to that level of code craftsmanship.

To fix this issue, include this one line in your environment.rb file

ActsAsCached::FragmentCache.setup!

That will ensure the cache helper to have the :ttl option.

This is a quick summary for this process so that I can refer to it later on, and hopefully someone will find it useful as well.

Memcached requires libevent to handle its network IO stuff. The bundled libevent in the standard yum repository is old so it’s pretty useless. The newer versions memcached runs on newer libevent library so I ended up compiling libevent and memcached from the latest stable sources. I’m using libevent-1.4.4-stable and memcached-1.2.5.

First off, uninstall the libevent that yum may have installed on your machine

# sudo yum remove libevent

Download the sources for libevent and memcached , unzip( # gunzip *.gz ), untar (# tar -xvf *.tar), CD to the libevent folder. We will compile the libevent first.

# ./configure –prefix=/usr/local

# make

# make install

Basically we are telling libevent to install itself under /usr/local/lib/. When we compile memcached, we need to point it to the correct location as well. Once libevent is done installing (it’s really quick), we can move on and complie memcached.

CD to the un-tar memcached folder,

# ./configure –with-lib-event=/usr/local/

# make

# make install

After memcached is installed, you can try

# memcached

In my situation, I ran into an error

error while loading shared libraries: libevent-1.4.so.2: cannot open shared object file: No such file or directory

It turned out that the new libevent get installed, it doesn’t “register” the actual library file (similar to DLL on Windows) with the system. When Memcached runs, it tries to look for the libevent-1.4.so.2 file but since libevent is still playing hide and seek somewhere, memcached cries.

To fix this, we need to manually load the libevent library file into the system via the ld configuration. From the man page of ld:

ld combines a number of object and archive files, relocates their data and ties up symbol references. Usually the last step in compiling a program is to run ld.

I like to think ld as the regsrv32 used to register DLL’s on Windows. Now to fix up the reference to the libevent so file, we need to create a file under /etc/ld.so.conf.d/

# vi /etc/ld.so.conf.d/libevent-i386.conf

then enter

/usr/local/lib/

Write and quit (:wq!)

The path in the libevent-i386.conf is the path where the actual .so files are located at. We set this path when we run the ./configure –prefix=/usr/local/ during the libevent compilation. Reloading the ld configuration with

# ldconfig

now, you can start memcached in verbose mode (-vv) for testing

# memcached -vv

If you see something like ..

slab class 1: chunk size 104 perslab 10082
slab class 2: chunk size 136 perslab 7710
slab class 3: chunk size 176 perslab 5957
slab class 4: chunk size 224 perslab 4681
slab class 5: chunk size 280 perslab 3744

….

slab class 37: chunk size 367192 perslab 2
slab class 38: chunk size 458992 perslab 2
<6 server listening
<7 send buffer was 126976, now 268435456
<7 server listening (udp)

Congratulations! Memcached is up and running!

PS:  I’m renting the VPS from www.slicehost.com and so far my experience with them ( 1.5 months) is excellent.