Todd Pinkerton

George Palmer (3dogsbark.com) presented a great overview at Barcamp London on scaling a rails webapp. The slideshow is available here. I wasn't there when he presented this, but the slides themselves are impressive -- lots of pictures and you can follow along easily, even without any narration. He covers moving from one server to two, scaling the database, clustering, caching, and even a design for globally-distributed sites under heavy load. Most startup-scale web apps don't have traffic requiring this level of infrastructure, but it's nice to have a plan in place to get to the next level when the need arises. This slideshow can help lay the groundwork for such a plan, and maybe even help you (and your team) sleep better knowing you have a way to manage growth.

amazon has recently announced their Elastic Computing Cloud service. It gives users the ability to quickly create a virtual machine in amazon's datacenter, install whatever OS they want, and use it for whatever -- webserver, cracking crypto, etc. The idea is to make scaling your application easy as demand increases. Instead of buying/installing/maintaining additional servers, you can just pay for a few hours of EC2 time. For now, the EC2 service is in beta, and invite-only. You can bring up many identical machines quickly, using an AMI image you create.The catch is that you must persist all your data before taking down your virtual instance -- the filesystem it provides gets destroyed whenever you reboot or stop running the instance, so you need to save your data off to the S3 storage service. There is some up-front work to be done once you get an account and begin using the EC2, however. Amazon provides a small selection of basic disk images, but to do anything useful you'll need to install your own software. I wanted to try running Ruby on Rails in this environment, so I create an AMI with all the ...

In my last post, I talked about how hard it was to find good resources for newbies to learn database tuning, especially with the very popular MySQL database -- the free or commercial version. This post will talk you through the basics of database tuning. Ready? here we go: 1) hardware is important. Ideally, your mySQL server will be on its own dedicated box. Sharing it with Apache and the rest of your application may be fine starting off, but eventually you'll want a dedicated machine for it. This also helps establish the right infrastructure for scalability, so you can add more webserver boxes which all connect to your database machine. So of course, things like RAM (speed and amount of), disk (speed and size of), and CPU horsepower all matter. Get as much as you can. 2) get the right tools. To even begin to tune the performance of your database, you need to be able to measure it. So get some good tools like * mytop (monitor mysql queries and performance) * ab (apache benchmark, for load-testing apache) * top (cpu/memory performance, process management) * iostat (disk/cpu performance) and and learn how to use them. Test your database using whatever queries and scenarios your ...

These days, everybody claims to be building the next YouTube, the next Google, "the Flickr of ..." something. They talk about slick AJAX web2.0 interfaces, or the awesome features they are building, or how great the content is. But deep down, every one of these Web2.0-could-be site's operator has a dreadful, paralyzing fear of the S-word: Scalability. Sooner or later, any moderately successful web application will need support a large number of users, lots of requests, and consume tons of bandwidth. I wonder what percentage of those "google adsense whores" paychecks go to paying their bandwidth bills? There exists all over the net tips to improve your website performance -- from configuring Apache with Worker threads, to tuning your CSS -- but usually the biggest performance bottleneck in a web application is the database. I was having a chat last week with a web startup (that shall remain nameless) about how hard it is to get support for their MySQL database. MySQL is the most popular open-souce and free database, but they also sell a commercial version. This company was trying to buy the commercial product -- trying to give MySQL their money! -- and having trouble getting the support they wanted. ...