Facebook performance

I continue to be surprised by Facebook's performance problems. Not just that they're not improving, but that they're actually getting worse. Maybe that's to be expected when you've got a highly interactive system with millions of active users that's written in a scripting language (PHP). But just saying "It's written in PHP" isn't really enough - it could be that the PHP interpreter has performance problems, or the scripts themselves could be inefficient.

And that doesn't even begin to address what are essentially data center questions. There may be insufficient computing or network resources, what resources they have may be badly allocated or administered, etc. I once worked on a hierarchical storage management system and there were users complaining that it was taking as long as 1/2 hour to open a file. I turned on disk tracing (this was AIX on an RS6k) and found that namei() on files in one particular directory were causing a single disk head to skip all over the place. From there it was easy to determine that the directory in question contained an insane number of files and there were indirect indirect indirect indirect ... blocks to hold the directory. By simply reorganizing the filesystem - striping it across multiple platters and multiple spindles and relieving that one poor disk head from having to do all the work - I was able to speed up lookups by three orders of magnitude. That is to say, I was able to get lookups down to 1/1000 of what they had been taking without changing a single line of code or forcing the user to change their (unfortunate) behavior. I suspect that there are lots of instances of those sorts of problems lurking inside the Facebook data center and I'm not sure that the operating systems they're using are instrumented well enough to be able to isolate them.

But still, I don't think sending over a megabyte of scripts down to the user's browser has nothing to do with it. What a mess.