TIG’s Lair

So you need to provide a count or counter for your web application, but you want it to scale. The naive approach would be to simply select count(*) from table. That will fail under load because it requires scanning your entire collection.

Semergence » Blog Archive » Scalable Counters for Web Applications

With MACHO having 73 million objects, and 130 million+ light curves on a machine with 16gb of memory and objects and light curves table being 7gb and 13 gb respectively, I deal with this issue every day. Partitions definitely the way to go…

And when it comes to buying into such data services (be they software, a platform, storage or compute time), there are certain questions that need to asked

Out of Cloud Chaos Comes Structure - GigaOM

I made some comments on the importance of cloud computing in science and the issues we must deal with over there on GigaOm at http://gigaom.com/2008/06/30/out-of-cloud-chaos-comes-structure/#comment-886452