Twitter Launched New Backend for Search

Twitter quietly flipped the switch on entirely new backend for their search, revealed in a blog post; "One of our main goals, but also biggest challenges, was a smooth switch from the old architecture to new one, without any downtime or inconsistencies in search results." "Scaling the old MySQL-based system had become increasingly challenging," they […]

Twitter quietly flipped the switch on entirely new backend for their search, revealed in a blog post; "One of our main goals, but also biggest challenges, was a smooth switch from the old architecture to new one, without any downtime or inconsistencies in search results." "Scaling the old MySQL-based system had become increasingly challenging," they note. So what's new? "Since we love Open Source here at Twitter we chose "Lucene," a search engine library written in Java, as a starting point."But they modify it give their demands for real-time search."

What type of demands? "Our demands on new system are immense: With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines. As we want the new system to last for several years, the goal was to support at least an order of magnitude more load."

They also have a goal of indexing a tweet after it's tweeted in less than 10 seconds.

[Source]