Zeitgeist optimizations

The Zeitgeist team has been hard at work lately. We recently moved from Launchpad and Bzr to freedesktop.org and git, just in time for the 0.9 release. Since then, Seif and I have been hammering away at making a bunch of speed improvements and trying to scale the engine to be able to handle a few billion events just as fast as it can handle a few hundred thousand.

Our first experiment has been focusing on the sqlite indexes. This one index tweak appears to increase our benchmark speeds by almost 45%:

CREATE INDEX event_timestamp_subj_interp_subj_id_id
                   ON event(timestamp, subj_interpretation, subj_id, id)

And here’s the graph to prove it:

Shorter lines are better. Blue is the 0.9 release, and the orange is the tiny index change. All of our raw data is in this google spreadsheet, so feel free to have a look. I can’t guarantee that you’ll immediately understand the information, but I’m open to explain it to those interested.

The Zeitgeist team has been hard at work lately. We recently moved from Launchpad and Bzr to freedesktop.org and git, just in time for the 0.9 release. Since then, Seif and I have been hammering away at making a bunch of speed improvements and trying to scale the engine to be able to handle ...