Running an X (Xorg) server on your Raspberry Pi is frustrating. You can either use the fbdev or fbturbo driver which will give an un-accelerated 2D environment with swrast 3D (OpenGL) all beating your poor RPi’s CPU. Overclocking it will only help you so much which is a pity considering that there is another layer on the SoC that would be perfect for that but is now unused.
Enter the VideocoreIV (VC4) and Eric Anholt (formally of Intel, now of Broadcom), who are going to breath new life into the RPi. The idea is to offload the 2D rendering, via Glamor, to the VC4 with OpenGL calls. Since a OpenGL stack needs to exist, that means there will be a Direct Rendering Manager (DRM) Linux kernel module and Gallium/DRI module in Mesa.
This is happening now, here is the current status of support via the Piglit test-suite: skip 19102, fail 3866, pass 3146, crash 153, total 26267