So, I went about knocking up a very basic test version (try it on your iphone). Is buggy, basic and sub-optimal, but proves it could work nicely. Try zooming with the double finger pinch action!
I also tried the new webkit CSS transforms to allow free zooming and rotation, however they are buggy and slow on the iphone! Oh well! Anyway, maybe we can have google maps lite on the iphone?