The reason this is done is to get rid of extra latency when processing user input. I guess detecting zoom gestures needs some time to distinguish them from normal clicks?
You're gambling that there are more people who would walk away due to latency, than there are who would walk away due to being unable to interact with your site at all.