Meh, it's fairly inevitable. Systems are only getting bigger and more competent. This kind of stuff requires resources and some solid thinking about algorithms and data structures. No doubt they could publish some amazing papers. But I don't suspect they have revolutionary breakthroughs ala the NSA with crypto.
I've written a small-scale system for VoIP calls, to archive and index all network traffic - it's amazingly useful for debugging. Several years back, on a single quad-core machine with 1 disk, I was able to handle 5TB (several billion messages) of signalling data per day - indexing, archiving, searching. I'm nothing amazing. (I went to SF thinking I could somehow turn this indexing system into amazing profit. First guy I showed says "oh yeah I recognize this, go get <some intro db textbook>". Oops, so much for breakthrough ideas.)
A team of me-equivalents (with some that _have_ gone to school) and a nice budget could end up designing something like these spy systems. They aren't the Manhattan Project requiring new knowledge no one in the world has. I'm no 3-sigma intellect. There are dozens of millions of people more intelligent than me. Complaining about it and shaming engineers will not work against such critical projects.
> But I don't suspect they have revolutionary breakthroughs ala the NSA with crypto.
I wouldn't be so sure - GCHQ recruits heavily from Cambridge and Oxford, and I once met a Mathematics PhD who told me in the UK his two main options for employment were academia or the service. They have a lot of crypto talent there and they're not working on the big data systems talked about here.
Sorry for poorly made comment. I meant to say that creating a data collection system won't require such breakthroughs. The GCHQ obviously has tons of amazingly bright people. I'm only talking about a data-collection system, and that it's not too hard, given the resources, to create.
I've written a small-scale system for VoIP calls, to archive and index all network traffic - it's amazingly useful for debugging. Several years back, on a single quad-core machine with 1 disk, I was able to handle 5TB (several billion messages) of signalling data per day - indexing, archiving, searching. I'm nothing amazing. (I went to SF thinking I could somehow turn this indexing system into amazing profit. First guy I showed says "oh yeah I recognize this, go get <some intro db textbook>". Oops, so much for breakthrough ideas.)
A team of me-equivalents (with some that _have_ gone to school) and a nice budget could end up designing something like these spy systems. They aren't the Manhattan Project requiring new knowledge no one in the world has. I'm no 3-sigma intellect. There are dozens of millions of people more intelligent than me. Complaining about it and shaming engineers will not work against such critical projects.