Build robust abstraction around WatchIndex

A number of processes use WatchIndex to observe and react to changes in the entity server. But WatchIndex can stop at any point and a number of places in our code base we don't re-watch. This has created a number of weird, latent timebombs.

Rather than try to play whack-a-mole with them all, we should build a client side abstraction around WatchIndex that has the right behavior AND a reconcilation for the time between Watch requests.

The API should allow Go code to request updates for a given attribute based watch query, with the changed entities sent to a channel. Behind the scenes, the abstraction is using a goroutine to maintain the correct data, observe changes, and forward them on.