The Life and Times of the Watters Family

Don’t Throw The Baby Out With The Bathwater

Dave Winer comments on a recent Wired article that raises concerns about the scalability issues associated with aggregators polling RSS/ATOM feeds.

The article points out the most significant current issues (poorly written aggregators & questionable feed content strategies), but even fixing those items won’t solve the problem for long. While some are rightly exploring the concept of a new, distributed model for subscriptions, others are suggesting that aggregators simply enforce hard restrictions on polling intervals. Doing so would be an incredibly unfortunate and short-sighted approach as it would unnecessarily limit a number potential, albeit non-traditional, uses for RSS/ATOM.

Most of the current proposals for this limitation recommend a maximum polling interval of once per hour. Some even suggest a default of once per day. All of these suggestions, however, seem predicated on the assumption that RSS/ATOM is only useful for syndicating a very narrow group of content types. Personally, I can fathom a number of uses for RSS/ATOM technology that would appropriately poll at intervals of once per 10 or even 5 minutes. Such envisioned uses include server uptime/activity monitoring, stock quotes, sporting game progress updates, etc.

The “appropriate” polling interval is best determined as a function of the nature of the content, and not strictly the format in which it is being distributed. By enforcing such a limit at the aggregator, I am forced, as a user to do one of two things:

1. Write an aggregator of my own without the limitation, OR
2. Write a new format & a new aggregatorish application to process the new format

Neither solution is optimal. Especially when I can easily use the basic tools provided my aggregator of choice to separate and manage the information with minimal effort. My suggestion would be to add an optional TTL or MaxPollingInterval to the feed that would be respected by the aggregator. This would allow authors to limit the amount of polling on their site to that which they determined was reasonable.

As the core RSS/ATOM technologies grow, new models for subscription may well be necessary in order to allow them to scale. But please, don’t arbitrarily limit the potential uses of the technology by placing unnecessary restrictions in my aggregator.

---

---