Yahoo Pipes

Nov 21, 2009

Yahoo Pipes is a really awesome product that a lot of people don’t know about, and I feel like Yahoo hasn’t really done a lot with. It’s basically a node-based data editor, but it can fetch source data from places on the Internet, which means you can have pipes with dynamic output. And then, you can subscribe to the results of those pipes or request them via JSON, which gives you a huge amount of flexibility.

I was spurred to write this because Jason Kottke was recently upset that the New York Times had put Errol Morris’ blog items, which used to be on their own, into a new thing called Opinionator. This is a great simple use-case for Pipes, since you can fetch data from an RSS feed. Here’s the final result:

Pipes looks at the RSS feed and sees that there’s a dc:creator tag in it, so we can use that to filter with. It knows the input is RSS items, so you can filter those on an item-by-item basis and it will retain just the items that match. It’s really easy to understand exactly what it’s doing and build your own pipes. The only problem is sometimes finding the operations you’re looking for, since there are a lot of them.

Go ahead and check it out in action. The New York Times Opinionator feed only has 10 items in it, and there are several authors, so this might not have any results at any given time if you go to look at it, but you can clone it and play with the source yourself. I’ve also noticed that Pipes has been a little flaky when playing with it this morning, so apologies if it doesn’t work.