News Aggregator 5 comments

28 Dec 2003 @ 19:55, by Flemming Funch I spent the evening making my own news aggregator, and remarkably I succeeded to my satisfaction. For any non-techies, a news aggregator is a program that sucks up data in RSS format, served by weblogs of various kinds, and presents it all in a uniform way. I've tried a bunch of different aggregators. What I liked best was Radio Userland, because it shows the feeds together, looking like a weblog, and it seemed to be able to mainly show me a flowing stream of new stuff. Which I liked, but I'd kind of like more options. But when my paid license expired, I hesitated to renew it, because I wasn't really using it for its weblog or other functions. So I tried a variety of other programs. FeedReader on Windows, which was nice. Except for that I don't like having to watch postings one at a time. I like the big overview. On Mac I then used Shrook for quite a while. It still had that 3-pane thing, and crashed every couple of hours, so it ended up not running most of the time. I tried installing NewsMonster, after its website made me feel kind of stupid, as it is so superior that it apparently can do everything, including a bunch of things I don't know what are. Except for that it couldn't find Java on my computer, and messed up some of the menus in Mozilla that it was supposed to integrate with. I installed Pears which runs in Python. Worked, but was a bit too simple. I installed AmphetaDesk, which required installing a whole bunch of Perl libraries first. And, now, I like the look of it. Quite a bit like Radio. But now there's again a bunch of things I'd want it to do that it doesn't do. So, I woke up late and thought that if I could make my own aggregator, and I could finish it so it was functional today, I'd go for it. I really have other things to do, but it is Sunday and christmas, so nobody would be missing me too much. Somewhat reluctantly I decided to look for a library that does the basic fetching of an RSS feed. My first thought was that I could just as well write that myself too, but that is the kind of arrogance that makes me often end up with projects full of features, but not quite finished, because I try to do it all myself. So I picked up the Magpie RSS library in PHP. Which seems simple enough, and I only needed some of its features. Now, I decided to set it all up on my server, as opposed on my local machine, so I can make the functionality available for other users of my weblog program, and so the feeds can be cached amongst those users. And what I wanted was to store feeds and postings in mysql, so they can be kept indefinitely, and to be able to keep track of which ones have been read and stuff like that. So that is what I set up. A cron job picks up all channels every hour, and figures out what are new or updated postings. And then some PHP pages show which feeds one is subscribed to, which are available from the pool that is already on the server, and allows addition of new feeds. And one can see them either one at a time, or mixed together. And I borrowed somewhat the look from AmphetaDesk. But then I added the ability to keep track of which items in each feed a given user has read, and which ones they've at all seen. Then it can avoid showing what has been marked as read, and it can mark new postings with a little NEW icon. And I made it so the postings can be grouped by feed or by date. And they can be sorted in various ways. And I made a way of saving interesting postings to a separate place before they scroll away. And I added in the 50 or so feeds that I normally watch. And this already works better for me than any of the other aggregators I've used. I'll tinker some more with it before I'll let anybody else use it. And there are a few more things I'd like to add. It should be able, of course, to pass a post on to my weblog program, if I want to quote it. I need some ways of searching through older postings. Some more options of viewing them. Like, headings only, short excerpts, with or without pictures, etc. Maybe a way of categorizing the saved postings. But this should do for today.

5 comments

---

New Civilization Network - Personal News Logs - Workgroup News Logs - Syndication