Hey all,

I'm a web developer by day, and I was becoming frustrated with navigating long lists in forums, and long filters on ff.net trying to find some new stuff.

So, I came up with an idea.

Lots of people favorite a story. As an author, you get to see people who have favorited one of your stories, but what if there was a tool that let you backreference favorites and then suggest stories based on people who have favorited similar stories?

For example: say I wanted to find suggestions for stories similar to my own fic... The Day Harry Potter Died. What if it would look at a list, see who had favorited THAT story, then gave me a list of the most common OTHER favorites among those people?

That's what I've been doing. Unfortunately, this is all unofficial, which means I don't have access to the ff.net database, which means I wrote a spider that went out and indexed over 1,500 users' favorites (over 370,000). This list is obviously far from complete, and while it will have fics from all fandoms, its Naruto/HP heavy as that's where I started.

So why am I posting this here?

I would like to get your feedback.

The tool can be found here... You will need to story id number for your reference story. Using my example, I'd put in the ID for The Day Harry Potter Died (5645536) into the box and hit go, which would give me this list.

Let me know what you think.

You can even add profiles for indexing here (when my server feels like cooperating). You can view stats here.

 

This is actually...pretty damn good. Nice work. It's like a small scale version of amazon's suggestions engine, which suprises me daily.

I got the best results with less famous stories using your tool, right now looking for something new. Entered Caliburn's Ambient, first 3 pages are mostly part of the DLP library.

 

Agreed. Wow!

Edit: Should I be bothered that most searches for my own stories turn up LessWrong or PerfectLionheart stories among the top matches?

 

This is pretty awesome! Nice job.

@Pers: I got LessWrong as well...it was the only Harry Potter story on the first page, as well.

 

Pers - I got the same thing, a lot of PerfectLionheart and LessWrong - although third on my list was Joe, so I'm guessing most of my reviewers have some modicum of taste.

Still, this is impressive stuff.

 

Part of that is the lack of information in the database. It's currently behaving a bit odd, but add in people who have your own stories favorited using this tool and the results will get better.

---------- Post automerged at 02:04 PM ---------- Previous post was at 02:02 PM ----------

Also, you can exclude readers who also have a PL story from being part of the results using the "Exclude" feature at the bottom of the search.

 

Heh, me too. (I have no idea who PerfectLionHeart is, though.)

I think a lot of popular stories will appear on every search, just because so many people have them favorited.

Nice piece of work, JordanL. You should ask if the ff.net folks would be interested in actually adding this to their site, or at least letting you have access to their DB (which would put less of a load on their servers than spidering the site).

A couple of small suggestions: if I want to filter by fandom, it's kind of annoying having to scroll through all the 'H's to find "Harry Potter." Might I suggest that by default, the filter could be set to highlight the fandom of the story being searched?

Also, unsurprisingly if I search for one of my stories, I tend to get my other stories as the top results. People might want stories by the same author included in their recs, but they could just look at the author's profile page for that, so I suggest a checkbox to allow automatically filtering out other stories by the same author from the results.

 

This is an excellent tool but results seem very similar; perhaps a larger sampling of favorite lists?

 

Done and done.

EDIT:

Of note: once a user profile has been added to the database, it gets refreshed for new favorites every week automatically.

 

Eh - I did some very specific searches, and if a story was listed on someone's favs, LessWrong appeared as well, even though both had no similarities.

The readerbase of LessWrong is so huge, that the story will probably appear on all lists, regardless of the story you submitted as reference.

 

I think a better idea (if at all feasible) would be to use user ids to find similar favorite lists and listing the differences.

 

There is a similar issue with most other fandoms, like in naruto "lord of the land of fire", "Perfect Lionheart", and "S'Tarkand" appear for nearly any result, and because the database was started from my favs, I will a lot as well.

I'm still exploring ways to remove "universal" stories. I've shied away from simply removing stories with X% of all users having it as a favorite, because of the strain such a query would put on the database. Most of the other filtering options have been added as ways to refine your search around these large stories.

---------- Post automerged at 03:38 PM ---------- Previous post was at 03:38 PM ----------

What do you mean by that b0b3rt?

 

Funny how my first search turned up one of your fics. Coincidence? I think not.

Seriously though, awesome work! I'll be using this to find some fics I might have overlooked soon, so thanks

 

That's because right now the database is mostly built from people who have my stuff favorited.

You can fix that by adding your own readers to the database here

 

@JordanL: You sir, are awesome.

This search engine is exactly what ffn lacks. Instead of spending hours searching for a new enjoyable story to read, and more often than not banging my head on my desk, I can simple cross-reference stories I did enjoy to read and start from there.

 

I think he means that you would do a search on say.. nonjon's ff.net account, and then it would look for other users that have favourited the same stories as nonjon, and then would return their username. This could potentially allow you to find a new author that may write stories that you like (given that their story preferences is an indication of how they may write).

 

If everyone on DLP adds their profile on this tool... this thing will be excellent. I've already found a few fics I'm gonna try reading and that don't happen often enough these days!

Thumbs up to you sir.

 

This is a fantastic idea!

Now all we need is a decent filter for the main listings that excludes anything to do with slash or a Harry/Draco or Harry/Snape pairing, and our browsing will be greatly improved.

 

Ichor: I stored the pairings when I scraped all the data, but in plain-text form. I can start displaying the pairings on the results page for people, but it would be... difficult to make a filter since I don't have access to the actual ff.net database.

h2o: That's what I'm hoping! Add as many profiles as you want, doesn't have to just be your own.

Ceebee: Hmm... I could look into something like that... although it could be a bit more database intensive. I might actually have to collect donation/use advertising and get a dedicated server if I go much further in the database area. We're already talking about a 5 table database with over 500,000 records and over 30 indexes...

Bukay: Thanks. I originally wrote this, literally, just for me. I was completely fed up with the trudge of finding new stories, and their hit-and-miss nature. So far, since I completed the major work on this, I've used it to find three stories that have gone directly into my own favorites, and left the author kind reviews. A tool like this on ff.net would be a boon for good authors just as much as it would be for readers.

 

Okay, I am impressed. It's a little flaky for me right now (probably high usage?), but I was just able to get a search through (that "past 5 searches" thing is nifty, btw) and I have found a potentially good fic to read already. Great work; I hope you'll be able to keep making it better.