1. DLP Flash Christmas Competition + Writing Marathon 2024!

    Competition topic: Magical New Year!

    Marathon goal? Crank out words!

    Check the marathon thread or competition thread for details.

    Dismiss Notice
  2. Hi there, Guest

    Only registered users can really experience what DLP has to offer. Many forums are only accessible if you have an account. Why don't you register?
    Dismiss Notice
  3. Introducing for your Perusing Pleasure

    New Thread Thursday
    +
    Shit Post Sunday

    READ ME
    Dismiss Notice

Recommendation Engine for FF.net

Discussion in 'Fanfic Discussion' started by JordanL, Nov 24, 2010.

  1. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    BTW all, I just greatly improved the reliability of my script to suggest a user. :)

    Previously it was constantly failing, throwing up error messages.

    The problem was this: fanfiction.net was blocking about 90% of the requests my server was making. Hilarious, IMO. My server has made, at most 2,500 requests in the last two days. That's child's play, even for a baby hosting account.

    Regardless, I simply wrote in a randomizer that now uses one of seven proxy servers to send the request. All this means that you guys should actually be able to add people to the database much easier now. :)
     
  2. Sesc

    Sesc Slytherin at Heart Moderator

    Joined:
    Dec 20, 2007
    Messages:
    6,216
    Gender:
    Male
    Location:
    Blocksberg, Germany
    I'm playing with this and cackling like mad. Great work, OP. I always love a new gimmick, and this is actually useful too. Well, that is, I've read 95% of everything the script returns for my own stuff, but this only means your tool is working :p


    Edit: Question though, does it matter what your own favourites are? I'm just asking because, sometimes the list almost reads like my own favourite list o_O
     
    Last edited: Nov 25, 2010
  3. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    No, all that matters is the story you are using as the seed.

    It takes that story, and then looks at people who have that story on their favorites. After it gets a list of people who have your seed story on their favorites, it then goes and looks at ALL of each of their favorites, and finds ones that a lot of people have on their list.

    The number of people who have your seed story on their list is considered the 100%. If every single one of them also have the same story, that story would score 100%.

    When it says "53.4% of people...", what it means is that 53.4% of the people in the data base who had your seed story on their favorites list, also had the suggested story on their favorites list.

    It's a wisdom of the crowds thing.

    If your profile has been submitted to the database, then yes your profile will contribute to anything you search that's also on your favorites list. But it won't be more or less important than any other profile in the database that matches your seed story.

    If you're really looking for something new, try adding a second seed story in the story id 2 field, or try adding something into the exclusion field, or try putting it on stories with less reviews. :)

    Lots of options.
     
  4. Perspicacity

    Perspicacity Destroyer of Worlds ~ Prestige ~ DLP Supporter

    Joined:
    Nov 27, 2007
    Messages:
    1,022
    Location:
    Where idiots are not legally permitted to vote
    High Score:
    3,994
    Added my own profile. Fun toy. The more I play with it, the more I'm convinced you're the New Guy of the Year. (And next year too, probably).
     
  5. pdo91

    pdo91 Professor DLP Supporter

    Joined:
    Jan 11, 2009
    Messages:
    495
    Location:
    Colorado
    This is awesome.

    I've gotten slightly better results when I search using two stories from different fandoms (using Forward [Firefly] and The Sun Soul [Pokemon] got good results - 100% of the people who matched the criteria also favorited Houses of the Holy, 66.67% also favorited Renegade Cause, The Lie I've Lived, The Denarian Knight, and The Song of the Trees). Nothing I haven't read before, mind, but it's got a wider spread. And of course, that will change when there are more than 1,500 users to be searched.

    JordanL, I would definitely support trying to work with FF.net on this. It would be beyond amazing if this referenced every single user over there.

    As far as suggestions, I'd make it more user friendly (which I'm fairly sure you've thought of doing if you plan to expand this like you say). A Home button, links to the Stats page, etc. On the topic of the Stats page, it'd be cool to see the top 15 stories, as well as fandoms and authors.

    Something that I think would be really helpful would be some way to filter for percentage favorited. While the stories that 66.67% like are definitely awesome, anyone who's been around long enough has read them all. It's similar to searching for stories with low reviews, but less 'unknown' and more 'controversial'. That's probably tricky, since it has to calculate it for every search, but it would be cool.

    Edit: After the 10th page or so, the list of fics went from 'been there, read it' to 'been there, read most of it'. So maybe just give us a way to jump to pages?
     
    Last edited: Nov 25, 2010
  6. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    Indeed. I'm nearly salivating at the possibilities with a larger user base given how well it's worked so far.

    It's possible, but I'm wary of approaching them. The much more likely scenario is that they try and pressure me with legal threats to take down what I have and then spend another two years duplicating it.

    Top 15 stories how? By number of favorites? I already have quite an array of stats of authors and fandoms on the stats page.

    Yeah, as I said earlier I identified this as a way to keep results relevant early on, but the tax on the server would be quite monstrous. I'd probably need a dedicated server before I truly entertained this idea, which runs in the vicinity of $100/month.

    That's certainly possible...
     
  7. pdo91

    pdo91 Professor DLP Supporter

    Joined:
    Jan 11, 2009
    Messages:
    495
    Location:
    Colorado
    Unfortunately, I have no problem imagining that happening. Shit sucks.

    The categories you have for authors and fandoms are pretty neat. The most reviewed story is that Twilight monstrosity, but what about the story on the most Favorites lists?
     
  8. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    Alright, implemented down at the bottom of the page. (And yes, it works by fandom as well.)

    BTW, does anyone know any other forums frequented by fanfiction readers/writers? I'm trying to get a broader base of what essentially are beta testers.

    EDIT: I'm afraid the stats aren't too terribly interesting at the moment though. All 15 of the top stories by favorites are Naruto stories (by virtue of the profiles that have been added so far), and every one of the stories by reviews are Twit-light.
     
  9. pdo91

    pdo91 Professor DLP Supporter

    Joined:
    Jan 11, 2009
    Messages:
    495
    Location:
    Colorado
    Dear god. The top 15 most reviewed stories are Twilight. As in, EVERY SINGLE ONE OF THEM IS TWILIGHT.

    :wall:

    How could we let this happen?
     
  10. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    Well, I have no way of knowing how accurate that is. This is just from the ~150,000 stories that have been scraped.

    EDIT:

    On a sidenote... I think it's being used pretty heavily now... in the last 12 hours I've had at least 5 brand new readers favorite/alert for my HP story, and I can't see how else they would have found it.
     
    Last edited: Nov 25, 2010
  11. b0b3rt

    b0b3rt Backtraced

    Joined:
    Sep 28, 2010
    Messages:
    252
    Because our most reviewed story is something like 12k reviews and theirs is over 56k (despite having been posted in 09 -.-)? Well, chances are after another 6 months or so MoR will be the top-reviewed HP fic and probably get on that list...
     
  12. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    If anyone here is feeling helpful, and has the time to waste, I've been simply going through review pages for various fandoms and submitting people who've given signed reviews to the database by right clicking and copying their profile URL.

    I nearly doubled the number of A:TLA stories in 10 minutes doing this.
     
  13. Beonid

    Beonid Seventh Year DLP Supporter

    Joined:
    Jul 12, 2008
    Messages:
    201
    Location:
    Melbourne
    I'll get down to it.

    How about a sort feature, like the one used for sorting favourites on FFN? I'd like to be able to sort by word count and update date, if at all possible.
     
  14. Klackerz

    Klackerz Bridgeburner

    Joined:
    Oct 22, 2009
    Messages:
    786
    Location:
    India
    This is awesome. I just got 2 new stories to read.

    Why can't you add the users automatically from the favorite list. Like if you add a user, automatically add all his favorite authors into the database.

    Also can you list the category in which a particular story belong in the Websites Stat Page
     
  15. NoxedSalvation

    NoxedSalvation Temporarily Banhammered

    Joined:
    Jun 24, 2010
    Messages:
    893
    Location:
    Germany
    This is a cool tool, I like it very much. Found me something new to read right away! Thanks for the hard work and keep improving the features please!
     
  16. Ayreon

    Ayreon Unspeakable DLP Supporter

    Joined:
    Sep 18, 2006
    Messages:
    764
    Location:
    Germany

    Let me make a guess.
    The readership of Twilight stories is from the demographic that has grown up with social networking sites and is used to boosting their own and their friends stats as much as possible, to appear popular.
    So you now have many reader who leave a "Great Chapter!" review for every single chapter.

    So in a 50 chapter story, you need just 100 people doing this and you've doubled your review count.
    Maybe they should only count the number of unique reviewers to get a better picture.
     
  17. Ceebee

    Ceebee High Inquisitor

    Joined:
    May 5, 2009
    Messages:
    548
    I both love this tool and despair at what it tells us.

    http://www.fanfiction.net/u/1265079/

    Five point 2 MILLION words. 160 stories. 100% slash. Doesn't that shit get old?

    Jesus.
     
  18. wordhammer

    wordhammer Dark Lord DLP Supporter

    Joined:
    Feb 11, 2010
    Messages:
    1,918
    Gender:
    Male
    Location:
    In the wood room, somewhere flat

    Spacebattles
    would find this interesting, and they have a wider focus for fanfiction than DLP.

    You might try some of the Yahoo Groups, but I'm only a aware of a few specific to HP to recommend. Also, a number of them require you to be accepted by the owner before you can post:
    Caer Azkaban - I mention this one as it has > 10,000 members
     
  19. Khazad-Dumb

    Khazad-Dumb Loves the Gay Porn DLP Supporter

    Joined:
    Feb 28, 2008
    Messages:
    1,419
    Location:
    Clutch City, USA
    The FanFiction Forum apparently has a wide userbase. These are the guys who brought you Shinji 40k, so there are some gems over there.
     
  20. JordanL

    JordanL Third Year

    Joined:
    Jan 17, 2009
    Messages:
    98
    That's certainly possible, I just didn't think anyone would want that... you know, sort by relevance.

    I could, but it would make every addition very, very server intensive. I'm not sure how to correctly convey the varying levels of server load, but if I did something like that I'd probably get a phone call from my host, wondering if I was running a botnet or something.
     
Loading...