1. DLP Flash Christmas Competition + Writing Marathon 2024!

    Competition topic: Magical New Year!

    Marathon goal? Crank out words!

    Check the marathon thread or competition thread for details.

    Dismiss Notice
  2. Hi there, Guest

    Only registered users can really experience what DLP has to offer. Many forums are only accessible if you have an account. Why don't you register?
    Dismiss Notice
  3. Introducing for your Perusing Pleasure

    New Thread Thursday
    +
    Shit Post Sunday

    READ ME
    Dismiss Notice

Scryer: Fanfiction Search Database (Alpha)

Discussion in 'Announcements' started by Lord Ravenclaw, Feb 17, 2014.

  1. SilverOtter

    SilverOtter Seventh Year

    Joined:
    Feb 20, 2011
    Messages:
    263
    Location:
    USA
    Now that Crossover searches are possible, would it be possible to do multiple crossover searches?

    There are 2 'Star Wars' and 1 'Star Wars: The Clone Wars' Fandoms and it would be nice if I could search all 3 of those at once
     
  2. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    I'm actually already working on this :) It requires a few changes with how queries are done currently, but I'm hoping to roll it out today or tomorrow.
     
  3. SilverOtter

    SilverOtter Seventh Year

    Joined:
    Feb 20, 2011
    Messages:
    263
    Location:
    USA
    Excellent! :D

    Is it possible to get the number of stories in each Fandom, like it shows on the fanfiction.net page? That's another thing I would like to see in the drop down menu.

    Here: https://www.fanfiction.net/crossovers/Harry-Potter/224/

    So in the drop down menu on Scryer, could you do like 'Star Wars (302)', instead of just 'Star Wars'?
     
  4. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    I've rolled out multi-crossover support. The counts are coming, but will take a bit more work.

    Additionally, old Scryer has been removed. https://scryer.darklordpotter.net/ is now Scryer 2.0.
     
  5. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Also done. It's sorted descending by count of fics.


    Gauging interest: I'm thinking of building a Chrome extension / FF extension that beaconed what you were reading back to Scryer. This way Scryer could keep bookmarks for you and tell you if you've already read something. How many would actually use it?

    I'd use it, so I'll probably build it regardless.

    There's a lot of other potential opt-in analytics that might be interesting. e.g. plotting reading speed per hour, and number of words read per week + stories read per week.
     
  6. Reign

    Reign Fourth Year

    Joined:
    Oct 27, 2008
    Messages:
    124
    Gender:
    Male
    I would be extremely interested in any analytics you could provide. There are many features I would love to see but in doing so it might turn Scryer into more of a pensieve. So I wont mention them.

    For the last few months I've started a project to record any Worm fan fiction I've read in a Google spreadsheet (Title/Link, word count, rating, last updated, date started, ability's, etc), as a trial before I do the same for Harry Potter. I've come across many annoying problems.

    One such problem is wasting time filling out a form for a story I only spent less than a minute reading, but with Scryer it could easily be saved, marked, blacklisted, etc so I would know not to interact with that story again.

    Any aggregated data you could get on a personal level or even as a group would be interesting and even fun to have. Being opt-in is just icing on the cake. If this is implemented it would be the first time I'd ever willing allow browser tracking.
     
  7. KGB

    KGB Headmaster

    Joined:
    Jan 18, 2011
    Messages:
    1,067
    Yes we need a blacklist/dislike feature. I have a script that hides authors/stories i've disliked. It saves a lot of time when searching for something to read.

    An integrated feature, that wouldn't produce blank page after a blank page would be even better.

    Not super sure about the analytics, but if you make it opt in I don't really have any problems.
     
  8. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Good idea. I've added that to the list, along with default search settings.

    You'll have to install the extension, which would be the "opt in".
     
  9. SilverOtter

    SilverOtter Seventh Year

    Joined:
    Feb 20, 2011
    Messages:
    263
    Location:
    USA
    The ability to login and have a tagging system where you could opt-in to viewing tags from every user, or a certain set of users who have similar tastes to you would be nice, but I think that would require a lot of work to do.
     
  10. FreakLord

    FreakLord Professor DLP Supporter

    Joined:
    Feb 13, 2011
    Messages:
    459
    This was what I once proposed to Raven. But it was more of a DLP sponsored tags.
     
  11. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Definitely on my TODO list. This feature will require some planning. Until traffic increases though, I'm not certain the value for effort is there yet.

    The viability of Scryer is really dependent on 1) getting more data and 2) building personalization / assistant-style features. Scryer has to be more than a search engine.

    --------------------------

    Minor update tonight, pairings (as much as FFN tells us, I haven't done any extra summary inference) are now shown in search results. Also squashed some various bugs.

    Edit: disabled pairings, there was an unfortunate bug in the parser that requires a reindex of FFN.
     
  12. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Pairing view (which characters the author claims are paired) has been reenabled. Additionally, sorting / searching by DLP scores has been fixed, which was broken by a bug in a new version of ElasticSearch.

    I've also brought back an older sorting mode broken in the past by another ElasticSearch upgrade called Popular and Recent. This mode ranks stories by update date and blends in reviews, follows, and favorites. It's most powerful when used with a more specific search (e.g. -slash + your favorite characters) or a smaller crossover category.

    Next up: 3 new fandoms (which ones? You decide.), user accounts + DLP account linkage to then lead to automatic FFN bookmarking via Chrome extension + rating/note taking.
     
  13. readerboy7

    readerboy7 Fourth Year

    Joined:
    Jan 11, 2013
    Messages:
    126
    Location:
    GMT + 12
    Would it be possible to have permantent searches? e.g. +fred, +george, -Mpreg, that sort of thing.

    As to new fandoms, I would prefer Naruto and Worm, although Worm doesn't really seem to be poular on FFnet, so...
    Whatever fandoms you think are best.

    Edit: I was browsing Scryer and I found that there was atleast one story by Perfect Lionheart on the list. Would it be possible to have blacklisted authors as well?
     
    Last edited: Nov 21, 2014
  14. Sauce Bauss

    Sauce Bauss Second Year ~ Prestige ~ DLP Supporter

    Joined:
    Apr 4, 2008
    Messages:
    61
    High Score:
    1411
    The problem with using Scryer to find good Worm fics is the assumption that there are good Worm fics to be found.
    There aren't any.
     
  15. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    After user accounts are implemented, I'm planning on adding saved searches, blacklisted authors, and default searches (your template for searches).
     
  16. Lium

    Lium First Year

    Joined:
    Jul 1, 2011
    Messages:
    43
    Location:
    The Greatest of Britains.
    Fandomwise, I'd like to see ATLA/Teen Titans.
     
  17. FreakLord

    FreakLord Professor DLP Supporter

    Joined:
    Feb 13, 2011
    Messages:
    459
  18. Zombie

    Zombie Black Philip Moderator DLP Supporter

    Joined:
    Apr 28, 2007
    Messages:
    6,036
    Naruto, Bleach I would like to see on the new Fandom. I can't think of a third. I have no third.
     
  19. Oz

    Oz For Zombie. Moderator DLP Supporter

    Joined:
    Jan 31, 2008
    Messages:
    9,027
    Gender:
    Female
    Location:
    Baile Átha Cliath
  20. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    I've deployed seven new fandoms and their respective crossovers into the search index.

    • Dresden Files
    • Fate/stay night
    • Naruto (still backfilling, ~50% done)
    • Stargate SG-1
    • Stargate Atlantis
    • Star Trek: TNG
    • Star Wars

    Crossovers have been loaded as well.

    Next up:
    • X-Men?
    • Doctor Who?
    • Dragon Age
    • ?

    I'll be rolling out new fandoms by request and popularity. Currently been looking at fandoms often crossed with HP.

    Edit:

    • Avatar: The Last Airbender
    • Legend of Korra
    • Halo
    • LOTR
    • Bleach
    • ASOIAF
    • Buffy
    • Mass Effect
    • Avengers
    • X-Men the Movie

    Are now all in progress backfilling. I'll enable access to them tomorrow once initial indexing has completed.
     
Loading...