1. Hey Guest, welcome back to DLP

    As you can see, we've changed our look. We've migrated from vBulletin to the Xenforo forum system. There may be issues or missing functionality, if you find anything or have feedback, please check out the new Xenforo Migration Feedback forum.

    Our dark ("Dark Lord Potter") theme is under heavy development. We also have a light ("Light Lord Potter") theme for those happier with a light background and darker text.

    Dismiss Notice
Dismiss Notice
Hey Guest! Are you any good at cooking? Got a favourite recipe that you love to cook or bring out to impress that special someone? Why not share it! A new forum called The Burrow has opened and it's all about homemaking!

Scryer: Fanfiction Search Database (Alpha)

Discussion in 'Announcements' started by Lord Ravenclaw, Feb 17, 2014.

  1. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    Announcing the newest in fiction search, Scryer. Scryer searches stories from FFN in a more powerful way than can currently be achieved with FFN's interface, and does so in a mobile-friendly manner. Scryer supports full-text search on title, author, and summary as well as filtering functions for characters, categories, and other fields.

    Please ensure noscript is turned off before attempting use.

    http://scryer.darklordpotter.net/


    Known issues
    None at the moment.

    Please leave feedback and bug reports in this thread.



    Future work
    • Chrome extension for reading tracking and analytics [soon]
    • Collaborative tagging StackOverflow style
    • Multiple site search (e.g. FicWad, Portkey, AO3)
    • Deeper integration into DLP & DLP Mobile
    • FFN Alert/Fav import for a better 'Update Feed'
    • Daily update summary emails based on the above

    Additionally, if you're a programmer, join us in DLP IRC #programming -- Scryer is open-source software built on top of the DLP API, contributions welcome. Scryer is Ruby on Rails and the DLP API is written in Java, and interfaces with ElasticSearch.
     
  2. Grubdubdub

    Grubdubdub Supreme Mugwump

    Joined:
    Feb 22, 2008
    Messages:
    1,604
    Other features that would be great are WIP/Completed classification, and a last updated/first published filters. The DLP Score integration is by far the best feature, it really is great. Huge thumbs up.
     
  3. Hero of Stupidity

    Hero of Stupidity Villain of Sensibility Prestige DLP Supporter

    Joined:
    Oct 5, 2010
    Messages:
    336
    Gender:
    Male
    Location:
    Hungary
    High Score:
    3,172
    I second the WIP/Completed. Maybe like FF the search function should fold itself up after the search is completed. Also I personalyl like to read with black background so maybe a Background change option would be good.

    Awesome job.
     
  4. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    Already covered in future work, it's 'in progress' but requires a full re-crawl to implement. Should be ready soon.

    I like that, I'll see what I can do.

    I've been looking at ways of making this easier to use and to make design sense. It won't be a fold-up, but I have other plans.

    You can use stylish if you want, but adding theme support to a search engine is really really far down my priorities list.
     
  5. FreakLord

    FreakLord Professor DLP Supporter

    Joined:
    Feb 13, 2011
    Messages:
    449
  6. Photon

    Photon Order Member

    Joined:
    Jul 15, 2012
    Messages:
    838
    Location:
    Poland
    What is indexed? All stories in Harry Potter category?
     
  7. Grubdubdub

    Grubdubdub Supreme Mugwump

    Joined:
    Feb 22, 2008
    Messages:
    1,604
    Running a search with no parameters shows that there are 520,000 stories indexed, about 150,000 less than ffn. Maybe the General genre stories make the difference?

    The Lucene Querey Syntax is working very well as far as I tested. Might be nice to add checkboxes for popular searches, such as Azkaban, Veela, -dumbles, Triwizard, and what not.
     
  8. Andrela

    Andrela Plot Bunny DLP Supporter

    Joined:
    Apr 19, 2012
    Messages:
    4,176
    Gender:
    Female
    Location:
    Silesia
    Thanks for doing this. Thumbs up.
     
  9. Klackerz

    Klackerz Bridgeburner

    Joined:
    Oct 22, 2009
    Messages:
    786
    Location:
    India
    Only the English stories are shown by default in Scryer. When you check the English stories in ffn, they are matching.

    As Raven said in the OP, currently only HP stories are indexed without any crossover support.
     
  10. KGB

    KGB Unspeakable

    Joined:
    Jan 18, 2011
    Messages:
    751
    One feature I'm missing from the FFN options is the time range filters. Being able to filter out old abandoned fic's is a must have.
     
  11. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    01-17-14 Changelog
    • (Fix) Fulltext: Summary/title/author can now contain forward slashes. e.g. 'H/Hr' is now supported.
    • (Fix) Search URL with query params will now perform a search upon load.
    • (New) Sorts: Added a Popular & Recent sort. This is currently a little slow, but mixes activity (reviews, follows, favorites) and recency measures. Combined with filters, this makes it easier to find somewhat interesting fics.
    • (New) Filters: Added a chapter number filter, so you can limit it to say 5-10 chapters.
     
  12. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    I agree, I'll add this to my to-do.
     
  13. Spanks

    Spanks Minister of Magic

    Joined:
    Jun 8, 2007
    Messages:
    1,296
    Location:
    New Jersey
    Let me know if this is dumb or not. I see that it shows the ratings for stories DLP has reviewed and I was wondering if it's possible to rate the stories on Scryer too. That way as we are browsing through our search results we'll see how other readers rated a story. It might also help us find those few fics that are good time wasters, but not good enough to be put up for review on DLP.

    Anyway, awesome work, sir.
     
  14. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    You can currently see what reviews on DLP are for Scryer results, but I haven't decided if I'll allow rating / upvoting from the Scryer results.
     
  15. Lord Raine

    Lord Raine Disappeared DLP Supporter

    Joined:
    May 4, 2010
    Messages:
    1,038
    Just wanted to let you know that I've used it for all of thirty seconds and I already found a fic I've been trying to re-find after reading it about a year and a half ago.

    This is fantastic. I almost want to ask you to get around to making it work beyond the HP section of the site, if only because I've been looking for the very first piece of fanfiction I ever read for a long time, and this would probably find it if anything could.

    Amazing work.
     
  16. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    Which fandom are you looking for? I'm currently looking at indexing everything HP crosses with (both the crossovers and non-crossovers), starting with a few smaller ones.
     
  17. Tommy

    Tommy The Green Ranger

    Joined:
    Nov 30, 2012
    Messages:
    202
    Location:
    In a bomb shelter, South Africa.
    Perhaps the most wanted fandam besides HP is probably Dresden, so that could be a good place to start. Then trickle down to some popular manga, etc.
     
  18. PotterFan

    PotterFan High Inquisitor

    Joined:
    Aug 12, 2011
    Messages:
    505
    Location:
    The Road of Life
    Naruto should be the next fandom....Definitely Naruto. I need -mpreg and -slash to cut through all the BS.
     
  19. phazer11

    phazer11 Third Year

    Joined:
    Mar 26, 2011
    Messages:
    95
    Location:
    IN THE DEEP SouthEastern USA
    Dresden Files crossovers and regular fictions would be my next request for inclusion in Scryer.
     
  20. Lord Raine

    Lord Raine Disappeared DLP Supporter

    Joined:
    May 4, 2010
    Messages:
    1,038
    If you're starting small, it won't be on the list. The very first one I read was a Naruto fic. I remember almost every detail about the story, and by literally any and all standards whatsoever we have today, it was truly objectively horrible. But it was the first one I ever read (and honestly it was so long ago that a lot of the cliches really weren't cliches yet), and I've always been trying to find it again. Plus the author wrote a few other things that weren't bad either. Mostly slice of life and comedy stuff.

    I'll confess that I haven't actually looked, but I wouldn't have imagined there would have been enough Dresden on fanfiction.net to really be worth filtering. It might be good as a test to see how well Scryer works in other fandoms, but I'm not really sure Scryer would be necessary to search like it would be for some of the larger ones.
     
    Last edited: Mar 8, 2014