1. DLP Flash Christmas Competition + Writing Marathon 2024!

    Competition topic: Magical New Year!

    Marathon goal? Crank out words!

    Check the marathon thread or competition thread for details.

    Dismiss Notice
  2. Hi there, Guest

    Only registered users can really experience what DLP has to offer. Many forums are only accessible if you have an account. Why don't you register?
    Dismiss Notice
  3. Introducing for your Perusing Pleasure

    New Thread Thursday
    +
    Shit Post Sunday

    READ ME
    Dismiss Notice

Scryer - DLP's FF Search Engine

Discussion in 'General Discussion' started by Lord Ravenclaw, Mar 31, 2016.

  1. zekses

    zekses First Year

    Joined:
    Dec 3, 2015
    Messages:
    33
    High Score:
    0
    Custom user tags.
    Exclusion of user tags from further search results
    More fandoms, per-user fandoms
    Search within all user fandoms (that doesnt timeout)
    More sort algorithms for example one from this article: https://animemonographia.wordpress....or-finding-good-fanfiction-on-fanfiction-net/
    Better readability of search results.

    Also, it's kinda strange to be called moaning when I already posted my ideas and implementation above for cross reference. Heck, I recommend scryer to ppl as a first choice and my app only for those who don't think it's enough.

    P.S. When I say that saved searches are almost useless I mean that it's good to have them, but they rate among the lowest possible time savers for me. By itself, it doesnt filter out anything you've already seen and only saves you maybe 15 seconds settings up the search, on which you later waste minutes re-reading through stuff. I am not saying it's completely useless. Just that it's not something big, whereas tags, when you enable them, will be that much better.

    ---------- Post automerged at 14:14 ---------- Previous post was at 12:27 ----------

    Speaking of readability: scryer is better than ffn in that sections in the summary dont jump wildly, but I still think that its extremely hard to quickly look at anything you want as everything is too small and almost unnoticeably different in color.
    Just a small comparison:
    [​IMG]
    [​IMG]


    The statistical information that is barely noticeable at the bottom of fic info with scryer matters more than you'd think. It should not be so small.

    P.S. If you havent read The Mating Frenzy you should, it's hilarious.
     
    Last edited: Mar 10, 2017
  2. oakgem217

    oakgem217 First Year

    Joined:
    Nov 30, 2014
    Messages:
    40
    Thanks a bunch. :)
     
  3. a51933

    a51933 Squib

    Joined:
    Jan 13, 2013
    Messages:
    6
    Thanks for this thread. I didn't know about scryer till I spotted this. It seems to cover all the search options I usually need. The only suggestion I might make is so e form of alert or RSS feed system. Something that would allow someone to create searches and then receive notifications or feeds showing any new results without having to actively search again. Maybe that would be beyond the scope of this project but it's the only thing i could thing of that isn't currently available.
     
  4. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    It's certainly on the list of things we want to do. There's a little infrastructure work going on behind the scenes currently, but once that's done alerts should be possible.
     
  5. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Much of our infrastructure work has been completed. Today I was able to release content tags as searchable entities:

    [​IMG]

    You can both exclude and include tags. For instance, I call this search The M'ladies.

    The most useful aspect of this is excluding slash or harem stories that would otherwise clutter up your search experience.

    I'm actively working on broadening the set of tags available and allowing them to be applied by users.
     
  6. SilverOtter

    SilverOtter Seventh Year

    Joined:
    Feb 20, 2011
    Messages:
    263
    Location:
    USA
    With the most recent update I cannot login anymore. It says invalid email/password. When I try to recover the password it says email address not found.

    Did this update break something or was the db wiped?
     
  7. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    User management hasn't been touched and the users table wasn't touched. I can still login.

    I looked for your DLP username and email and couldn't find them. If you didn't use these, PM me what you used.
     
  8. Miner

    Miner Order Member

    Joined:
    May 27, 2015
    Messages:
    845
    Gender:
    Male
    Location:
    East Coast
    I just opened Scryer and messed around with it for a little bit so I might not have utilized all the features (exclusions and whatnot), but I noticed while browsing that stories that have been deleted from FFN are still showing up with links to its original story page with FFN messages like Error Code 1 or whatever.

    Not sure if this is a fixable problem, or even if it's been addressed in the past, but if it hasn't, then is it possible to get rid of the deleted stories?
     
  9. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Click the little flag when you run across these and it'll remove it. Scryer doesn't continuously recrawl all pages of story listings in order to reduce load on FFN. It will be recrawling monthly however so they should disappear themselves over time.
     
  10. draykfyre

    draykfyre Second Year DLP Supporter

    Joined:
    Sep 13, 2011
    Messages:
    54
    I'm not sure this is quite the the correct place for this, but hopefully it's alright.

    I attempted to get Scryer running earlier today locally on my machine, as I thought maybe it would be neat to have a local version for when I get involved in a relatively new/small fandom, and maybe try to contribute a bit if I have any decent ideas for improvements (probably not as life is busy, but who knows). I've got everything but crawling working now as far as I can tell, but I can't figure out where the code that handles crawling ffn for entire fandoms is to figure out how to run it. Am I missing something?

    Edit: Digging into it a little more, it's also a bit unclear what fields go in the elasticsearch indices. Are those defined anywhere?

    Edit2: I see a commit message that mentions using scrapy - are the crawlers open source?
     
    Last edited: Apr 6, 2017
  11. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    39
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Can I use DLP login/password for Scryer?
     
  12. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Sorry, I just saw this. At this time, the crawlers are not open source. Mainly because they're pretty embarrassing, but I'll see what I can do to clean them up for public consumption.

    If you'd like to be my guinea pig, PM me and I can drop you the source before I make it publicly available.

    At this time no, but I do want to make a federated login.
     
  13. Quiddity

    Quiddity Squib ~ Prestige ~

    Joined:
    Apr 14, 2014
    Messages:
    8
    Location:
    New Zealand
    High Score:
    1100
    By the by, the new Scryer ad is dope.
     
  14. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO

    Palindrome's skills are unmatched. :)
     
  15. readerboy7

    readerboy7 Fourth Year

    Joined:
    Jan 11, 2013
    Messages:
    126
    Location:
    GMT + 12
    Is it possible for me to search for "all crossovers except x, y, z"? Say, if I wanted to search for all Naruto crossovers, excluding the crossovers with Bleach and Inuyasha?

    If not, could that function be added?
     
  16. draykfyre

    draykfyre Second Year DLP Supporter

    Joined:
    Sep 13, 2011
    Messages:
    54
    Could Legend of Zelda be turned on for indexing?
     
  17. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    I'll add this to my list, shouldn't be too difficult to add an "invert" toggle. :) Sorry I missed this until now!

    Done. It'll appear in a few hours.
     
  18. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Update, it's indexed. Enjoy!
     
  19. draykfyre

    draykfyre Second Year DLP Supporter

    Joined:
    Sep 13, 2011
    Messages:
    54
    Awesome, thanks!
     
  20. draykfyre

    draykfyre Second Year DLP Supporter

    Joined:
    Sep 13, 2011
    Messages:
    54
    I made a partial dlp stylish theme for scryer. None of the inputs are styled, as I'm only good enough with css to be dangerous, but it makes the search results a bit easier on the eyes in a dark room.

    To use just install stylish and copy paste the css into a new theme applying to all of scryer.darklordpotter.net
    https://gist.github.com/dspencer001/b047229702543be5ddb5a2b16b822d21

    [​IMG]
     
Loading...