1. Hi there, Guest

    Only registered users can really experience what DLP has to offer. Many forums are only accessible if you have an account. Why don't you register?
    Dismiss Notice
  2. Introducing for your Perusing Pleasure

    New Thread Thursday
    +
    Shit Post Sunday

    READ ME
    Dismiss Notice

Scryer - DLP's FF Search Engine

Discussion in 'General Discussion' started by Lord Ravenclaw, Mar 31, 2016.

  1. MonkeyEpoxy

    MonkeyEpoxy Prisoner DLP Supporter

    Joined:
    Aug 11, 2011
    Messages:
    2,865
    Gender:
    Male
    Location:
    Colorado
    So this is where we can post fandom requests? I wouldn't mind seeing Dragon Age and Elder Scrolls on it
     
  2. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Sure, done. They should be finished indexing here soon.
     
  3. lopeck

    lopeck Sixth Year

    Joined:
    Oct 22, 2013
    Messages:
    179
    Gender:
    Male
    Location:
    Germany
    Is there a reason not to index everything? Beyond having to input them manually I mean.
     
  4. Zombie

    Zombie Lo-fi Snake Jazz Moderator DLP Supporter

    Joined:
    Apr 28, 2007
    Messages:
    5,572
    Because there isn't a demand for everything I would imagine.
     
  5. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Indexing takes continuous resources, especially for crossovers. I have a tentative plan to index more by more intelligently scheduling fandoms, but even then I'm not sure I'm interested in indexing all fandoms -- certainly not ones below some size (as yet determined) and even then I want to focus my limited time and server resources on the fandoms users want. If someone requests a fandom, I'll definitely index it. Otherwise, I've been focusing on the 20% of fandoms that make up 80% of the total stories you could ever want.
     
  6. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    New Feature

    Today I released an extension to the relationship search in Scryer which allows queries looking for, say, [A, B] to also find pairings of [A, B], [A, B, C], [A, B, D], [A, B, C, D] and so on. This works for single individuals too, so if you just want H/someone(s), that works. Excludes are also enabled, so lets say you really liked H/Fleur+many, but hate Hermione (you'd be wrong to, but you know, just in case), you can search for [Harry, Fleur], but exclude Hermione. If you're into moresomes, and like individually one person or another, but not together, you can exclude [B, C] to ensure you only get [A, B, ...] or [A, C, ...].

    It's not totally tested yet, so give it a go and let me know if you run into issues.
     
  7. Snarf

    Snarf Squanchin' Party Bro! ~ Prestige ~ DLP Supporter

    Joined:
    Apr 27, 2007
    Messages:
    15
    Gender:
    Male
    Location:
    Forty-Six & 2
    High Score:
    1,832
    I'm currently seeing only data that's 5 days old or further back.
     
  8. Henry Persico

    Henry Persico Groundskeeper DLP Supporter

    Joined:
    Sep 13, 2011
    Messages:
    342
    Location:
    Argentina
    Same here, and since wednesday I can't acces vía Chrome, it doesnt load the https.
     
  9. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
  10. KGB

    KGB Death Eater

    Joined:
    Jan 18, 2011
    Messages:
    983
    Trying to log in with DLP throws up a 422 error.

    This is on Firefox, Android
     
  11. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    I think I saw the error fly by. The issue is there's already a KGB signed up with a different email from your DLP email. Is that you?
     
  12. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Open State Filtering

    So, you can exclude stories you've opened now. You don't need to do anything different, open stories from search results while being logged in. If you see the eye-icon, it can be excluded. Obviously, this only works if you're logged in.

    You can exclude all stories you've opened or exclude stories that have been updated since you last opened them from Scryer.

    Happy Scrying!

    Screen Shot 2019-03-26 at 19.47.55.png
     
  13. KGB

    KGB Death Eater

    Joined:
    Jan 18, 2011
    Messages:
    983

    I do remember making an account for it ages ago, so it's probably me.

    Can you either merge or delete that account? Since I don't remember the password and its from before saved searches so nothing will be lost.
     
  14. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Done. You should be able to sign in.
     
  15. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Hear ye, hear ye. Two new features!

    Hidden Stories and Authors

    A long-standing feature request, you can now hide stories OR all stories written by a given author.

    For instance, maybe you come across a story by megamatt09. So you can hide that story (or, as you should in this case, hide all stories by megamatt09.)

    Screen Shot 2019-04-07 at 14.03.36.png

    If for some reason you wanted to show him again, go to Manage Hidden:

    Screen Shot 2019-04-07 at 14.01.27.png

    And unhide megamatt (bad call, IMO.)

    Screen Shot 2019-04-07 at 14.08.14.png

    All Stories by an Author

    As seen above, "Other stories by Author" takes you to a page of all stories by that author. Just a convenience feature.
     
  16. coarsehalogen

    coarsehalogen Muggle

    Joined:
    Jun 22, 2019
    Messages:
    1
    Gender:
    Male
    Would you be willing to share your indexing script? I'm trying to index a few fandoms myself, but fanfiction.net doesn't provide structured metadata, and the Scryer search engine doesn't have the ability to search on the story text, and I have the resources to be able to index the full text.
     
  17. Nuit

    Nuit Dark Lord

    Joined:
    Feb 14, 2010
    Messages:
    1,915
    Location:
    The Peach State
    Might be on the GitLab. https://gitlab.com/projectdlp/scryer
     
  18. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    https://gitlab.com/projectdlp/accio is the actual indexer (you'll need to be logged into GitLab to see it). It delivers semi-structured metadata to background jobs running under https://gitlab.com/projectdlp/scryer. You can see where the actual indexing is done at https://gitlab.com/projectdlp/scryer/blob/master/app/workers/story_index_base.rb.

    That's correct. Honestly, I'm not really convinced it's worth the time or resources to index story content (for Scryer's aims anyways). I could be convinced that processing chapters to pull out interesting keywords/themes/character mention timelines, but the actual content--I'm not sure.
     
  19. HMM

    HMM Squib

    Joined:
    Apr 18, 2017
    Messages:
    7
    Location:
    London, UK
    High Score:
    0
    Thanks for the amazing tool! Could you index the Gravity Falls and A Series of Unfortunate Events fandoms please?
     
  20. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,331
    Location:
    Denver, CO
    Done and indexed.
     
    HMM
Loading...