1. Another story competition is beginning! (Q1 2018)

    "You're bleeding on my floor."

    We've moved to a quarterly format, so you really have no excuse not to participate this time.

    So check out the new thread discussing scoring, rules, and other such matters in the in the Story Competitions forum and get cracking.

    Dismiss Notice

Scryer - DLP's FF Search Engine

Discussion in 'General Discussion' started by Lord Ravenclaw, Mar 31, 2016.

  1. zekses

    zekses Squib

    Joined:
    Dec 3, 2015
    Messages:
    17
    High Score:
    0
    If I am correct in my understanding of why "Published After" and "Updated After" filters were introduced it would be better to merge those into "Active After", imo
     
  2. zekses

    zekses Squib

    Joined:
    Dec 3, 2015
    Messages:
    17
    High Score:
    0
    Mostly because other usecases for these will likely take less than 1% of searches. Again, imo.
     
  3. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    I appreciate your input, but I disagree. I find it useful to search for stories initially published after a given date, regardless of when it was last updated.
     
  4. zekses

    zekses Squib

    Joined:
    Dec 3, 2015
    Messages:
    17
    High Score:
    0
    I am actually really interested where do you use searches like this.
    Is it for finding pre-canon Harry Potter fics, or is there some other use case I am not aware of?
     
  5. Jon

    Jon The Demon Mayor Admin DLP Supporter

    Joined:
    Jun 5, 2006
    Messages:
    7,449
    Location:
    Australia
    Fics go stale for lack of a better word.. I use it to pick a published date. Eg 2013 and then sort by review. This eliminates big stories before that. A quick and easy way to find popular stories.
     
  6. Sataniel

    Sataniel Seventh Year

    Joined:
    Jan 24, 2016
    Messages:
    201
    High Score:
    0
    It's useful to filter stuff that came out between parts of the series. Or for new fics that managed to gather significant wordcount quickly.

    Also we can request fandoms to be added? "Persona" would be useful, what with the influx of the fics after Persona 5 came out.
     
  7. zekses

    zekses Squib

    Joined:
    Dec 3, 2015
    Messages:
    17
    High Score:
    0
    Yes, I was kind of confused about use cases because "published after" is probably most useful onem but it came with a whole baggage of "updated before" and such. It's a legit useful thing.
     
  8. Jon

    Jon The Demon Mayor Admin DLP Supporter

    Joined:
    Jun 5, 2006
    Messages:
    7,449
    Location:
    Australia
    A use case for updated before would be for stories (with HP for example) without certain book plot elements.

    e.g want a story that has no chance of having horcruxes? Set the update before date for the day before DH was released.

    There are uses for it, but they're more niche, which is fine.

    Scryer exists to find stories, having more options to narrow searches is never a bad thing.
     
  9. bobbybrown

    bobbybrown Squib

    Joined:
    Feb 12, 2015
    Messages:
    7
    Location:
    Sweden
    I don't know if this is the correct place to post this but for some reason i can't seem to find any story newer than 8 days using Scryer. Is this on my end or is there a option i need to change?
     
  10. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    I've been rewriting how Scryer works under the hood, it's been reindexing the world and not focusing on keeping the existing index up to date. Should be solved soon.
     
  11. Dagro

    Dagro Second Year

    Joined:
    Sep 25, 2007
    Messages:
    53
    Wish: Would be cool if I could both include & exclude Genres&characters at the same time. For example when using "Excluded Characters + 'OC' " I can not search for any specific characters anymore. Same with genres. I don't know how the include / exclude tags system works, I have similar tags in my default summary search, but if that ist is open to suggestions I would like to add something a long the lines:
    - self insert/SI, high school, modern, university, OC, AU/Alternative Universe in various forms, Titantic (!!!), CEO of YX, etc. just the usual crazy grey noise I would like to filter^^

    I am using already filter via summary along those lines with wonderful results but depending on how the tag system works and how easy it is to integrate different tags, some of those might help to generally filter a bunch of unwanted content.

    Not sure if I posted in this thread before, I LOVE scryer, thanks so much! <3
     
    Last edited: Sep 16, 2017
  12. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    The best way to do this currently is to use "Required Characters" (or Genres) to add characters you want, and Exclude Characters to remove the ones you don't want. e.g. Required[Harry P.], Exclude[Hermione G., Ron W.].

    Include/exclude tags are somewhat as you'd expect. If you include a tag, all stories returned will have that tag, but may have other tags. Exclude allows you to then provide tags you do not want to be returned. e.g.

    Story 1: tags [harem, slash]
    Story 2: tags [harem, gender bender]
    Story 3: tags [harem]

    With a include=[harem] and exclude=[slash], Story 2 and Story 3 would be returned.

    The tags are currently limited to system-generated tags, Scryer attempts to determine from characters, relationships, and the summary whether a story fits certain criteria for each tag. I do want to expose tagging to users in the future, but it's not high on my list given how few people would actually tag stories.
     
  13. zekses

    zekses Squib

    Joined:
    Dec 3, 2015
    Messages:
    17
    High Score:
    0
    @Lord Ravenclaw, I am now laughing rolling on the floor after reading your recent scryer commits and realizing that for the past week we have been doing basically the same thing in two different languages XD
     
  14. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    I've been replacing my Python logic which is terrible :)
     
  15. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    After a lot of query rewriting to update from ElasticSearch 2.x to 5.x, Scryer is moved over. The index should be faster now. This also unblocks work on improving search features. As an added bonus, it's also up to date.
     
    Last edited: Sep 24, 2017
  16. Zombie

    Zombie John Waynes Teeth Prestige DLP Supporter

    Joined:
    Apr 28, 2007
    Messages:
    788
    Location:
    Within the Garden of Nurgle.
    High Score:
    2,094
    Was there really that big of a version difference? 2.x to 5.x seems like a big leap.
    I'm excited to hear this.
     
  17. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    ES 2.x is what was installed on Scryer's old index server. 5.x is what powers the DLP Forums. I wanted to unify the two. Unfortunately, there was no way to upgrade the old index and required reindexing all stories into the newer ElasticSearch, so I took the opportunity to fix some sins of the past.

    2.x to 5.x isn't really a big change, though they removed a bunch of functionality. Mostly, they're going for the illusion of stability via larger version numbers to appeal to larger customers. 1.x was the version we used at my last job for a looong time. Then 2.x came out, and they skipped 3.x and 4.x and went right to 5.x.
     
  18. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,236
    Location:
    Denver, CO
    Update: Sorting by DLP score works again.
     
  19. swe

    swe Muggle

    Joined:
    May 23, 2017
    Messages:
    2
    High Score:
    0
    Lord Ravenclaw, could you please add Twilight fanfiction to the index? It is a big enough fandom (219K stories) that Scryer would be really helpful.
     
  20. Xiph0

    Xiph0 Administrator Admin

    Joined:
    Dec 7, 2005
    Messages:
    8,892
    Gender:
    Male
    Location:
    People's Republic of California
    Every day we stray further from God's light.