1. Hey Guest, welcome back to DLP

    As you can see, we've changed our look. We've migrated from vBulletin to the Xenforo forum system. There may be issues or missing functionality, if you find anything or have feedback, please check out the new Xenforo Migration Feedback forum.

    Our dark ("Dark Lord Potter") theme is under heavy development. We also have a light ("Light Lord Potter") theme for those happier with a light background and darker text.

    Dismiss Notice
Dismiss Notice
Hey Guest! Are you any good at cooking? Got a favourite recipe that you love to cook or bring out to impress that special someone? Why not share it! A new forum called The Burrow has opened and it's all about homemaking!

DLP-style FF.net Search Engine

Discussion in 'General Discussion' started by Olfrik, Jul 14, 2006.

  1. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    Hello everybody,

    I am once again thoroughly peeved by the search engine on FF.net. Therefore I have finally decided to make my own version. Any suggestions for it, practical or legal are welcome, as well as wishes.

    I thought of a small program, where I can implement my own search parameters. My ideas so far mostly include the capability to do negative searches.
    Example: Find fics, that are all genres but romance.

    And the exclusion of the various wording of shipping variations such as HPDM HP/DM etc.

    And there will be an option: NO SLASH

    I have 2 weeks free from work from now on, that means I will finally have the time to work it.

    The information base will have to be the listing in FF.net. I can't see a way to connect to the db. If anyone has a suggestion on how to directly ask the FF.net DB, please tell.

    So, Please make your suggestions, I may be able to work them in. If anyone has seen such a tool and I was too stupid to find it, please flame me - for I deserve it in that case.
     
  2. Xantam

    Xantam Denarii Host

    Joined:
    Jan 8, 2006
    Messages:
    1,247
    Location:
    Steel City
    Well the ff.net search can find stories that don't have certain words like "slash". I don't think this would work because not all slash stories are labeled properly.
     
  3. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    That labeling would be a problem. What makes me sick is that I have to put in a positive term to search. Well I always put Harry in, but basically, I want to find a good unknown story, a new one by eliminating everything that sucks. Thats the idea. So that I can find something new to read and not have to be bombarded with the shit there. Half of it is sick fluff.

    Edit: More then 80% is fangirl fluff.
     
  4. Taure

    Taure Magical Core Enthusiast Prestige DLP Supporter

    Joined:
    Mar 5, 2006
    Messages:
    948
    Location:
    United Kingdom
    High Score:
    13,152
    I'm not quite sure I understand...you are going to make a search engine to find stories on ff.net, but don't infact have access to the ff.net database?

    It's a good idea, but seems like a waste of time to me, as theres no way in hell that the ff.net admins are going to let you fiddle around with their site.
     
  5. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    The HTML of the listing is very clean, I can just run a parser over it. And the admins would never know, I'd make it a desktop app. And the content isnt theirs only the service.
    So if they give the listings for free, I can do what I want with it, as long as I dont implement another service.
     
  6. CGB

    CGB Auror

    Joined:
    May 7, 2006
    Messages:
    643
    Location:
    Germany
    I think it's a good idea. A agree that slash is a problem, but if you can sort most of what you don't like out, it's far easier than normal.
     
  7. ip82

    ip82 Prisoner

    Joined:
    Nov 14, 2005
    Messages:
    2,921
    FFN will NEVER allow access to their database from some unknown outside server, even for reading only.

    The only way I can see would be to create your own database; Say write a script to download info about all the stories and write them into a database of your own... and do that once per day.

    In short, not worth the effort.
     
  8. Cervus

    Cervus Raptured to Hell

    Joined:
    Aug 29, 2005
    Messages:
    849
    Location:
    Manchester, England.
    There was a piece of software out a while ago that allowed people to search FF.net from their desktop (I believe it may have been called Slurp), so it is possible. The author of said software, however, worked with the premise that stories with more reviews should be higher up on your results list. We, of course, all know that review counts equal diddly squat.
     
  9. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    I wouldn't read the DB but the listings, that's not optimal but should work fine, as far as I have seen. And I didnt plan to put everything new in there but only the most recent. I want it as a way to search new stories without reading 10 times about cute HPDM fluff and vomit twice. So that would just appear on the admins radar like any other robot.
     
  10. Amerision

    Amerision Galactic Sheep Emperor DLP Supporter

    Joined:
    Apr 1, 2006
    Messages:
    2,541
    Location:
    The Gardens in the Desert Sand
    Hmm, it sounds like searches would take alot longer as you are using the listings, but it should work.

    Ask Lord Ravenclaw, the guy is the god of computers around here.
     
  11. cazten

    cazten Slug Club Member

    Joined:
    Nov 22, 2005
    Messages:
    198
    Location:
    ca
    just a litle thought, if you exlude stories with a search that have the term slash in it, what happens to the stories that specifically label their story as "NOT Slash".
    Even though we now no its not slash it has the term in summery for the search?
     
  12. saL

    saL Second Year

    Joined:
    Jan 13, 2006
    Messages:
    57
    Location:
    Under Fidelius
    a good idea if you get this working
    maybe you can even implement a trunk operator, it always bothered me that i had to search for things like independence independent.... when it could be done with a simple indep*
     
  13. Dark Prince

    Dark Prince Guest

    Good idea. More than half of the stories on FF.Net are slash and I have no interest in that.

    I don't know much about computers but maybe you can add another feauture to the search engine, like '5 chapters or more' or '10 chapters or more' so when you search you don't get stories that have one chapter with 200 words.
     
  14. Jeram

    Jeram Elder of Zion DLP Supporter

    Joined:
    Jun 27, 2006
    Messages:
    1,597
    As a computer guy, I'd be happy to throw in my expertise - this type of project is something I've been thinking about for a while. I actually wrote a program to parse out details like word count, chapter count, etc, so I have a little experience with the mess that ff.net uses to format its page.

    -J
     
  15. Brooklynight

    Brooklynight Seventh Year

    Joined:
    Jul 1, 2006
    Messages:
    295
    Location:
    New York
    I think that its a great idea since most people on DLP would agree that finding a good fic on ff.net is excrutiating. A slash filter would be much appreciated but I'm not sure how successgul it would be simply because some many fics aren't marked as slash, prehaps you could incorporate some type of fic flagging system.
     
  16. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    The flagging wouldn't make any sense if everyone would have to do it themselves. I just want to write a small desktop app, not implement a service to pool the flags. It would be the easiest and overall best way as an internet service but I think I would get in serious trouble with the admins, because it would just look like an independent interface to their own db, essentially copying FF.net. That wasnt my intent and would be shut down immediately. I have some ideas concerning the pooling and flagging but it would take much more time than I planned to invest. So I'll start small.

    Jeram:
    which technology is your code? the parsing is actually the easiest for me, I write interfaces and such professionally, the gui is my weak spot.
     
  17. Zevrillion

    Zevrillion Founder Retired Staff

    Joined:
    Apr 2, 2005
    Messages:
    753
    Location:
    New Zealand
    Interesting project. What about an addon to firefox instead? Instead of adblock we have a slashblock that just filter out slash...


    Olfrik, what language have you planned to write the app in?
     
  18. Jeram

    Jeram Elder of Zion DLP Supporter

    Joined:
    Jun 27, 2006
    Messages:
    1,597
    Ah, excellent, because parsing is probably my weakest ability.

    I'm familiar with C, C++, Visual Basic, C# (which in this case is what I'd prefer to use - more stable than VB, easier than C++) - I'm less experienced with Perl and Python.

    -J
     
  19. Olfrik

    Olfrik Seventh Year

    Joined:
    Mar 2, 2006
    Messages:
    229
    Location:
    Berlin
    I was planning on java, because I want it to run on linux as well. And the parsing, regex libs etc. are quite good.
    The idea of making it a firefox addon is good. I'll look into it. I dont really know how much possibility there is. That would eliminate my platform problem however.

    EDIT:
    The Firefox addon would be JavaScript and XML. That means for me the parts are easy to do, but I have no idea how to put it all together. I'll need time to come up with a structure for that, I dont have any experience with Mozilla Dev. But it looks promising, and I wont have to program the gui because its generated by firefox from xul. :)
     
    Last edited: Jul 18, 2006
  20. Lord Ravenclaw

    Lord Ravenclaw System God Admin

    Joined:
    Apr 2, 2005
    Messages:
    4,191
    Location:
    Denver, CO
    Impractical without database access. Their backend scripts/frontend search are custom-tailored to their needs and is also paginated so you'd be limited to whatever they throw out and not be able to show 100 results, or 150, etc. without multiple pageloads per search. Once you get that though you could filter it further, but really it's a waste of time for most intensive purposes.