Dark Lord Potter Forums
Go Back   Dark Lord Potter Forums > Common Room > General Discussion
Donate Register Rules Library List IRC Chat FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Notices

Donate to DLP Scryer Banner

 
Thread Tools Display Modes
Old 07-14-2006, 11:48 AM   #1
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
DLP-style FF.net Search Engine

Hello everybody,

I am once again thoroughly peeved by the search engine on FF.net. Therefore I have finally decided to make my own version. Any suggestions for it, practical or legal are welcome, as well as wishes.

I thought of a small program, where I can implement my own search parameters. My ideas so far mostly include the capability to do negative searches.
Example: Find fics, that are all genres but romance.

And the exclusion of the various wording of shipping variations such as HPDM HP/DM etc.

And there will be an option: NO SLASH

I have 2 weeks free from work from now on, that means I will finally have the time to work it.

The information base will have to be the listing in FF.net. I can't see a way to connect to the db. If anyone has a suggestion on how to directly ask the FF.net DB, please tell.

So, Please make your suggestions, I may be able to work them in. If anyone has seen such a tool and I was too stupid to find it, please flame me - for I deserve it in that case.
Olfrik is offline   Reply With Quote
Old 07-14-2006, 12:02 PM   #2
Xantam
Minister for Magic
 
Xantam's Avatar
 
Join Date: Jan 2006
Location: Steel City
Gender: Male
Posts: 1,243
Well the ff.net search can find stories that don't have certain words like "slash". I don't think this would work because not all slash stories are labeled properly.
Xantam is offline   Reply With Quote
Old 07-14-2006, 12:08 PM   #3
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
That labeling would be a problem. What makes me sick is that I have to put in a positive term to search. Well I always put Harry in, but basically, I want to find a good unknown story, a new one by eliminating everything that sucks. Thats the idea. So that I can find something new to read and not have to be bombarded with the shit there. Half of it is sick fluff.

Edit: More then 80% is fangirl fluff.
Olfrik is offline   Reply With Quote
Old 07-14-2006, 12:10 PM   #4
Taure
Order Member
Magical Cores Are Real
 
Taure's Avatar
 
Join Date: Mar 2006
Location: United Kingdom
Gender: Male
Posts: 869
DLP Supporter Donor Star
High Score: 13,152
I'm not quite sure I understand...you are going to make a search engine to find stories on ff.net, but don't infact have access to the ff.net database?

It's a good idea, but seems like a waste of time to me, as theres no way in hell that the ff.net admins are going to let you fiddle around with their site.
__________________

HPATTGH: 1/35 Complete
Taure is offline   Reply With Quote
Old 07-14-2006, 12:16 PM   #5
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
Quote:
It's a good idea, but seems like a waste of time to me, as theres no way in hell that the ff.net admins are going to let you fiddle around with their site.
The HTML of the listing is very clean, I can just run a parser over it. And the admins would never know, I'd make it a desktop app. And the content isnt theirs only the service.
So if they give the listings for free, I can do what I want with it, as long as I dont implement another service.
Olfrik is offline   Reply With Quote
Old 07-14-2006, 02:17 PM   #6
CGB
Auror
 
Join Date: May 2006
Location: Germany
Age: 28
Gender: Male
Posts: 643
I think it's a good idea. A agree that slash is a problem, but if you can sort most of what you don't like out, it's far easier than normal.
CGB is offline   Reply With Quote
Old 07-14-2006, 03:31 PM   #7
ip82
Prisoner
 
ip82's Avatar
 
Join Date: Nov 2005
Posts: 2,921
FFN will NEVER allow access to their database from some unknown outside server, even for reading only.

The only way I can see would be to create your own database; Say write a script to download info about all the stories and write them into a database of your own... and do that once per day.

In short, not worth the effort.
__________________
IP82 stories:

Potter's Resistance 1: Breaking Ties -> DLP Thread
Yin and Yang -> DLP Thread

Ficwad Profile || HP Recycle Bin || Crap filter C2
Rule no. 1 - You do not talk about Alpha Fight Club...
ip82 is offline   Reply With Quote
Old 07-14-2006, 03:47 PM   #8
Cervus
Raptured to Hell
 
Cervus's Avatar
 
Join Date: Aug 2005
Location: Manchester, England.
Age: 32
Gender: Male
Posts: 849
Send a message via AIM to Cervus Send a message via MSN to Cervus Send a message via Yahoo to Cervus
There was a piece of software out a while ago that allowed people to search FF.net from their desktop (I believe it may have been called Slurp), so it is possible. The author of said software, however, worked with the premise that stories with more reviews should be higher up on your results list. We, of course, all know that review counts equal diddly squat.
Cervus is offline   Reply With Quote
Old 07-15-2006, 03:59 AM   #9
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
Quote:
Originally Posted by ip82
FFN will NEVER allow access to their database from some unknown outside server, even for reading only.

The only way I can see would be to create your own database; Say write a script to download info about all the stories and write them into a database of your own... and do that once per day.
I wouldn't read the DB but the listings, that's not optimal but should work fine, as far as I have seen. And I didnt plan to put everything new in there but only the most recent. I want it as a way to search new stories without reading 10 times about cute HPDM fluff and vomit twice. So that would just appear on the admins radar like any other robot.
Olfrik is offline   Reply With Quote
Old 07-15-2006, 04:36 AM   #10
Amerision
Heir
Galactic Sheep Emperor
 
Amerision's Avatar
 
Join Date: Apr 2006
Location: The Gardens in the Desert Sand
Posts: 2,541
DLP Supporter Donor Star
Hmm, it sounds like searches would take alot longer as you are using the listings, but it should work.

Ask Lord Ravenclaw, the guy is the god of computers around here.
__________________


I dream of gardens in the desert sand
I dream of love as time runs through my hand
I dream of fire
Those dreams are tied to a horse that will never tire
And in the flames
Her shadows play in the shape of a man's desire

Amerision is offline   Reply With Quote
Old 07-15-2006, 03:45 PM   #11
cazten
Slug Club Member
 
Join Date: Nov 2005
Location: ca
Age: 28
Posts: 198
just a litle thought, if you exlude stories with a search that have the term slash in it, what happens to the stories that specifically label their story as "NOT Slash".
Even though we now no its not slash it has the term in summery for the search?
__________________
[CENTER][/
cazten is offline   Reply With Quote
Old 07-15-2006, 04:56 PM   #12
saL
Second Year
 
saL's Avatar
 
Join Date: Jan 2006
Location: Under Fidelius
Gender: Male
Posts: 57
a good idea if you get this working
maybe you can even implement a trunk operator, it always bothered me that i had to search for things like independence independent.... when it could be done with a simple indep*
__________________
There is only power and those authors that are to naive to let Harry have it
saL is offline   Reply With Quote
Old 07-15-2006, 05:30 PM   #13
Dark Prince
Guest
 
Posts: n/a
Good idea. More than half of the stories on FF.Net are slash and I have no interest in that.

I don't know much about computers but maybe you can add another feauture to the search engine, like '5 chapters or more' or '10 chapters or more' so when you search you don't get stories that have one chapter with 200 words.
  Reply With Quote
Old 07-16-2006, 10:41 PM   #14
Jeram
Chief Warlock
Elder of Zion
 
Jeram's Avatar
 
Join Date: Jun 2006
Gender: Male
Posts: 1,575
DLP Supporter Donor Star
As a computer guy, I'd be happy to throw in my expertise - this type of project is something I've been thinking about for a while. I actually wrote a program to parse out details like word count, chapter count, etc, so I have a little experience with the mess that ff.net uses to format its page.

-J
Jeram is offline   Reply With Quote
Old 07-16-2006, 10:48 PM   #15
Brooklynight
Seventh Year
 
Brooklynight's Avatar
 
Join Date: Jul 2006
Location: New York
Age: 28
Gender: Male
Posts: 295
I think that its a great idea since most people on DLP would agree that finding a good fic on ff.net is excrutiating. A slash filter would be much appreciated but I'm not sure how successgul it would be simply because some many fics aren't marked as slash, prehaps you could incorporate some type of fic flagging system.
Brooklynight is offline   Reply With Quote
Old 07-17-2006, 08:35 AM   #16
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
The flagging wouldn't make any sense if everyone would have to do it themselves. I just want to write a small desktop app, not implement a service to pool the flags. It would be the easiest and overall best way as an internet service but I think I would get in serious trouble with the admins, because it would just look like an independent interface to their own db, essentially copying FF.net. That wasnt my intent and would be shut down immediately. I have some ideas concerning the pooling and flagging but it would take much more time than I planned to invest. So I'll start small.

Jeram:
which technology is your code? the parsing is actually the easiest for me, I write interfaces and such professionally, the gui is my weak spot.
Olfrik is offline   Reply With Quote
Old 07-17-2006, 03:41 PM   #17
Zevrillion
Founder
Supreme Commander
 
Zevrillion's Avatar
 
Join Date: Apr 2005
Location: New Zealand
Gender: Male
Posts: 753
Interesting project. What about an addon to firefox instead? Instead of adblock we have a slashblock that just filter out slash...


Olfrik, what language have you planned to write the app in?
__________________


"Glorious are the Ori, who lead us to salvation, who did fight the evil that would doom us all to mortal sin. Did they defeat the old spirits and cast them out? And now, with the strength of our will, they do call upon us to prevail against the corruption of all unbelievers."
Zevrillion is offline   Reply With Quote
Old 07-17-2006, 10:13 PM   #18
Jeram
Chief Warlock
Elder of Zion
 
Jeram's Avatar
 
Join Date: Jun 2006
Gender: Male
Posts: 1,575
DLP Supporter Donor Star
Ah, excellent, because parsing is probably my weakest ability.

I'm familiar with C, C++, Visual Basic, C# (which in this case is what I'd prefer to use - more stable than VB, easier than C++) - I'm less experienced with Perl and Python.

-J
Jeram is offline   Reply With Quote
Old 07-18-2006, 04:35 AM   #19
Olfrik
Seventh Year
 
Olfrik's Avatar
 
Join Date: Mar 2006
Location: Berlin
Age: 41
Gender: Female
Posts: 229
Send a message via AIM to Olfrik
I was planning on java, because I want it to run on linux as well. And the parsing, regex libs etc. are quite good.
The idea of making it a firefox addon is good. I'll look into it. I dont really know how much possibility there is. That would eliminate my platform problem however.

EDIT:
The Firefox addon would be JavaScript and XML. That means for me the parts are easy to do, but I have no idea how to put it all together. I'll need time to come up with a structure for that, I dont have any experience with Mozilla Dev. But it looks promising, and I wont have to program the gui because its generated by firefox from xul.

Last edited by Olfrik; 07-18-2006 at 04:56 AM.
Olfrik is offline   Reply With Quote
Old 07-18-2006, 01:20 PM   #20
Lord Ravenclaw
Administrator
System God
 
Lord Ravenclaw's Avatar
 
Join Date: Apr 2005
Location: Denver, CO
Gender: Male
Posts: 4,100
Impractical without database access. Their backend scripts/frontend search are custom-tailored to their needs and is also paginated so you'd be limited to whatever they throw out and not be able to show 100 results, or 150, etc. without multiple pageloads per search. Once you get that though you could filter it further, but really it's a waste of time for most intensive purposes.
__________________
Scryer - FF Search Engine
Lord Ravenclaw is offline   Reply With Quote


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
OLD fanfiction request/search Aduc Story Search 9 04-18-2006 09:58 PM
In search of a fic Mordecai Story Search 8 02-19-2006 02:13 AM
Search for Ged Lutris Story Search 2 01-20-2006 07:28 PM
Fun with Search engines Necrule Paen The Humor Mill 3 10-24-2005 09:25 PM


All times are GMT -4. The time now is 08:53 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Copyright ©2005 - 2016 DLP Group. All rights reserved.
No personal intellectual property on this site may be used without the credit and express permission of the respective authors.