I have now implemented a tool that actually spiders the ff.net site picking a random section (to create more diversity in results) and adds new user profiles to the database.

Currently, to prevent the ff.net firewall from flagging my server as a DoS attack and blocking its IP, it adds 120 profiles per hour. This is in addition to the profiles that are in the database and recrawled for updates.

This should greatly expand the quality of results for the more "fringe" fandoms, (i.e. not Harry Potter, Twilight or Naruto).

The method for the indexing is to crawl the reviews pages of stories inside of a section and grab the User IDs of people who have left registered comments, under the assumption that someone who comments is also likely to favorite, and that their favorites are more likely to be relevant to each other in a way useful to the tool.

EDIT:

Interesting note: according the data I've collected so far, less than 5% of ff.net's registered users account for nearly all of the day-to-day activity.

 

You can now download all results as an ebook in ePub format.



There is a link next to each result, as well as a place in the sidebar where you can simply request a specific story.

 

I decided to check out his ePub thing, and grabbed CoolReader, allegedly the most popular ePub reader on Sourceforge.

I don't know if these are noob mistakes, or not:

1. The ePub paragraphs are all indented, whereas ffnet's have a blank line separating each paragraph. This is probably a noob error.

2. The chapter structure is gone. ePub's allow for chapters and linked contents pages, but it seems that isn't implemented. It makes it difficult to flick to the chapter that we're looking for now.

I don't mean to be a whinger, JordanL. I don't use ereaders yet, so maybe these two problems are non-issues on such devices. I thought I'd best bring them up anyway.

 

The indentation is a style I actually implemented so that it looked more like a book, but I can see what you're saying.

The chapters, however, are seperated. You should be able to open the Table of Contents in you ereader and jump between them.

I personally use Calibre.

EDIT: I agree after looking at it again. Got rid of the indentation and made it have a line break instead.

 

Yeah, it looks like CoolReader can't interpret the Table of Contents yet.

I tried Sony's Reader, because it's regarded as highly compatible with the various ePub implementations around the place. It displays the TOCs just fine. However, it has other major problems. The first 6 chapters of The Lie I've Lived [storyid 3384712], only show the disclaimers for each chapter, with the actual chapters' contents missing. Strangely, from chapter 7 onwards, the chapters are displayed in their entirety.

CoolReader might not have retained the TOC for The Lie I've Lived, but at least it displayed the entire fic from chapter 1.

That's pretty weird. I'll give Calibre a try.

edit: I overlooked Calibre initially, because it's an ePub reader/editor, and I was restricting myself to dedicated readers only. Downloading now.

 

Hmmm... I'll take a look at that.

EDIT:

Upon review, I think Sony just made a bad eReader program... it follows the spec for epub exactly.

 

I got the info on Sony's Reader compatibility, Calibre, and others from this comparison site. http://www.jedisaber.com/ebooks/Readers.asp

They rave about Calibre there. The only reason I overlooked it was that I was going for a dedicated reader and wanted to avoid the editors... probably just to narrow down my search field.

Calibre is displaying everything okay for me. I'll dig a little deeper. Thanks for the heads up on that.

 

Hey, no problem. I'll still try to make it work in those readers if I can.

 

Good luck. It doesn't look like the ePub reader software field is very mature. Half of them don't even have proper TOC support.

 

Fortunately the Nook and iPhone support it pretty well.

But yeah, I agree, the support is lackluster.

---------- Post automerged 03-11-2011 at 11:18 AM ---------- Previous post was 03-10-2011 at 08:01 PM ----------

Well, I just got laid off today, one week after I signed the lease on a new apartment... I'm definitely not keeping the server up without donations.

If it doesn't make the goal the server will come offline March 26th. It's been fun, I'll make sure to backup all the data for when I get a new job and can afford to put it back up.

 

Another update...

I found another job in one day, and it should allow me to pay for the server out of pocket for a while, so no danger.

Continue to use the site with confidence.

 

Congratulations! If I may ask, what line of work are you in? (If it happens to be different than the copyright on the site says. )

 

I work as a web developer.

 

have no problems reading the epubs with calibre, and they look fine on my iphone4 though stanza

 

Good, those were the two test cases I used as well, and both worked fantastic for me.

 

I had an idea for an addition that would be pretty convenient - but I don't know what kind of database you're working with and if this is even possible.

Basically, anyone who has spent enough time reading fanfiction and uses your engine has either read most of the results or isn't interested in the rest of them (even while using the Tailored option).

So, how about an addition that gives us the option of showing the most favorited stories which were published within the last month or so? Maybe based on the story-ID's the user entered as search-parameters, in order to filter out the usual trash.

 

Well, I certainly got enough donations this month. Someone donated $200!

Working on implementing a way for people to rate results and help improve them.

We're up to 147,000 users, 14.7 million favorites and 2.8 million stories indexed.

 

Well I received a request today from a FFn author claiming that it was illegal for me to offer ebook format downloads.

 

Maybe it is. IANAL.

The author [or ff.net] has copyright of the fic. If you're storing the fic on your server and re-distributing it... then it's probably a copyright violation.

When an author deletes/modifies their story on ff.net, are ebook downloads from your engine instantly updated to reflect that?

If you're just passing the end user a script which runs on their own PC to download and convert the fic, then it might be okay. I know that legal cases have been made about "facilitating" copyright infringement, but I don't know if that'd apply in this case. That's definitely something to talk to an IP lawyer about, if you want to know for certain. You might have to rob a bank to pay for the privilege of his/her opinion though.

Ebook conversion is a really great feature that you've built into your search engine, and I've never once thought about any copyright implications the few times I've used it. I used to regularly save and archive all kinds of fics. Even DLP has its old archive of library fics copied from ff.net. I don't think the DLP library archive has been updated in a few years, but I don't recall any complaints about it either.

Can you just take down ebook links on a case-by-case basis if an author complains about it? If it gets too messy, no one's going to blame you for removing the feature. IP can be a legal minefield.

 

If it's not against FFn's terms-of-service, then it's not.

Sniped by yak. Long answer: Depending on the agreement between author and FFN (which I admit I haven't read very carefully) then it depends on who owns the redistribution copyrights for posted fics; if it's just the author or both the author and FFN. I'm guessing if the author makes a specific request you should honor it, but what are they going to do about it, really?