Hi,
I'm a new nutch user. My company wants me to look into using this technology
to index our internal wiki website as well as sharepoint docs (using tika).
Right now, I just want nutch to index the entire wiki site but I'm having
problems. I've read other people's problems with this but I haven't found a
solution that worked for me.
I have nutch 1.0 installed.
The wiki site is MoinMoin if that helps. The pages don't have extensions
like .html. They're in the form of http://wiki:8000/Engineering as an
example. So all pages only have 1-level depth paths.
I'm running nutch with the follow command:
bin/nuch crawl urls -dir crawl -depth 100 -topN 1000000 > crawl.log
I have a urls folder with a file called wiki that points to the top-level
page of the site.
I set the crawl-urlfilter.txt to accept everything except the default
exclusions:
-^(file|ftp|mailto):
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
-[?*!@=]
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
+.
And I set the db.ignore.external.links property in nutch-default.xml to true
so it doesn't go outside of the site. (db.ignore.interal.links is set to
false)
After the crawl command completes, the search returns some pages, but there
are still some pages that are maybe 2 or 3 levels from the starting page
that don't show up on search.
Any help would be appreciated.
Thanks,
Kane
--
View this message in context: http://old.nabble.com/problem-crawling-entire-internal-website-tp27908943p27908943.html
Sent from the Nutch - User mailing list archive at Nabble.com.
