Not sure if anyone noticed, but search was broken. It has a "common word" filter that helps limit the number of returned threads when a common word is used. But it seemed broken, for example, claiming the search "Camaro" was too common and refused to search. I set the threshold to 0% and rebuilt the search index. Seems to work properly now.
I also added a 30 second blackout period for back to back Guest searches. Cause ad bots/crawlers. Though if I ever want to get this indexed in Google and Bing, I should probably consider setting that back to 0 seconds.
Search is fixed
Re: Search is fixed
Web crawlers don't use search. They follow every link on every page.
Re: Search is fixed
I have seen search engines do dynamic searches to create search results. I guess that's not a crawler but it's something...