Excellent question!
Bots that have been crawling the site I've seen so far are:
- Anthropic
- Applebot
- Baidu
- Bing
- Facebook
- Yahoo
There's another one I can't remember and I'm not sure why we haven't seen the Googlebot yet? I started researching it last night but stopped short of finding the answer why.
These bots are relatively benign and they are scraping our content for search results which is a good thing and desirable and to feed the LLM machine which is not so good but unavoidable unless we want to lock everything down like at Rogue Nation.
Neither robots nor guests can read any of the information in your profile or the forums in the private member's section.
For example, since they've crawled the site if you use Bing and search for "Citadel 81 forums", we show up first. We aren't showing up on Google at all yet but I'm going to give it another week before knocking on their door.
Since we aren't selling anything, I don't really care about what's called SEO or search engine optimization but it would be nice if people could find us through Google.
Now the guests, on the other hand, concern me a bit more. These are mostly the spammers and scammers looking to get through the door.
I can check where they are coming from through the built in IP lookup for admins. I see quite a few coming from the U.S. which may just be proxies. Current "guests" are from France, Germany, Brazil, Argentina, Singapore, Mexico and Taiwan.
Here's a random one I picked to show what I can see about them
If they become too numerous and start affecting performance and bandwidth, I will route everything through Cloudflare for an additional layer of protection.
I will add that the xenForo platform has about 100X more built in protections and spam filters than RN, DI, CA and probably TLA combined. That's one of the benefits of using $200 software compared to free open-source software.
Here's what the guests look like when they run into a brick wall or try to register
I keep an eye on this traffic regularly. So far it is pretty normal. When it gets into the hundreds or thousands, then we'll employ further remediation measures.
