How do web crawlers such as googlebot (and for othe, lesser known search engines) fit into that?
Does it mean this API can be discriminating to starting search engine companies?

Just use Firefox. That’s it.