I just recognized that a lot of my api calls are caused by various bots like google and semrush (already asked them why they don't respect the robots.txt).
Although I thought there is the robots.txt for a while now which explicitly says bots should not crawl the api.
Has anyone else seen this and has an idea how we could fix/prevent that?
Edit: is there any usecase where it would make sense to have the api crawled? I mean best would if xano just blocks those request on cdn level, would save a lot of resources on our instances.