Hi. I'd like a chron job (task) that runs each day to check the sitemap.xml and look for updated pages. Then I'd like to convert the urls in the sitemap to an array and check their last updated date. I've tried using the API Requset but the response I get back from the website. I get the following error in the response:
406 - Client browser does not accept the MIME type of the requested page.
I'm assuming this has something to do with incorrect header info on the get or maybe xano can receive html/xml content?
This scraping type feature is only one part of what I'm building but would prefer to have it all in once place vs using Make.com or n8n to do this part.