Hi all, I’m currently in the process of optimizing the backend data flow for my platform, Astroma. We are migrating our core logic over to Xano to take advantage of its powerful API orchestration and background processing capabilities, but I've run into a structural question regarding API efficiency.
Our application relies heavily on fetching real-time data packets from an external engine. The payload returned from this third-party API is massive, nested JSON, and highly unoptimized. Currently, when a user logs in, Xano hits the external API, receives the raw payload, loops through the data to transform it into a clean format, and then passes it to the frontend.
As our daily active user count climbs, this setup is causing two major issues: we are flirting with the external API's rate limits, and the time-to-first-byte (TTFB) is too high due to the heavy data transformation happening inside the Xano function stack on every request.
I want to redesign this workflow using Xano’s internal features:
Setting up a Cron Job / Background Task that pings the external API once every night at midnight UTC to grab the global data packet.
Using Xano's data manipulation tools to instantly parse, clean, and store that structured data into a local Xano table.
Implementing a conditional caching layer so that when a user requests their dashboard, Xano checks our local, optimized table first before ever considering an external API call.
Has anyone implemented a similar server-side caching and data-syncing routine for an external provider? Are there any hidden gotchas regarding memory execution limits during heavy JSON parsing inside background tasks that I should watch out for? Appreciate any advice or snippets you can share!