Hello - I've been struggling with this for a few days and hope someone might be able to think of a more efficient way to tackle this problem.
In my app, I have a user CSV input that includes the possibility for dynamic variables. For example, a user could upload a CSV with the following content:
As you can see, the CSV has three data columns, and one column that is referencing the variables in the data columns.
Ideally, in Xano, I'd be able to convert this information into a CSV file where the variables have been replaced:
Also, I am trying to make the columns dynamic. So another user could upload a CSV with an additional dynamic variable (animal, for example):
Here, I'd like my function to recognize the new variable, and replace it as well:
While I've been able to solve this with small datasets (<1000 line items), I'm running into issues when the CSV uploads reach 10,000+ line items (even as a background task).
I think the double-loop approach I am utilizing might be highly inefficient. To describe my approach:
First I return the user uploaded CSV via CSV Stream
I then loop through the stream on each line item
Next I do another loop (within the first) to get the key / value pairs of the line item's columns
Then I replace each variable in the message with the matching key / value for that line item
I then store the updated message in a variable and continue to the next loop
Once I've looped through all 10k+ line items and replaced each variable, I save the file as a CSV attachment in my database
Thanks in advance for any help. Please let me know if you need more info.