Complex Data Manipulation Processes

App Development
1

Hi all, we have an app that requires some data manipulation and I would be happy to hear thoughts and suggestions as to how to do this in the best way in Tadabase.

Basic app structure:

  • Students table
  • Volunteering Locations table
  • Volunteering Assignments table

The first 2 tables are pretty basic/standard. The Volunteering Assignments table has columns:

  • Student (connection)
  • Location (connection)
  • date

So for example, the assignments for today would look like (imagine 200 records like this for each day):

  1. John, Farm A, 05.05.26
  2. Jack, Farm A, 05.05.26
  3. Will, Farm B, 05.05.26

Each day, the manager is responsible to make the assignments for all 200 students (so 200 records per day). Let’s say the manager logs in this morning, and decides to create all 200 assignments:

  1. Manager clicks “create empty records for today’s assignments”.
  2. This triggers a backend server function that creates 200 new records for all 200 students, with the date of today:
    1. John, [empty location], 05.05.26
    2. Jack, [empty location], 05.05.26
    3. Will, [empty location], 05.05.26
  3. The manager then starts assigning the locations (inline editing).
  4. When done, we have today’s assignments.

This is the easy part. Now, let’s say after step 2, the manager decides to click on “pre-fill today’s assignments with the data from date X”. We now have 200 “empty” assignment records waiting to be filled, but they are “irrelevant”. So what do we do? We trigger a backend function that:

  1. Batch updates all 200 current “empty” records to be inactive (API batch update)
  2. Fetches all 200 records from date X (API fetch)
  3. Prepares a CSV with the data from those records, but with today’s date instead.
  4. Imports the CSV into the assignments table (API import)

Notice that in this case we have “stale” records (inactive) that are obsolete, so we also need a scheduled task that will delete them from time to time.

I was wondering if Tadabase has built-in tools that I don’t know about, which we can use to improve this process. Ideally to rely on backend functions as less as possible.. but I don’t see any other way for now.

(PS - if the Import API feature had the ability to MATCH and UPDATE current records, and not only import all records blindly, that would make the process much cleaner, but such an option only exists in the manual import process).

2

Hi @crumbs, I think your setup is good, but it is more complex than it needs to be.

Your current flow is essentially: Create 200 records > Decide to copy another day > Deactivate those 200 > Recreate 200 via > CSV import > Clean up later.

That works, but it creates extra problems of inactive junk records, CSV generation/import, and a scheduled cleanup.

You already have the correct 200 records for today, but they just don’t have a Location yet. Instead of replacing them, you can simply copy the Location from date X into today’s existing records. This copies values between matching records, it doesn’t recreate those records, just updates them. No new records are created, you are only updating the existing 200 records.

You can do this using Tadabase REST API Pipe using these functions.

  • Get All Records
  • Filtering by Specific Dates
  • Update Existing Record

This is the process: Get assignments from date X > Get today’s assignments > For each source record, find today’s record where Student = same Student > Update today’s Location field.

To build this in Tadabase

  1. Get Source Assignments
  • Action: Get All Records
  • Table: Volunteering Assignment

2. Filter: Date = source date (this should be passed into the pipe as a parameter, e.g. source_date)

  1. Return fields should include Assignment ID, Student (connection ID), and Location (connection ID)

  2. Get Today’s Assignments

  • Action: Get All Records
  • Table: Volunteering Assignments
  • Filter: Date = today (or a passed parameter like today_date if needed)
  • Return fields should include Assignment ID and Student (connection ID)
  1. Update Assignment
  • Action: Update Existing Record
  • Table: Volunteering Assignments
  • Field: Location = source Location

Inputs

  • record_id (today’s matching record - this comes from the Today Assignments pipe)
  • location_id (from the source record - this comes from the Source Assignments pipe)

The matching must be done using the Student connection ID (not the student name), meaning source.Student ID must equal today.Student ID before performing the update.

The auto-created Record ID is not always easily accessible or usable in pipes, especially when you need to pass it between steps for matching or updates. A common workaround is to create a text field (e.g. RID, text 50). Then add a record rule on create to populate that field with the record’s Record ID. This gives you a reliable field you can return in pipes and use for matching or updates.

3

Hey @Co1in , thanks for your response!

So I thought about the “update existing records by API” approach, however I thought the problem with it is the duration of the process.

Fetching 200 records (date X), fetching 200 records (today), and most critically - updating 200 records - would take at least 3.5 minutes (calculating one API call per second). I haven’t used the Tadabase REST API pipe, but I do have experience with using the API directly. Would using the pipes be much faster by any chance?

My reasoning for using the “import by API” method is that it should take a few seconds compared to a few minutes of updating records individually through API.

4

Hey, you are spot on with your thoughts about update timing.

I would not assume a Tadabase REST API pipe is materially faster if it is still updating records one at a time. Under the hood, you are still effectively performing individual record operations, so the total time tends to scale with record count.

From testing, record-by-record operations (whether via API or pipes) are typically in the ~1–2 seconds per record range. That would put 200 updates somewhere in the few-minute range rather than a few seconds, depending on payload size, pipe complexity, and execution context.

The key distinction is:

  • Pipes give convenience and keep logic inside Tadabase
  • API/import provides potential for bulk-style processing

So, your assumption is generally correct:

  • Per-record updates (pipe or API) are slower, linear scaling
  • Bulk import-style operations are significantly faster for large batches.

Your import-by-API method will almost certainly outperform a per-record update approach.

That said, the trade-off is:

  • The Import approach is faster, but more complex (CSV handling, cleanup logic)
  • Pipe/update approach is cleaner and safer, but slower.

One other thing to consider is whether you need real-time execution. If not, scheduling the update process (or running it in the background) can remove the user-facing performance concern while keeping the cleaner architecture.

Also worth noting: the browser-based JS API tends to hit limits much faster than server-side/API approaches, so you were right to move away from that.

Col

5

Awesome, thanks for your input!