How to structure app to collect large amounts of data

In our business, we have to collect large amounts of data from our customers when they sign up. We have around 60 different forms (currently in JotForm), each with 200-300 fields each. The information the customer provides during the signup process determines which one of those 60 forms they need to fill out.

We’d also like to break each form into smaller pieces and have the progress saved. One of the biggest problems we have right now is people leaving the form because it’s so long, and then coming back to find that their information was not saved and they have to start over (JotForm has a save and continue feature, but if the customer doesn’t click the save button and email themselves a link, it doesn’t work.)

My first thought was to create a new data table for each form, and then direct the customer to a details page with the correct form after they sign up. I’ve tried a few different ways using Page Rules, Form Rules, and Javascript to show/hide components, but nothing seems to work well and the app structure quickly gets out of control and difficult to manage, especially if we start breaking the 60 forms down into smaller pieces so that each piece can be saved independently.

Has anyone dealt with something like this before? Is there a smart way to structure this or is it a situation where you just have to slog away at it?

Thanks in advance!

@mattb it’s a great question and building an app this big can only be done with proper planning. Glad you posted this question.

Can you elaborate a bit more on the conditions the signup uses to determine which forms need to be filled? It would get very messy (and lag) to have 60 form rules. Perhaps, we can find a better way to do this with the “Switch” pipe.

Are the forms very similar? No way to consolidate it? If you’re talking about ~15,000 fields (60 forms X 250 fields), this can easily be a huge bottleneck especially if you’re working with many calculated fields. If that’s not avoidable, perhaps we can export all the questions from the JotForms and use the Builder API to generate the tables and fields.

Until you add some more details I would say to do it along these lines:

  • Use switch pipe to set the value for which form to fill out.
  • Set that field back to signup/users form
  • Route them to their profile page with a static link that goes to the form they must fill out
  • Make the forms each multi-page detail pages and each page should update the ‘current_status’ status of this form.

I’m really just spitballing, once you add some more context I’d be able to clarify further and elaborate on what I mean. But, if you make this into 60 data tables you would need to manually do this 60 times! If the questions are very unique for each form, you must also keep in mind the field limitations per table.

I look forward to hearing back from you so we can figure this out together.

Hi @moe,

Thanks for jumping in!

During customer signup, there are 2-3 pieces of information the customer provides in order for us to determine which form they need to fill out. Those 2-3 pieces of information are (1) the county the property is located in, (2) the type of property, and (3) the city the property is located in (only for some areas where the county is not specific enough).

I haven’t dug into pipes yet, but the switch pipe sounds like it would be useful for this. I found out quickly that form rules were NOT the way to go!

The forms themselves are somewhat similar but still different enough that it would be difficult to consolidate them. Fields like the property address and acreage are the same across all of the forms. However, the majority of the fields are more location-specific and type-specific.

Theoretically, the most you could consolidate the forms would be into just four, one for each property type, and basically, just eliminate every non-unique question, but it makes more work for the customer because they would have to answer more questions than necessary, and more work for us since we would have to dig through the responses to pull out the information we actually need.

That being said, I’m still considering going that route if the alternative proves to be too much to build and manage. I’ve realized there’s a downside to everything, and at this point, I just want to make the best decision I can and move forward with something.

When you say this:

what do you mean by updating the ‘current_status’? Is that a way to save progress on a multi-page form?

Also, you mentioned field limitations per table… what ARE the limitations?

Thank you for your time!