Best Practices and Common Pitfalls

Best Practices and Common Pitfalls

Building a successful import process involves more than just correct configuration; it also requires a solid workflow to prevent common errors. This guide outlines best practices for testing, data validation, and file management to help you create more robust and reliable imports.

Always Test with a Small File First

Before running an import with thousands of rows, create a small test file with just 5-10 representative rows of data. Running a manual import with this small file allows you to quickly verify your entire setup.
  1. Benefit: It's much easier to check the results on a handful of records and examine a small cache file. This allows you to catch errors in configuration, mapping, or data transformation before they affect a large amount of data.
  2. Common Pitfall: Running a large, untested import for the first time, which can lead to widespread incorrect data, thousands of unwanted new records if misconfigured, or a long wait just to find out a simple header was misspelled.

Validate Your Data Before You Upload

The Import Module expects clean, consistently formatted data. Most import failures that are not related to configuration are due to problems within the source file itself.
  1. Best Practice: Before uploading, open your file and check for common issues:
    1. Consistent Date Formats: Ensure every single date matches the format you specified in the configuration.
    2. Clean Identifiers: Look for extra spaces, special characters, or inconsistent formatting in your Identifier Column.
    3. Correct Repeater Formatting: If importing into repeaters, double-check that your values are separated by actual tab characters and that alignment is correct for blank values.
  2. Common Pitfall: Assuming the data exported from another system is perfectly clean. Data exports often contain hidden characters, mixed date formats, or other inconsistencies that will cause errors during the import.

Organize Your SFTP Folders

A well-organized SFTP site makes managing your import files much easier. Instead of placing all files in a single directory, group them into folders based on their purpose, the form they populate, or the system they come from.
  1. Best Practice: Use a clear and consistent folder structure. For example:
    1. /imports/IEP/
    2. /imports/ReportCards/
    3. /imports/Assessments/
  2. Common Pitfall: Placing dozens of unrelated import files in the same folder. This can lead to confusion, make it difficult to find the correct file, and increase the risk of pointing a configuration to the wrong source.

Choose the Right Tool for the Job

The module offers several ways to handle one-to-many data relationships. Choosing the correct one is key.
  1. Use a Repeater when... you have a list of similar items that can have a variable number of entries for each person (e.g., emergency contacts, medications).
  2. Use Indexed Expressions when... you have a fixed number of related entries that need to go into separate, static fields on your form (e.g., Guardian 1, Guardian 2, Guardian 3).
  3. Use Record Matching when... you have multiple, distinct records for the same person that are identified by a unique key (e.g., report cards for different school years or terms).
  4. Common Pitfall: Trying to use indexed expressions to populate a repeater, or trying to use the repeater method to populate fixed guardian fields. These features solve different problems and are not interchangeable.

    • Related Articles

    • Introduction to Import Data Transformation

      Sometimes, the data in your source file isn't in the exact structure or format that your forms require. Data Transformation is an optional but powerful step in the import process that allows you to clean, restructure, and manipulate your source data ...

    • Overview of the Import Process

      The Import Module is a powerful tool for bringing external data directly into your forms. It streamlines workflows, reduces manual entry, and enhances data accuracy by using structured files like CSV, TSV, XML, or JSON to automatically populate or ...

    • Preparing a CSV or TSV File for Import

      A correctly formatted CSV (Comma-Separated Values) or TSV (Tab-Separated Values) file is essential for a successful import. While these are standard formats, your file must adhere to specific rules to ensure data is processed accurately. Basic File ...

    • Running a Manual Import

      A manual import is a bulk import that you initiate yourself from the Import Module. It's the primary method for performing one-time data loads, testing a new configuration, or updating a specific subset of records. There are two ways to start a ...

    • Import Error Code Reference

      When an import fails, the system will log an error code to help you diagnose the problem. This guide explains what each error code means and provides step-by-step instructions on how to resolve the issue. E0001: Cannot Read File Problem The system ...