Development · integration · reverse engineeringOstwald · Alsace · Remote projectscontact@dtouch.fr
Sources · transformations · controls

Imports, ETL & data recovery

An import requires an understanding of the source, the transformation rules and the objects expected upon arrival. Dtouch prepares correspondence, checks and recoveries for the project.

Qualifying sources and correspondence

The data can come from files, a database, an export or an API. We look at fields, identifiers, types and relationships before defining processing.

The mapping specifies what corresponds to an existing object and what can be created or updated. Ambiguous values ​​are identified to avoid silently transforming uncertainty into reference data.

Preparing for transformations and controls

Date formats, units, separators and encodings require explicit rules. Fields calculated or resulting from matches must be explainable.

Tests on a sample make it possible to verify hypotheses before passing them to a batch. The results are compared to the source and the uses that will exploit the data.

Anticipating duplication and recovery

A resumption may be interrupted or require a second passage. Identifiers and update rules determine how to restart without multiplying objects.

The logs distinguish between processed, ignored, and rejected items. Errors must be attached to a line or object to allow targeted correction.

Validating the result in the career paths

The number of imported objects is not enough to evaluate the recovery. We verify relationships, research, documents and operations that depend on the transferred data.

The transmission of mappings, reports and reminder conditions allows the teams to understand and maintain the processing.

Understanding the exchanges

Connect applications and keep exchange control.

Example of a path with deferred processing: the connector prepares the data, organizes their transfer and keeps the results of the exchanges for monitoring and recovery.

Connect applications and keep exchange control.Enlarge the diagram
  1. 01

    Sources

    API, files and databases

    Identifiers • formats
    Incoming data ↓
  2. 02

    Adaptation

    Business correspondence

    Fields • units • references
    Transformed data ↓
  3. 03

    Processing queue

    Organize requests

    Priority • planning
    Tasks ready ▼
  4. 04

    Execution

    Treating tasks

    Controls • matches
    Writing in the target ▼
  5. 05

    Destination

    Application or ERP

    Objects • management rules
    Returns and variances ▼
  6. 06

    Follow-up and resumption

    Control returns

    Journal • status • reminder
Connect applications and keep exchange control.Enlarge the diagram
Functional diagram · synthetic reading of flows and control points.
Identify

Keep the reference of the source and that of the target application.

Transform

Describe correspondences and distinguish direct calls from deferred processing.

Take over

Find the state and context of an exchange to be retired.

Understand and prepare

A data recovery is being prepared with its difficult cases.

The lines that require a decision are part of the project: unknown reference, possible duplicate, missing document or price corrected by a user. We prepare the treatment to make these situations visible and enable targeted recovery.

A sample that represents your activity

A file of a few clean lines gives an incomplete view. The work sample must also represent the variants, particular characters, attachments and missing values encountered in the activity. For a supplier catalogue, it may include already known items, new references and category changes. People who use the data help to recognize cases where automatic creation is acceptable and those who request a decision.

Define what the import is allowed to modify

The same reference may already have a corrected description, a chosen price and an internal category. The mapping must specify which fields remain the source and which local information should be kept. The preparation distinguishes creation, updating and possible deactivation. In a business application, this rule protects the work of users while allowing the updating of the data received. Deletions and changes in sensitive status are explicitly decided.

Preparing for stocktaking and recovery

Tracking reconciles received items with items processed, refused or left on hold. It must make it possible to find the reference concerned and the reason for a deviation. A reminder can only include the necessary elements, with a rule defined for the data already present. The logs are designed for the people who will use the connector: information useful to the developer and a report readable by the team meet additional needs.

Deliverables that make treatment sustainable

The project can include the correspondence table, the transformation rules, a test batch and its results, then the conditions for transition to real data. The recipe checks the relationships and uses after import: search, file, document or order depending on the subject. Agedis prepares backups and system prerequisites when needed; Cloudlab completes the home pane of the application. The processing is transmitted with its resumption conditions and its known limits.

Your job deserves
the right tools.

A specific need, a software to evolve, a project to frame.

Exchange on your project
Expertise that complements each other

The right skills for the rest of your project.

Complementary expertise, the same high standards.