Data import via the IEF format

This page explains details on how to import data in IUCLID via the IIP using job information in the IIP Exchange format (IEF). One file usually contains many jobs, e.g. for many literature references for the IUCLID inventory or many documents (of the same or different type) for a specific dataset. The IEF format is a format designed to be used by applicants, to transfer data from inhouse systems to their inhouse IUCLID instance, prior to dossier generation and submission via the ECHA submission portal.

For a very detailed format description check your download page that you receive after your registration.

The format allows to

• use business values instead of IUCLID internal IDs and codes, for picklist values and references.
  • Supply references to a document / literature reference / reference substance by providing key business values. E.g for document references provide the document name. There is no need to know the UUID in the target instance – which may eventually be different for different target instances in a task force constellation for the same document.
  • Values for a picklist can be provided in clear text, e.g. “mg/l”
• simplify the data supply, e.g. a range can be supplied in one string instead of separation into up to 5 different fields
• be flexible in the representation, e.g. decimals can have either a “,” or “.” as delimiter.

In combination with the processing of the IEF data in the IIP tool, the IEF unfolds its full power:

• For each single entity, preview of the job data prior to application to IUCLID
• Preview of the anticipated operation – will a new instance be created or an existing updated
• Listing of “matching candidates” in the IUCLID target instance that qualify for being updated.
• Flexible display and comparison of both the job data and all matching candidate in full detail.
• Possibility to change the suggested update candidate and operation by the end user – or to delete the job.
• Link to the IUCLID native web interface from the IIP for the selected entity for immediate verification of any operation.

As the IIP also allows the export of all supported entities in IEF format, the format can also be used for data exchange, when native I6Z transfer mechanisms are to coarse.

In a future release

• the IIP will display differences between a job information and a matching existing entity in IUCLID on field level, including attachments.
• the user will be able to update any field in the the job data

In summary, this provides essential features for an applicant

• very transparent options to upload data for IUCLID
• to improve data quality and apply data governance rules on the data in the target IUCLID instance, e.g. by avoiding data duplicates in IUCLID that have the same core information (e.g. the same study ID for literature reference)

Processing of an IEF file in the IIP

An IEF file can have information to create or update many instances in IUCLID at once. Upon loading an IEF file into , the information in the IEF file is parsed and displayed as a list of jobs, dependent on the number of entities in the IEF file.

The following screenshot shows the user interface with jobs information loaded from an IEF file (here: to create literature reference entities in IUCLID):

A quick explanation of the main controls of the interface:

• In the upper left side you see the list of jobs (blue frames). This list shows the operation that the IIP suggests to perform dependent on the previous analysis of the data in the target IUCLID instance. E.g. in the above case the IIP suggest to update an existing entity, as it has found an existing literature reference entry with the same Study Number.
• Dependent on the already existing data in the target instance the IIP will suggest to either update an existing instance or create a new one. This is reflected in the first column “target update status” in the job list.
• In the lower left pane the IIP display you the matching candidates in the target instance that the IIP has found and suggests to update, if any have been found. Per job, the user can review the suggestions of the IIP and change or delete it (removing or changing the checkbox). The criteria of what the IIP suggests as candidate differ per entity type (see further below).
• On the right side, information of a selected item is displayed, always in the same mask, either from an instance in IUCLID or a job information. At any time the user can click on either the job or any of the matching candidates, to display the information in the job or the data of any of the matching candidates. At this time the job info has not been applied to IUCLID, but the user can preview the data in the same display as for the existing data in IUCLID. This allows a powerful comparison, the correct selection of the update candidate and a detailed preview of the changes that would occur when executing the particular job.
• To quickly navigate between the job information and the currently selected update candidate (if any), the user can also use the tabs on the top of the right side.
• To better differentiate information from jobs and from IUCLID, job listing or job details on the right have a blueish frame styling, and IUCLID entities use a reddish frame styling in listing and details.
• If the job information has an error, this will display in 2nd column “job status” – see e.g. the”bomb” icon in the screenshot above. Those jobs cannot be executed.
• If after the user’s review the job should not be executed at all, it can be deleted from the job list (button in the upper right corner).
• Upon completion of the user review, the user now can execute the jobs. Only then the actual operations are done, as suggested by the IIP or modified by the user.

A very similar process applies to all other entities that are to be imported. The only variations are

• the criteria how matching candidates are to be searched for (see further below)
• whether you can import directly (for entities in the IUCLID inventory) or where you need to select a target dataset first (for SUBSTANCE, MIXTURE, TEMPLATE entities)

How-To quick guide

As every supported entity can be both imported and exported in IEF format, the easiest way is to connect the IIP to a running IUCLID instance with test data, export some data (using the existing filters), reimport it, modify it and see the format in action, e.g. as follows:

• Navigate to the perspective “IUCLID entity management”, select “Substance” in the combobox and after display of the existing substances select a desired dataset. The documents in the dataset will be displayed in the lower pane. Optionally filter the documents using the filter text box.
• Export this document data in IEF format using the “Export in IEF format” button. Dependent on whether your data contains attachments or not, you will receive the IEF file in your Browser’s Download folder or in the export folder on the IIP server. In the latter case you need access to that export folder via your local File browser to see the exported IEF. For server installations please ask your IT admin what the name and location (e.g. as drive letter or UNC path) of this server folder is, as accessible for you from your PC.
• To test re-importing this data, move it to a directory below the import folder of the IEF. This is a similar directory accessible to both the user as well as the IIP.
• In the IUCLID web interface, create a new empty substance dataset. Note: As of now, the IIP cannot create the parent entities for datasets.
• In the IIP, open the “Import Job Management” perspective and navigate to the newly created substance dataset. Then, import the IEF file created in the the previous steps from the import folder. The left pane will populate with job data. To verify the jobs, click on each line to see details on the right pane.
• Then, you can click “Execute jobs” to actually import the jobs in the new dataset to execute those jobs and create or update entities in IUCLID. Please note the status information in the leftmost column changing during the execution. Use the mouse to hover above the symbols to see their meaning. With the checkbox above the left pane you can hide successfully terminated jobs, to only see jobs with error status. For successfully executed jobs you will get the IUCLID UUID identification and can click on this ID to open the instance in IUCLID directly. For errors you will be able to display the available error information.
• You can repeat the above import steps with the same IEF file – in this case the IIP will detect that IUCLID already contains the correct information and will not update the data in IUCLID. Please click on some jobs and you will see the “matching candidate(s)” in the left lower pane (see chapter below for a further explanation).
• Now you can try to change some of the data in the IEF file and observe the effects in terms of matching candidates and operation, dependent on your exact changes. The IEF will correctly validate the IEF for syntax correctness.

Supported IUCLID entities

The IEF supports the upload of metadata and attachments for all main IUCLID entities:

• From the IUCLID inventory:
  • Literature references (for GLP study reports and publications plus associated metadata)
  • Reference substance
  • Test material
• All document types in all datasets (substance, mixture, template).
  Other datasets (articles, categories) should also work, but have not been explicitly tested, as they are not relevant for PPP.

A brief overview of the IEF format

For each of the supported IUCLID entities, the IIP processes data in a specific IEF definition. The data has to be supplied in this predefined format. For the import into a dataset, one IEF file can contain data from different document types.

The core idea of the IEF is to use the IUCLID field names as header information in columns and add the data below in the respective columns. For nested or repeated data blocks (e.g. table rows) the data spans across multiple lines. As every supported entity can be both imported and exported, the best way to understand the IEF format is to use and modify existing IUCLID data and export the data in IEF format.

The IUCLID field names can be displayed in the IIP attribute mask, when you move the mouse on the type symbol before the label, as in the following example:

Each entity in IUCLID can contain single-value fields and multi-value fields (e.g. multiple attachments), the latter either as individual field or as part of a repeating block of fields, like a tabular row. Therefore, the input IEF file can contain multiple lines for one entity, where in each line the different values for one (or many) repeating attributes can be supplied.

For a detailed format description check your download page that you receive after your registration.

Job status, data validation and update

Each entity in the IEF will generate one job entry in the initial “New” status. When the IEF data of the entity cannot be correctly validated (e.g. wrong date format, referenced attachment cannot be found), then the job status is Error_Import (a bomb icon).

When the job list is processed, the status changes to “Processing” and then either to “Created”, “Updated” or “Error”.

Job lifetime and logging

Jobs are being processed on the IIP. The effect of a job execution is displayed in the web browser and also added to a log file that is created in the same directory and with the same filename as the originating IEF file used to create the jobs.

When the IIP browser tab is closed or refreshed (F5), then jobs in status “Processing” will continue to be processed and the results will be written to the log file. Other jobs will be deleted.

Matching candidates

For each individual job in the IEF, the IIP tries to find a matching candidate in IUCLID using specific attributes, in order to determine whether

• to update an existing instance in IUCLID (If exactly one matching candidate is found
• to create a new instance of that entity in IUCLID (If zero, two or more candidates are found)

Prior to job execution, the matching candidate(s) can be displayed in the lower left corner by clicking on each job. If there is one single matching candidate it will be highlighted and pre-selected. This selection can either be removed or changed to another suggested candidate.

The criteria on how matching candidates are retrieved is available in the online help of the IIP (see the blue (?) button)

Creation of IEF files

As input, the IIP processes a file called IEF (IUCLID Exchange Format) with a predefined structure – that can be generated from an inhouse reporting tool – plus referenced attachments (like the PDF for study reports). This allows companies to prepare data in IEF format from existing inhouse data / systems and then create or update a running IUCLID instance with this information in one batch, without having to manually enter the values in the IUCLID web interface field by field. This ensures better data quality and increases the efficiency of filling IUCLID. A format description of the IEF format can be found on your download page that you receive after your registration. Alternatively, every of the above entities can be exported in IEF format to generate samples in the correct format.

The creation or change of an IEF file with either an ASCII editor or Excel is not recommended.

The IEF format is dependent on and derived from the IUCLID format that is subject of change for every major IUCLID version (the next major version is scheduled for Q4/2023). Due to the format changes, you need to choose the correct IEF template, dependent on what IUCLID version you connect to. There is no separate download of templates – simply export a sample of an existing document in IUCLID to create a sample IEF file and then remove the data section.