The plugin adopts a version control based on a hashing system. Each pipeline has a version control, that is, it has a hashing key that allows to identify if the information subset (hashset) has still not been integrated or if it suffered any change in the data, requesting a new integration.
This hashing technique is applicable to an information segment (hashset), that is, it allows to identify the attributes that make up the hashing key (by ignoring integration requests when the entity suffers changes in attributes that are not a part of the key composition).
All pipelines have a hashing key (defined using user fields) on the source system and a hashing file (defined in format JSON), for the version control. If a physical entity (table) has several pipelines, you must create as many hashing keys as the number of pipelines that come from the table (you can eventually share the hashing file).
Composition of the Pipeline Hashing Key:
- SourceHash: attribute used to calculate the hash on the source system (calculated every time a reading request is performed);
- TargetHash: attribute used to preserve the hash value on the target system (updated every time a successful integration is performed);
- IsSync: attribute used on the integration lists as a conditional filter (this attribute specifies if the record has been integrated, it is controlled by comparing the values of the attributes “SourceHash” and “TargetHash”);
Composition of the Pipeline Hashing File:
- Algorithm: identifies the hashing algorithm;
- BaseEntity: identifies the base entity (main table);
- LyncEntities: identifies the entities related (including the main entity);
- Attributes: identifies the attributes that make up the hashing key (that is, used for the hash calculation);
- Filter: identifies the conditional filter to be applied for the hash calculation;
- FieldSourceHash: identifies the SourceHash attribute (when ignored, the system assumes SourceHash);
- FieldTargetHash: identifies the TargetHash attribute (when ignored, the system assumes TargetHash);
- FieldSyncHash: identifies the IsSync attribute (when ignored, the system assumes IsSync);
The following image presents the conceptual methodology of the hashing supported by the plugin.
The plugin distributes several hashing files examples in: VKPluginSAMPLESHASHETS, that you can explore and use when developing your pipelines.
