Master Data Storage
Last updated
Last updated
The Master Data Management plugin permits your SanteDB CDR instance to store multiple copies of a single record based on the source system which provided the data. The MDM services are available on the following resources:
Patient
Place
Provider
Materials (and Manufactured Materials)
Device Entities
Persons
The use of Master Data Management services require a compatible Matching and Merge service to operate. This is how the MDM solution detects candidate master records.
Within the data storage persistence layer, the MDM plugin will update/create locals using whatever underlying class was sent from the caller. Each caller is provided a complete copy of the record which they can update/edit.
The MDM plugin then determines:
Is there a MASTER record which the IRecordMatching service has identified as positive match?
Is the configuration enabled to automatically link these records?
If no MASTER is found, or the configuration does not permit auto-linking then a new MASTER of the base type is constructed (Entity in case of Patient, Place, Material, etc. or Act in the case of Administration, Procedure, Observation, etc.) and links the LOCAL record to this newly created master instance using the relationship type MDM-MasterRecord.
The master record should have no data associated with it. It is merely an anchor which is used to collect the local records. Typically a master record can have extensions, notes, identifiers, names, etc. associated with it, however these attributes should apply to the anchor point rather than for establishing ROT data.
In the case where a match between a LOCAL and MASTER record is ambiguous (i.e. does is a Probable match or when Auto-Link is disabled) the link between records is considered a candidate and the local is related to the MASTERs which it is a candidate for using the MDM-CandidateLocal relationship type.
In the scenario below, Good Health Clinic has registered John Doe born Jan 1 1980 with ID 1230493. The MDM layer has constructed a master record. A subsequent registration by HIV Health Clinic for John Doe, born sometime in 1980 with ID 3029402 as registered after.
Assuming that the match configuration was setup such that the HIV record had a match score below "DEFINITE" match, the MDM plugin would establish a new MASTER which contains one LOCAL. That LOCAL would also be related to the MASTER from Good Health Clinic with relationship type MDM-CandidateLocal.
This candidate link exists until:
A subsequent update from either the HIV Health Clinic (to record 3029402) or Good Health Clinic (to record 1230493) means that the patients no longer meet threshold for match.
The match configuration is updated, which scores the link below the threshold for probable match.
An administrator establishes a new link between the HIV Clinic and the MASTER of MDM-IgnoreCandidate (which indicates a known false-match).
An administrator manually indicates that the HIV Health Clinic record is a known match to the MASTER (in which case the MDM-MasterRecord link is removed and updated (as shown below).
The MDM solution permits the establishing of a Record Of Truth (ROT). The Record Of Truth is a new local instance which belongs to the SYSTEM and can only be updated when the security principal is permitted MDM Record Of Truth.
A ROT record is a special pointer at a LOCAL whereby the LOCAL points to the MASTER record, and the MASTER points at the ROT with relationship type of MDM-RecordOfTruth.
It is possible to manually elevate an existing LOCAL as a ROT record, however this is not yet supported on user interfaces available.
It is a commonly asked how the solution operates when a sensitive LOCAL record is shared with common health infrastructure. In the case of SanteDB's iCDR, when a MASTER record has multiple LOCALS the information returned to a caller is a synthesized copy of data within the LOCAL records.
The synthetization process uses a most-restrictive policy enforcement mechanism before considering a local for inclusion in the result. In the example illustrated above, the information sourced from the HIV Health Clinic is tagged as TABOO information.
When a caller asks the CDR for the MASTER, the MDM service will call the IPolicyDecisionService and ask for a decision on each of the flagged policies in each of the locals and will return either:
The UNION of data from both Good Health Clinic and HIV Health Clinic when TABOO has a GRANT, or
Only data from Good Health Clinic when TABOO policy outcome is DENY, or
Only the data from Good Health Clinic and an indicator of elevation required (i.e. indicator more data is present) when the TABOO policy outcome is ELEVATE.
The MdmDataManagementService should be configured in your application context. Upon start, this daemon will read the contents of the ResourceMergeConfigurationSection to determine the appropriate instances of MdmResourceListener<T> to construct and register with the provider.
For example, given a configuration as provided below
The MdmDataManagementService on start would register two resource listeners for Patient and Place. It would furthermore, instruct each resource listener to use the specified match configuration for detecting candidate local<>master linkages.
The resource listener instances created by the data management service subscribe to the repository events for the indicated type. The resource listener receives an event broadcast from the repository service whenever an event occurs and takes appropriate actions.
The Saving and Inserting events of the repository service are subscribed on service start. Whenever data through a repository is being actioned the events are raised and the following actions occur on the MDM Resource Listener:
The instances are validated through OnPrePersistenceValidate() method. Here the process ensures:
The caller is not attempting to update a MASTER record. If it is, the appropriate LOCAL for that principal is exchanged.
If the caller is attempting to create a RECORD OF TRUTH , it has appropriate permission to do so
The Saving() and Inserting() event handlers are called next. These methods:
Call the matching service and attempt to establish whether the LOCAL is a candidate for an existing record
Appropriately obsolete/create resource links between the objects.
Cancels the caller (i.e. overwrites the default call path)
Returns a BUNDLE which is treated as a transaction on the persistence layer.
When a repository's Find() or Get() methods are called the repository will call the Retrieving and Querying event handlers. The MDM resource listener for that repository will perform the following operations:
Cancel the default path of the repository
Re-write the HDSI expression tree to query attributes related to the ROT or a LOCAL
The resource listener also subscribes to the Retrieved and Queried event handlers where any local objects which were fetched from the underlying data persistence layer are synthesized using their MASTER links (and appropriate policies applied).
The following section outlines several common behaviors and patterns to assist in understanding how MDM works in SanteDB.
When the CDR receives a request to create a new entity which is under MDM control, as shown below.
The source record is issued a new UUID
The MDM layer will attempt to determine if the source record matches any MASTER records that currently exist within the database (using the resource merge configuration).
If no matching MDM MASTER is found, then the a new MASTER record is created and the SOURCE linked to the MASTER with its own UUID.
The relationship is indicated with RelationshipType MDM-MASTER
The classification of this relationship is AUTO
Any future queries based on the demographics will result in the MASTER_A record being returned and synthesized from the source records according to the synthesization rules.
When the CDR receives a registration request for a new object which is under MDM control, such as shown below:
The source record is issued as new UUID
The IRecordMatchService is called with the specified configurations. If this process classifies a record as MATCH and AutoLink is turned on for that match configuration
The SORUCE is linked to the MASTER with relationship MDM-Master
The classification of this relationship is AUTO
The weighting of the match is stored
A query for EID A123 would now result in a single result with the synthesized data from both sources and links to the sources where the data was obtained.
When the CDR receives a request to create an object with similar attributes however as above, however:
The attribute score from the fuzzy matching engine falls below the threshold of a definite match, or
The automatic linking feature for the match configuration is disabled
Such as the patient illustrated below.
The behavior is as follows:
The SOURCE is issued a new UUID
A new MASTER is created with a new UUID
The SOURCE is linked to the new MASTER
The relationship type is MDM-Master
The classification of the relationship is AUTO
The SOURCE is linked to the suspected MASTER
The relationship type is MDM-CandidateLocal
The classification of the relationship is AUTO
The score of the match is stored as the relationship strength
When the CDR receives a request to update a source record, it does so against the source (or if the update was attempted against the MASTER the SOURCE is created or located). When this occurs, the matching is re-run and if determined that the records now match the relationships are re-calculated.
For example, given this update to SOURCE_C.
The SOURCE_C record would be updated to match and the matching re-run, it may be determined that SOURCE_C is in fact the same person as MASTER_A
In this case, SOURCE_C is DETACHED from MASTER_C and then attached to MASTER_A.
The CDR also provides interfaces for manually reconciling candidate matches. Take for example, an instruction to reconcile SOURCE_C and MASTER_A.
In this method (also known as a LOCAL>MASTER merge) the MDM-Candidate link is translated into a MDM-Master and the classification set to VERIFIED.
Any operation which removes all MDM-Master links from a MASTER record will result in the automatic obsolete of the MASTER record.
Callers can directly manipulate the relationships using the IRepositoryService<EntityRelationship> services to implement custom logic for merging and linking. By default the MDM layer registers IRecordMergeService<T> instances which implement a default behavior for external calls to MERGE data.
Victim Type | Survivor Type | No Permission | Write MDM | Merge MDM |
LOCAL | LOCAL | Local Merge | Local Merge | Local Merge |
LOCAL | MASTER | Local Merge | Local/Master Relink | Local/Master Relink |
MASTER | MASTER | Local Merge | Local/Master Relink | Master Merge |
A LOCAL>MASTER link operation is initiated by calling Merge() in IRecordMergeService<T> , and passing a UUID of a MASTER record and the linked duplicate UUIDs that are either:
The explicit UUIDs of local records (i.e. the UUID pointing at the local record), or
The UUID of masters where the caller does not have Merge MDM Master permission
This operation works by unlinking the local/duplicate record relationships (of MDM-Master ) and re-establishing the link on the locals to the surviving master. This is a form of linking where neither the master nor the local records are merged.
For example, if a principal has Write MDM Master permission, and issues the following ADT^A40 :
The process would be:
Load the record XXXXX in domain TEST (MASTER)
Load the record YYYYY in domain TEST (MASTER)
Call Merge() for MASTER>MASTER
Client has Write MDM Master so record XXXXX remains the target of the merge.
Client does not have Merge MDM Master so record YYYYY is resolved to LOCAL record for TEST_HARNESS at site TEST (LYYYY)
Merge proceeds as LOCAL>MASTER
LYYYY is disconnected from master YYYY
LYYYY is attached to master XXXX
If no further locals point at YYYY it is obsoleted.
A MASTER to MASTER merge is the process whereby one master record is merged into another, and all associated LOCAL relationships are migrated to the survivor. Callers may only initiate a master merge when the principal carries the Merge MDM Master policy permission, otherwise an alternate merge strategy is used.
Giving applications using the HL7v2 or HL7 FHIR interfaces permission for Merge MDM Master is not recommended as any use of a business identifier in those messages would result in a master merge, since the query stage loads master records on those interfaces.
The process for performing a master merge is:
Load record XXXX in domain TEST (MASTER)
Load record YYYY in domain TEST (MASTER)
Call Merge() for MASTER>MASTER
Client has Write MDM Master so record XXXX remains target of merge
Client has Merge MDM Master so record YYYY remains source of merge
Merge proceeds:
All MDM-Master associations previously pointing at YYYY are rewritten to XXXX
All identifiers directly on YYYY master record are copied to XXXX
YYYY is obsoleted
A local merge occurs when a source system indicates it has resolved duplicates in its own database issues an appropriate merge request to the iCDR. The local merge is the default operation whenever the caller is not granted explicit permission to perform MDM merges on the server.
The process for performing a local merge is:
Load record XXXX in domain TEST (MASTER)
Load record YYYY in domain TEST (MASTER)
Call Merge() for MASTER>MASTER
Client does not haveWrite MDM Master so a local record for XXXX is resolved (LXXXX)
Client does not have Merge MDM Master so a local record for YYYY is resolved (LYYYY)
If LYYYY and LXXXX were resolved, then the merge occurs:
The record LYYYY is obsoleted
All identifiers from LYYYY are copied to LXXXX so that any query to LYYYY are resolved to the survivor
