QOF  0.8.7
Data Structures | Files | Enumerations
Merge: Merging QofBook structures
Query Object Framework

Data Structures

struct  QofBookMergeRule
 One rule per entity, built into a single GList for the entire merge. More...
struct  QofBookMergeData
 mergeData contains the essential context data for any merge. More...

Files

file  qofbookmerge.h
 

API for merging two QofBook structures with collision handling.


Enumerations

enum  QofBookMergeResult {
  MERGE_UNDEF, MERGE_ABSOLUTE, MERGE_NEW, MERGE_REPORT,
  MERGE_DUPLICATE, MERGE_UPDATE, MERGE_INVALID
}
 Results of collisions and user resolution. More...

qof_book_merge API

typedef void(* QofBookMergeRuleForeachCB )(QofBookMergeData *, QofBookMergeRule *, guint)
 Definition of the dialogue control callback routine.
QofBookMergeDataqof_book_merge_init (QofBook *importBook, QofBook *targetBook)
 Initialise the QofBookMerge process.
void qof_book_merge_rule_foreach (QofBookMergeData *mergeData, QofBookMergeRuleForeachCB callback, QofBookMergeResult mergeResult)
 Dialogue Control Callback.
QofBookMergeDataqof_book_merge_update_result (QofBookMergeData *mergeData, QofBookMergeResult tag)
 called by dialogue callback to set the result of user intervention
gint qof_book_merge_commit (QofBookMergeData *mergeData)
 Commits the import data to the target book.
void qof_book_merge_abort (QofBookMergeData *mergeData)
 Abort the merge and free all memory allocated by the merge.

Detailed Description

Collision handling principles.

  1. Always check for a ::GUID first and compare. qofbookmerge only accepts valid QofBook data and therefore ALL objects in the import book will include valid GUID's.
  2. If the original import data did not contain a GUID (e.g. an external non-GnuCash source) the GUID values will have been created during the import and will not match any existing GUID's in the target book so objects that do not have a GUID match cannot be assumed to be MERGE_NEW - parameter values must be checked.
  3. If import contains data from closed books, store the data from the closed books in the current book as active. i.e. re-open the books.

More information is at http://code.neil.williamsleesmill.me.uk/

Each foreach function uses g_return_if_fail checks to protect the target book. If any essential data is missing, the loop returns without changing the target book. Note that this will not set or return an error value. However, g_return is only used for critical errors that arise from programming errors, not for invalid import data which should be cleaned up before creating the import QofBook.

Only qof_book_merge_update_result and qof_book_merge_commit return any error values to the calling process. qof_book_merge_init returns a pointer to the QofBookMergeData struct - the calling process needs to make sure this is non-NULL to know that the Init has been successful.

Todo:
Prior to libqof2 rationalise internal variable names.

Typedef Documentation

Definition of the dialogue control callback routine.

All MERGE_REPORT rules must be offered for user intervention using this template.
Commit will fail if any rules are still tagged as MERGE_REPORT.

Calling processes are free to also offer MERGE_NEW, MERGE_UPDATE, MERGE_DUPLICATE and MERGE_ABSOLUTE for user intervention. Attempting to query MERGE_INVALID rules will cause an error.

For an example, consider test_rule_loop, declared as:

void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder);
void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder)
{
g_return_if_fail(rule != NULL);
g_return_if_fail(mergeData != NULL); printf("Rule Result %s", rule->mergeType);
qof_book_merge_update_result(mergeData, rule, MERGE_UPDATE);
}

The dialogue is free to call qof_book_merge_update_result in the loop or at the end as long as the link between the rule and the result is maintained, e.g. by using a GHashTable.
The parameters are:

  • data : pointer to the QofBookMergeData metadata context returned by init.
  • rule : pointer to the QofBookMergeRule that generated the collision report
  • remainder : guint value returned from g_slist_length for the number of other rules remaining with the same result. This might be useful for a progress dialogue, it might not. When updating MERGE_REPORT, remainder must equal zero before calling qof_book_merge_commit or the import will abort.

If the dialogue sets any rule result to MERGE_INVALID, the import will abort when qof_book_merge_commit is called. It is the responsibility of the calling function to handle the error code from qof_book_merge_commit, close the dialogue and return. The merge routines in these files will already have halted the merge operation and freed any memory allocated to merge structures before returning the error code. There is no need for the dialogue process to report back to QofBookMerge in this situation.

Definition at line 321 of file qofbookmerge.h.


Enumeration Type Documentation

Results of collisions and user resolution.

All rules are initialised as MERGE_UNDEF. Once the comparison is complete, each object within the import will be updated.

MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE can be reported to the user along with all MERGE_REPORT objects for confirmation. It may be useful later to allow MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE to not be reported, if the user sets a preferences option for each result. (Always accept new items: Y/N default NO, ignores all MERGE_NEW if set to Y etc.) This option would not require any changes to qofbookmerge.

MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE are only actioned after conflicts are resolved by the user using a dialog and all MERGE_REPORT objects are re-assigned to one of MERGE_NEW, MERGE_DUPLICATE or MERGE_UPDATE. There is no automatic merge, even if no entities are tagged as MERGE_REPORT, the calling process must still check for REPORT items using qof_book_merge_rule_foreach and call qof_book_merge_commit.

MERGE_INVALID data should be rare and allows for user-abort - the imported file/source may be corrupted and the prescence of invalid data should raise concerns that the rest of the data may be corrupted, damaged or otherwise altered. If any entity is tagged as MERGE_INVALID, the merge operation will abort and leave the target book completely unchanged.

MERGE_ABSOLUTE is only used for a complete match. The import object contains the same data in the same parameters with no omissions or amendments. If any data is missing, amended or added, the data is labelled MERGE_UPDATE.

Every piece of data has a corresponding result. Only when the count of items labelled MERGE_REPORT is equal to zero are MERGE_NEW and MERGE_UPDATE items added to the existing book.
MERGE_DUPLICATE items are silently ignored. Aborting the dialogue/process (by the user or in a program crash) at any point before the final commit leaves the existing book completely untouched.

Enumerator:
MERGE_UNDEF 

default value before comparison is made.

MERGE_ABSOLUTE 

GUID exact match, no new data - ignore

MERGE_NEW 

import object does not exist in the target book - add

MERGE_REPORT 

import object needs user intervention - report

MERGE_DUPLICATE 

import object with different GUID exactly matches existing GUID - ignore

MERGE_UPDATE 

import object matches an existing entity but includes new or modified parameter data - update

MERGE_INVALID 

import object didn't match registered object or parameter types or user decided to abort - abort

Definition at line 125 of file qofbookmerge.h.


Function Documentation

void qof_book_merge_abort ( QofBookMergeData mergeData)

Abort the merge and free all memory allocated by the merge.

Sometimes, setting MERGE_INVALID is insufficient: e.g. if the user aborts the merge from outside the functions dealing with the merge ruleset. This function causes an immediate abort - the calling process must start again at Init if a new merge is required.

Definition at line 937 of file qofbookmerge.c.

{
    QofBookMergeRule *currentRule;

    g_return_if_fail (mergeData != NULL);
    while (mergeData->mergeList != NULL)
    {
        currentRule = mergeData->mergeList->data;
        g_slist_free (currentRule->linkedEntList);
        g_slist_free (currentRule->mergeParam);
        g_free (mergeData->mergeList->data);
        if (currentRule)
        {
            g_slist_free (currentRule->linkedEntList);
            g_slist_free (currentRule->mergeParam);
            g_free (currentRule);
        }
        mergeData->mergeList = g_list_next (mergeData->mergeList);
    }
    g_list_free (mergeData->mergeList);
    g_slist_free (mergeData->mergeObjectParams);
    g_slist_free (mergeData->targetList);
    if (mergeData->orphan_list != NULL)
        g_slist_free (mergeData->orphan_list);
    g_hash_table_destroy (mergeData->target_table);
    g_free (mergeData);
}
gint qof_book_merge_commit ( QofBookMergeData mergeData)

Commits the import data to the target book.

The last function in the API and the final part of any QofBookMerge operation.

qof_book_merge_commit will abort the entire merge operation if any rule is set to MERGE_INVALID. It is the responsibility of the calling function to handle the error code from ::qof_book_mergeCommit, close the dialogue and return. qof_book_merge_commit will already have halted the merge operation and freed any memory allocated to all merge structures before returning the error code. There is no way for the dialogue process to report back to qof_book_merge in this situation.

qof_book_merge_commit checks for any entities still tagged as MERGE_REPORT and then proceeds to import all entities tagged as MERGE_UPDATE or MERGE_NEW into the target book.
This final process cannot be UNDONE!

Parameters:
mergeDatathe merge context, QofBookMergeData*
Returns:
  • -2 if any rules are tagged as MERGE_INVALID
    • mergeData will have been g_free'd).
    • note that this will be before any operations are done on the target QofBook.
  • -1 if mergeData is invalid or no merge has been initialised with qof_book_merge_init - the calling process must check the value of mergeData
  • +1 if some entities are still tagged as MERGE_REPORT - use ::qof_book_merge_update_rule and try again (mergeData is retained).
  • 0 on success - mergeData will have been freed.

Definition at line 998 of file qofbookmerge.c.

{
    QofBookMergeRule *currentRule;
    GList *check, *node;

    g_return_val_if_fail (mergeData != NULL, -1);
    g_return_val_if_fail (mergeData->mergeList != NULL, -1);
    g_return_val_if_fail (mergeData->targetBook != NULL, -1);
    if (mergeData->abort == TRUE)
        return -1;
    check = g_list_copy (mergeData->mergeList);
    g_return_val_if_fail (check != NULL, -1);
    for (node = check; node != NULL; node = node->next)
    {
        currentRule = node->data;
        if (currentRule->mergeResult == MERGE_INVALID)
        {
            qof_book_merge_abort (mergeData);
            g_list_free (check);
            return (-2);
        }
        if (currentRule->mergeResult == MERGE_REPORT)
        {
            g_list_free (check);
            return 1;
        }
    }
    g_list_free (check);
    qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
        MERGE_NEW, mergeData);
    qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
        MERGE_UPDATE, mergeData);
    /* Placeholder for QofObject merge_helper_cb - all objects
       and all parameters set */
    while (mergeData->mergeList != NULL)
    {
        currentRule = mergeData->mergeList->data;
        g_slist_free (currentRule->mergeParam);
        g_slist_free (currentRule->linkedEntList);
        mergeData->mergeList = g_list_next (mergeData->mergeList);
    }
    g_list_free (mergeData->mergeList);
    g_slist_free (mergeData->mergeObjectParams);
    g_slist_free (mergeData->targetList);
    if (mergeData->orphan_list != NULL)
        g_slist_free (mergeData->orphan_list);
    g_hash_table_destroy (mergeData->target_table);
    g_free (mergeData);
    return 0;
}
QofBookMergeData* qof_book_merge_init ( QofBook importBook,
QofBook targetBook 
)

Initialise the QofBookMerge process.

First function of the QofBookMerge API. Every merge must begin with init.

Requires the book to import (QofBook *) and the book to receive the import, the target book (QofBook *). Returns a pointer to QofBookMergeData which must be checked for a NULL before continuing.
Process:

  1. Invoke the callback ::qof_book_merge_foreach_type on every registered object class definition.
  2. Callback obtains the registered parameter list for each object type. This provides run time access to all registered objects and all object parameters without any changes to QofBookMerge - no registered object or parameter is omitted from any merge operation.
  3. Use qof_object_foreach to invoke the callback ::qof_book_merge_foreach, one object at a time on every instance stored in mergeBook. This is the first point where real data from the import book is accessed.
  4. qof_book_merge_foreach obtains the ::GUID for the object from the import book and runs the first check on the original book, checking for any exact GUID match. With the full parameter list, the rules for this object can be created. If there is a GUID match, the data in each parameter of the import object is compared with the same semantic object in the original book. If there is no GUID in the import object or no GUID match with the original book, the original book is searched to find a parameter match - checking for a MERGE_DUPLICATE result.
  5. ::qof_book_merge_compare sets the QofBookMergeResult of the comparison.
  6. Inserts the completed rule into QofBookMergeData::mergeList GSList.
Returns:
NULL in case of error, otherwise a QofBookMergeData* metadata context.

Definition at line 900 of file qofbookmerge.c.

{
    QofBookMergeData *mergeData;
    QofBookMergeRule *currentRule;
    GList *check;

    g_return_val_if_fail ((importBook != NULL)
        && (targetBook != NULL), NULL);
    mergeData = g_new0 (QofBookMergeData, 1);
    mergeData->abort = FALSE;
    mergeData->mergeList = NULL;
    mergeData->targetList = NULL;
    mergeData->mergeBook = importBook;
    mergeData->targetBook = targetBook;
    mergeData->mergeObjectParams = NULL;
    mergeData->orphan_list = NULL;
    mergeData->target_table =
        g_hash_table_new (g_direct_hash, qof_book_merge_rule_cmp);
    currentRule = g_new0 (QofBookMergeRule, 1);
    mergeData->currentRule = currentRule;
    qof_object_foreach_type (qof_book_merge_foreach_type, mergeData);
    g_return_val_if_fail (mergeData->mergeObjectParams, NULL);
    if (mergeData->orphan_list != NULL)
        qof_book_merge_match_orphans (mergeData);
    for (check = mergeData->mergeList; check != NULL; check = check->next)
    {
        currentRule = check->data;
        if (currentRule->mergeResult == MERGE_INVALID)
        {
            mergeData->abort = TRUE;
            return (NULL);
        }
    }
    return mergeData;
}

Dialogue Control Callback.

This function is designed to be used to iterate over all rules tagged with a specific QofBookMergeResult value.

Parameters:
callbackexternal loop of type QofBookMergeRuleForeachCB
mergeResultQofBookMergeResult value to look up.
mergeDataQofBookMergeData merge context.

Note : MERGE_NEW causes a new entity to be created in the target book at commit which is then assigned as the targetEnt of that rule. If mergeResult == MERGE_NEW, the rules returned by qof_book_merge_rule_foreach will have a NULL set for the targetEnt. This is because commit has not yet been called and no changes can be made to the target book. The calling process must handle the NULL targetEnt and NOT call any param_getfcn routines for the target entity. The import entity is available for display.

Uses qof_book_get_collection with the QofBookMergeRule::mergeType object type to return a collection of QofEntity entities from either the QofBookMergeData::mergeBook or QofBookMergeData::targetBook. Then uses qof_collection_lookup_entity to lookup the QofBookMergeRule::importEnt and again the qof_book_mergeRule::targetEnt to return the two specific entities.

Definition at line 1050 of file qofbookmerge.c.

{
    struct QofBookMergeRuleIterate qiter;
    QofBookMergeRule *currentRule;
    GList *matching_rules, *node;

    g_return_if_fail (cb != NULL);
    g_return_if_fail (mergeData != NULL);
    currentRule = mergeData->currentRule;
    g_return_if_fail (mergeResult > 0);
    g_return_if_fail (mergeResult != MERGE_INVALID);
    g_return_if_fail (mergeData->abort == FALSE);
    qiter.fcn = cb;
    qiter.data = mergeData;
    matching_rules = NULL;
    for (node = mergeData->mergeList; node != NULL; node = node->next)
    {
        currentRule = node->data;
        if (currentRule->mergeResult == mergeResult)
            matching_rules = g_list_prepend (matching_rules, 
                currentRule);
    }
    qiter.remainder = g_list_length (matching_rules);
    g_list_foreach (matching_rules, qof_book_merge_rule_cb, &qiter);
    g_list_free (matching_rules);
}

called by dialogue callback to set the result of user intervention

Set any rule result to MERGE_INVALID to abort the import when qof_book_merge_commit is called, without changing the target book.

The calling process should make it absolutely clear that a merge operation cannot be undone and that a backup copy should always be available before a merge is initialised.

Recommended method: Only offer three options to the user per rule:

  1. Allow import data to be merged into target data
    • change MERGE_REPORT to MERGE_UPDATE
  2. Allow import data without an exact match to be added as new
    • change MERGE_REPORT to MERGE_NEW IF mergeAbsolute = FALSE
  3. Ignore import data and leave target data unchanged
    • change MERGE_REPORT to MERGE_ABSOLUTE or MERGE_DUPLICATE

Handle the required result changes in code: Check the value of qof_book_mergeRule::mergeAbsolute and use these principles:

To ignore entities tagged as:

  • MERGE_REPORT, you must check the value of mergeAbsolute.
    • if mergeAbsolute is TRUE, change MERGE_REPORT to MERGE_ABSOLUTE
    • if mergeAbsolute is FALSE, change MERGE_REPORT to MERGE_DUPLICATE
  • MERGE_NEW, set MERGE_DUPLICATE.
  • MERGE_UPDATE, you must check the value of mergeAbsolute.
    • if mergeAbsolute is TRUE, change MERGE_UPDATE to MERGE_ABSOLUTE
    • if mergeAbsolute is FALSE, change MERGE_UPDATE to MERGE_DUPLICATE

To merge entities that are not pre-set to MERGE_NEW, set MERGE_UPDATE.
Attempting to merge an entity when the pre-set value was MERGE_NEW will force a change back to MERGE_NEW because no suitable target exists for the merge.

To add entities, check mergeAbsolute is FALSE and set MERGE_NEW.
An entity only be added if mergeAbsolute is FALSE. Attempting to add an entity when mergeAbsolute is TRUE will always force a MERGE_UPDATE.

It is not possible to update the same rule more than once.

  1. MERGE_NEW is reserved for new objects and is only pre-set if all parameters, including GUID, have already failed to match any relevant object. ::qof_book_mergeCommit will create new entities for all rules tagged as MERGE_NEW.
    • if mergeAbsolute is TRUE and the user wants to import the data, requests to set MERGE_NEW will be forced to MERGE_UPDATE because an entity with that GUID already exists in the target book.
    • if MERGE_NEW is pre-set, requests to change to MERGE_UPDATE will be ignored because a new entity is needed.
  2. MERGE_UPDATE is reserved for existing objects - ::qof_book_mergeCommit will require a matching entity to update and will force a change to back to MERGE_NEW if none is known to exist, using the principle above.
  3. MERGE_INVALID will cause an abort of the merge process.
  4. MERGE_UNDEF and MERGE_REPORT cannot be set - the entity result will be unchanged.
  5. MERGE_DUPLICATE and MERGE_ABSOLUTE are handled identically but are semantically different - QofBookMergeRule::mergeAbsolute is used to dictate which to set:
    • if mergeAbsolute is TRUE but MERGE_DUPLICATE is requested, force a change to MERGE_ABSOLUTE.
    • if mergeAbsolute is FALSE but MERGE_ABSOLUTE is requested, force a change to MERGE_DUPLICATE.

qof_book_merge_commit only commits entities tagged with MERGE_NEW and MERGE_UPDATE results.
Entities tagged with MERGE_ABSOLUTE and MERGE_DUPLICATE results are ignored.

The calling process must check the return value and call ::qof_book_merge_abort(mergeData) if non-zero.

Parameters:
mergeDatathe merge context, QofBookMergeData*
tagthe result to attempt to set, QofBookMergeResult
Returns:
-1 if supplied parameters are invalid or NULL, 0 on success.

Definition at line 966 of file qofbookmerge.c.

{
    QofBookMergeRule *resolved;

    g_return_val_if_fail ((mergeData != NULL), NULL);
    g_return_val_if_fail ((tag > 0), NULL);
    g_return_val_if_fail ((tag != MERGE_REPORT), NULL);
    resolved = mergeData->currentRule;
    g_return_val_if_fail ((resolved != NULL), NULL);
    if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_DUPLICATE))
        tag = MERGE_ABSOLUTE;
    if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_NEW))
        tag = MERGE_UPDATE;
    if ((resolved->mergeAbsolute == FALSE) && (tag == MERGE_ABSOLUTE))
        tag = MERGE_DUPLICATE;
    if ((resolved->mergeResult == MERGE_NEW) && (tag == MERGE_UPDATE))
        tag = MERGE_NEW;
    if (resolved->updated == FALSE)
        resolved->mergeResult = tag;
    resolved->updated = TRUE;
    if (tag >= MERGE_INVALID)
    {
        mergeData->abort = TRUE;
        mergeData->currentRule = resolved;
        return NULL;
    }
    mergeData->currentRule = resolved;
    return mergeData;
}