Skip to content

Copy proposer - #158

Merged
tim-band merged 8 commits into
mainfrom
copy_proposer
Nov 3, 2026
Merged

tim-band merged 8 commits into
mainfrom
copy_proposer

Conversation

@tim-band

Copy link
Copy Markdown
Collaborator

Fixes #130
Also adds a copy proposer that allows the copying of a column that has a "source" role. This can be used for all the OMOP person_id columns that should all copy from each other.

Tim Band added 7 commits September 23, 2026 14:43
Also:
- fixed role change text
- fixed remove all roles
- allow intervals and copying from same column in aliased table
You can't copy or anchor from nullable (or via nullable) to non-nullable.
You can't copy from an FK to an FK that points to a different table.
Bugs fixed.
]

def _can_copy_type(self, copy_from, copy_to) -> bool:
return type(copy_from) is type(copy_to)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_can_copy_type is defined but never called. Was it meant to stop copying between incompatible column types (e.g. text → int)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I added a test to check that copying from a different type is not proposed, and the test failed! So, now I wired in this function.

or a related table.
:param engine: The source database.
:param buckets: The buckets for the intervals in the source database.
"""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstring here looks copy-pasted from DateAfterProposer: says “Initialise a date after proposer”, documents a buckets param that doesn’t exist on this class, and :param anchor: should be :param anchor_related:

Suggested change
"""
"""
Initialise a copy proposer.
:param metadata: The metadata of the source database.
:param column: The column to generate.
:param anchor_related: The anchor column, which must be in the same
table as column or a related table.
:param engine: The source database.
"""

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Fixed.

@yhong123

yhong123 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Shall we update the documentation builtin_generators.rst as follows?

Around builtin_generators.rst:329 :

Two roles currently exist: ``start`` (consumed by the date interval proposer
above) and ``source`` (consumed by the copy proposer, described below). You
can still set either role by hand-editing ``config.yaml`` instead of using
``role``, if you prefer; both routes update the same field.

And add a new section right after the “Date interval” table and before “Date component extraction proposers”:

Copy proposers
^^^^^^^^^^^^^^^

Propose copying a column's already-generated value verbatim, rather than
generating it independently --- useful for columns that must always agree with
another column by construction (for example a denormalised ``person_id``
duplicated across related tables). This only appears once another column in
the same table, or a directly related table (via a foreign key), has been
given the ``source`` role.

Roles are set per column and stored in ``config.yaml`` under
``tables.<table>.columns.<column>.roles``. The easiest way to set one is the
``role`` command inside ``configure-generators``:

.. code-block:: shell

   (rental.rental_date) role set source
   (rental.rental_date) role list
   source
   (payment.payment_date) role on rental.rental_date list

A proposal is only offered when it is safe to copy: the source and
destination must agree on whether they are foreign keys (and, if so, point at
the same target table and column), and a non-nullable destination column will
never be offered a nullable source --- a ``NULL`` source value would leave the
destination with nothing to copy.

.. list-table::
   :widths: 20 30 50
   :header-rows: 1

   * - Applies to
     - Generator proposed
     - Notes
   * - Any column with a ``source``-roled column in the *same* table
     - ``generic.anchored_provider.copy``
     - Copies the other column's generated value for this row directly.
   * - Any column with a ``source``-roled column in a *related* table
       (via a foreign key)
     - ``generic.anchored_provider.copy_fk``
     - Looks up the related row already generated in the destination database
       and copies its value, the same way ``normal_date_fk`` looks up its
       anchor.

Unlike the other proposers in this section, ``copy``/``copy_fk`` is never
scored against the real data (it always reports a neutral fit) --- datafaker
cannot tell you whether the two columns you've linked are actually always
equal in the source database. Only use the ``source`` role where that
equality genuinely holds by design (e.g. a known denormalisation); copying
between two columns that merely happen to share a foreign key target but
represent different real-world entities (a patient vs. a clinician, say)
will produce misleadingly-consistent synthetic data.

@tim-band
tim-band merged commit 2afc5a0 into main Nov 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross-table anchored date needs onclause

3 participants