🎨 Data Designer Tutorial: Seeding Synthetic Data Generation with an External Dataset¶

📚 What you'll learn¶

In this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.

If this is your first time using Data Designer, we recommend starting with the first notebook in this tutorial series.

📦 Import Data Designer¶

• data_designer.config provides access to the configuration API.

• DataDesigner is the main interface for data generation.

In [ ]:

import data_designer.config as dd
from data_designer.interface import DataDesigner

import data_designer.config as dd from data_designer.interface import DataDesigner

⚙️ Initialize the Data Designer interface¶

• DataDesigner is the main object responsible for managing the data generation process.

• When initialized without arguments, the default model providers are used.

In [ ]:

data_designer = DataDesigner()

data_designer = DataDesigner()

🎛️ Define model configurations¶

• Each ModelConfig defines a model that can be used during the generation process.

• The "model alias" is used to reference the model in the Data Designer config (as we will see below).

• The "model provider" is the external service that hosts the model (see the model config docs for more details).

• By default, we use build.nvidia.com as the model provider.

In [ ]:

# This name is set in the model provider configuration.
MODEL_PROVIDER = "nvidia"

# The model ID is from build.nvidia.com.
MODEL_ID = "nvidia/nemotron-3-nano-30b-a3b"

# We choose this alias to be descriptive for our use case.
MODEL_ALIAS = "nemotron-nano-v3"

model_configs = [
    dd.ModelConfig(
        alias=MODEL_ALIAS,
        model=MODEL_ID,
        provider=MODEL_PROVIDER,
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=1.0,
            top_p=1.0,
            max_tokens=2048,
            extra_body={"chat_template_kwargs": {"enable_thinking": False}},
        ),
    )
]

# This name is set in the model provider configuration. MODEL_PROVIDER = "nvidia" # The model ID is from build.nvidia.com. MODEL_ID = "nvidia/nemotron-3-nano-30b-a3b" # We choose this alias to be descriptive for our use case. MODEL_ALIAS = "nemotron-nano-v3" model_configs = [ dd.ModelConfig( alias=MODEL_ALIAS, model=MODEL_ID, provider=MODEL_PROVIDER, inference_parameters=dd.ChatCompletionInferenceParams( temperature=1.0, top_p=1.0, max_tokens=2048, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ), ) ]

🏗️ Initialize the Data Designer Config Builder¶

• The Data Designer config defines the dataset schema and generation process.

• The config builder provides an intuitive interface for building this configuration.

• The list of model configs is provided to the builder at initialization.

In [ ]:

config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)

config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)

🏥 Prepare a seed dataset¶

• For this notebook, we'll create a synthetic dataset of patient notes.

• We will seed the generation process with a symptom-to-diagnosis dataset.

• We already have the dataset downloaded in the data directory of this repository.

🌱 Why use a seed dataset?

• Seed datasets let you steer the generation process by providing context that is specific to your use case.

• Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.

• During generation, prompt templates can reference any of the seed dataset fields.

In [ ]:

# Download sample dataset from Github
import urllib.request

url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv"
local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv")

# Seed datasets are passed as reference objects to the config builder.
seed_source = dd.LocalFileSeedSource(path=local_filename)

config_builder.with_seed_dataset(seed_source)

# Download sample dataset from Github import urllib.request url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv" local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv") # Seed datasets are passed as reference objects to the config builder. seed_source = dd.LocalFileSeedSource(path=local_filename) config_builder.with_seed_dataset(seed_source)

🎨 Designing our synthetic patient notes dataset¶

• The prompt template can reference fields from our seed dataset:
  • {{ diagnosis }} - the medical diagnosis from the seed data
  • {{ patient_summary }} - the symptom description from the seed data

In [ ]:

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="patient_sampler",
        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
        params=dd.PersonFromFakerSamplerParams(),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="doctor_sampler",
        sampler_type=dd.SamplerType.PERSON_FROM_FAKER,
        params=dd.PersonFromFakerSamplerParams(),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="patient_id",
        sampler_type=dd.SamplerType.UUID,
        params=dd.UUIDSamplerParams(
            prefix="PT-",
            short_form=True,
            uppercase=True,
        ),
    )
)

config_builder.add_column(dd.ExpressionColumnConfig(name="first_name", expr="{{ patient_sampler.first_name }}"))

config_builder.add_column(dd.ExpressionColumnConfig(name="last_name", expr="{{ patient_sampler.last_name }}"))

config_builder.add_column(dd.ExpressionColumnConfig(name="dob", expr="{{ patient_sampler.birth_date }}"))

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="symptom_onset_date",
        sampler_type=dd.SamplerType.DATETIME,
        params=dd.DatetimeSamplerParams(start="2024-01-01", end="2024-12-31"),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="date_of_visit",
        sampler_type=dd.SamplerType.TIMEDELTA,
        params=dd.TimeDeltaSamplerParams(dt_min=1, dt_max=30, reference_column_name="symptom_onset_date"),
    )
)

config_builder.add_column(dd.ExpressionColumnConfig(name="physician", expr="Dr. {{ doctor_sampler.last_name }}"))

config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="physician_notes",
        prompt="""\
You are a primary-care physician who just had an appointment with {{ first_name }} {{ last_name }},
who has been struggling with symptoms from {{ diagnosis }} since {{ symptom_onset_date }}.
The date of today's visit is {{ date_of_visit }}.

{{ patient_summary }}

Write careful notes about your visit with {{ first_name }},
as Dr. {{ doctor_sampler.first_name }} {{ doctor_sampler.last_name }}.

Format the notes as a busy doctor might.
Respond with only the notes, no other text.
""",
        model_alias=MODEL_ALIAS,
    )
)

data_designer.validate(config_builder)

config_builder.add_column( dd.SamplerColumnConfig( name="patient_sampler", sampler_type=dd.SamplerType.PERSON_FROM_FAKER, params=dd.PersonFromFakerSamplerParams(), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="doctor_sampler", sampler_type=dd.SamplerType.PERSON_FROM_FAKER, params=dd.PersonFromFakerSamplerParams(), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="patient_id", sampler_type=dd.SamplerType.UUID, params=dd.UUIDSamplerParams( prefix="PT-", short_form=True, uppercase=True, ), ) ) config_builder.add_column(dd.ExpressionColumnConfig(name="first_name", expr="{{ patient_sampler.first_name }}")) config_builder.add_column(dd.ExpressionColumnConfig(name="last_name", expr="{{ patient_sampler.last_name }}")) config_builder.add_column(dd.ExpressionColumnConfig(name="dob", expr="{{ patient_sampler.birth_date }}")) config_builder.add_column( dd.SamplerColumnConfig( name="symptom_onset_date", sampler_type=dd.SamplerType.DATETIME, params=dd.DatetimeSamplerParams(start="2024-01-01", end="2024-12-31"), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="date_of_visit", sampler_type=dd.SamplerType.TIMEDELTA, params=dd.TimeDeltaSamplerParams(dt_min=1, dt_max=30, reference_column_name="symptom_onset_date"), ) ) config_builder.add_column(dd.ExpressionColumnConfig(name="physician", expr="Dr. {{ doctor_sampler.last_name }}")) config_builder.add_column( dd.LLMTextColumnConfig( name="physician_notes", prompt="""\ You are a primary-care physician who just had an appointment with {{ first_name }} {{ last_name }}, who has been struggling with symptoms from {{ diagnosis }} since {{ symptom_onset_date }}. The date of today's visit is {{ date_of_visit }}. {{ patient_summary }} Write careful notes about your visit with {{ first_name }}, as Dr. {{ doctor_sampler.first_name }} {{ doctor_sampler.last_name }}. Format the notes as a busy doctor might. Respond with only the notes, no other text. """, model_alias=MODEL_ALIAS, ) ) data_designer.validate(config_builder)

🔁 Iteration is key – preview the dataset!¶

1. Use the preview method to generate a sample of records quickly.

2. Inspect the results for quality and format issues.

3. Adjust column configurations, prompts, or parameters as needed.

4. Re-run the preview until satisfied.

In [ ]:

preview = data_designer.preview(config_builder, num_records=2)

preview = data_designer.preview(config_builder, num_records=2)

In [ ]:

# Run this cell multiple times to cycle through the 2 preview records.
preview.display_sample_record()

# Run this cell multiple times to cycle through the 2 preview records. preview.display_sample_record()

In [ ]:

# The preview dataset is available as a pandas DataFrame.
preview.dataset

# The preview dataset is available as a pandas DataFrame. preview.dataset

📊 Analyze the generated data¶

• Data Designer automatically generates a basic statistical analysis of the generated data.

• This analysis is available via the analysis property of generation result objects.

In [ ]:

# Print the analysis as a table.
preview.analysis.to_report()

# Print the analysis as a table. preview.analysis.to_report()

🆙 Scale up!¶

• Happy with your preview data?

• Use the create method to submit larger Data Designer generation jobs.

In [ ]:

results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-3")

results = data_designer.create(config_builder, num_records=10, dataset_name="tutorial-3")

In [ ]:

# Load the generated dataset as a pandas DataFrame.
dataset = results.load_dataset()

dataset.head()

# Load the generated dataset as a pandas DataFrame. dataset = results.load_dataset() dataset.head()

In [ ]:

# Load the analysis results into memory.
analysis = results.load_analysis()

analysis.to_report()

# Load the analysis results into memory. analysis = results.load_analysis() analysis.to_report()

⏭️ Next Steps¶

Check out the following notebook to learn more about:

• Providing images as context

• Generating images