PhysPlot Project Guide

This guide is written for maintainers, Codex sessions, and contributors who need to understand the whole project before changing it.

Working Philosophy

PhysPlot should feel simple to a researcher and predictable to a developer.

  • The spreadsheet is the master visual control.

  • Column roles are explicit and visible above the table.

  • Every meaningful GUI action should map to a backend workflow step when it affects reproducibility.

  • Saved protocol sequences are real Python, not opaque project files.

  • Bulk automation must replay the same sequence that the user built in the GUI.

  • Plot mode, plotter module, and plot configuration belong in the sequence. Run Sequence should not invent a separate plotting choice.

  • The GUI must call backend APIs rather than reimplementing loaders, transformations, plotting, fitting, workflows, or bulk logic.

Project Structure

PhysPlot/
  pyproject.toml                Package metadata for python-physplot
  README.md                     Public project overview and installation
  AGENTS.md                     Quick instructions for Codex/agent sessions
  docs/
    CODEX_PROJECT_GUIDE.md      Maintainer guide, structure, examples
  physplot/
    api.py                      Public PhysPlot facade
    bulk.py                     Folder/batch runner
    workflow.py                 Import workflow steps from .py source
    execution.py                StepResult, per-step execution, rerun snapshots
    __main__.py                 CLI entry point
    core/
      dataset.py                DataFrame wrapper, roles, metadata
      transformations.py        Registered transforms + config/transformations plugin resolution
      formula.py                Formula evaluation
      column_resolver.py        Name/number column resolution
    loaders/
      base.py                   Built-in auto/csv/txt/excel/dataframe loaders
    plotting_modules/
      base.py                   BasePlotter contract
      registry.py               PlotterRegistry and default registration
      *_plotter.py              Modular plotter implementations
    steps/
      *.py                      Replayable workflow step classes
    inc/
      *.png, *.jpeg             Logo and icon assets
  physplot_gui/
    __main__.py                 python -m physplot_gui entry point
    app/
      runner.py                 GUI launcher
      main_window.py            Main orchestration window
      mode_manager.py           Simple/Advanced stacked panel switcher
      gui_state.py              Shared GUI state across modes
      plugin_discovery.py       config/data_importers and config/transformations discovery
    fit_styles.py               config/figureforge_fit_styles presets (user folder first)
    plot_styles.py              config/templates figure templates (user folder first)
    panels/
      simple_mode_panel.py      Three-panel Simple Mode
      advanced_mode_panel.py    Build Protocol / Run Sequence tabs
      recorder_mode_panel.py    Sequence table/code and run sequence layout
      bulk_panel.py             Folder runner controls
    widgets/
      central_table.py          Spreadsheet table, editing, copy/paste
      column_role_header.py     Role dropdown header
      mode_switcher.py          Simple/Advanced switcher
      status_bar.py             Bottom status bar
    plotting_modules/
      user_modules.py           config/plotter_modules and config/plot_types discovery
    user_paths.py               Documents/PhysPlot resolution and config search order
    workflow.py                 Sequence loading and config/protocol_modules discovery
  config/                       User-editable modules (see config/README.md)
    data_importers/             Data Importer loaders
    transformations/            Mathematical Transformation functions
    plotter_modules/            Plotter Modules
    plot_types/                 Plot Type presets (JSON)
    protocol_modules/           Reusable protocol fragments
    sequences/, pipelines/      Complete sequences and transformation pipelines
    templates/                  Figure templates (JSON)
    fit_functions/              Legacy curve-fit models
    figureforge_fit_styles/     LSQ fit-style presets (JSON)
    figureforge_plugins/        Figure Editor plugins
  installer/windows/PhysPlot.iss  Inno Setup script; seeds Documents/PhysPlot/config
  scripts/build_windows_installer.ps1  PyInstaller + Inno Setup build driver
  tests/                        Backend and GUI smoke tests

Main User Flows

Application Header and Menus

The modern GUI is launched by physplot-gui, python-physplot-gui, or python -m physplot_gui. physplot_gui/app/runner.py sets the application name/display name to PhysPlot before constructing MainWindow.

The top header is intentionally branded but compact:

  • physplot/inc/lsf.jpeg and physplot/inc/physlab.png sit on the far left.

  • physplot/inc/PhysPlotWide1.png remains centered.

  • The Simple/Advanced mode switcher remains on the right.

The native menu bar is built in MainWindow._build_menu_bar():

  • File: import/export data.

  • Protocol: import/export/apply/copy/clear sequence actions.

  • View: Simple and Advanced mode shortcuts.

  • Plot: generate plots and update the integrated preview.

  • Help: documentation, repository, issue tracker, and About dialog.

On macOS, Qt may place these menus in the system menu bar while the app process still appears as Python when launched directly with python -m physplot_gui.

Simple Mode

Simple Mode has exactly three panels below the table:

  1. Data Importer

  2. Apply Mathematical Transformation

  3. Plotter Module

It is intended for everyday use: load data, assign roles, transform columns, choose an optional template, generate a plot, edit it in the Figure Editor, and export.

Build Protocol

Build Protocol is the editable protocol sequence. It has two views:

  • Table view: readable operation list with delete buttons.

  • Code view: generated Python source that can be edited and applied back to the table.

Important behavior:

  • Import Sequence.py loads workflow steps from a Python file.

  • Export Sequence.py writes the current protocol as Python.

  • Apply This Sequence runs the current sequence as shown.

  • Apply Code to Table appears only in Code view and rebuilds the protocol table from the edited Python source.

  • Deleting a row removes those workflow steps and immediately replays the revised protocol so the spreadsheet reflects the edited sequence. A failure during that replay is shown in the Status column and status bar, not in a modal dialog.

  • The Status column shows OK, Failed, or Skipped per row from PhysPlot.last_results (physplot/execution.py::StepResult). Results are matched to rows by step identity, so replacing a step clears its status.

  • Right-click a row for “Rerun from this step”: PhysPlot.rerun_from restores the snapshot taken before that step and resumes there. Snapshots are tagged with a fingerprint of the earlier steps’ code; if those changed, the latest still-valid earlier snapshot is used.

  • GUI replays use run_workflow_detailed(raise_on_error=False); headless run_workflow still raises the first error and keeps no snapshots.

  • Rows are editable (double-click or context menu), can be disabled with the “On” checkbox, moved up/down, and inserted after the selected row. All of it goes through MainWindow._timeline_step_pairs / _commit_timeline, which rebuild pp.workflow from the row order, then _resume_from_step. Rows containing a LoadDataStep are pinned first.

Writing an editable step

Steps describe their own editable fields; the GUI editor (physplot_gui/widgets/step_editor.py) has no step-specific code.

  • Subclass physplot.steps.base.WorkflowStep, accept enabled=True in __init__, and store it on self.enabled.

  • Implement _describe_fields() returning {name: field(value, type, ...)} built with physplot.steps.fields.field. Types: str, int, float, bool, column, choice, path, literal, value. Use optional=True for fields that may be blank and number_field= to link a column field to its column-number fallback.

  • Override _set_field(name, value) only when a field does not map to an attribute (see SetRoleStep).

  • Build to_code() with self._format_code([(name, value), ...]) so enabled=False round-trips; enabled steps produce the same code as before.

  • Provide template(columns) and add the class to STEP_TYPES in physplot/steps/__init__.py to offer it in “Insert Step…”.

  • apply is wrapped automatically so a disabled step is a no-op, and execute_steps reports it as skipped with skip_reason="disabled".

Run Sequence

Run Sequence is for bulk automation. It should stay intentionally small:

  • Input Folder

  • Sequence File, optional; blank means use the current Build Protocol sequence

  • Output Folder

  • Run Bulk Workflow

Plot choice is not a Run Sequence option. If a sequence contains:

PlotModuleStep(
    plotter_id="basic",
    plot_type="scatter",
    config={},
)

then bulk execution uses that exact plotter step. If the sequence does not contain a plot step, bulk execution exports data/metadata but does not invent a new plot.

Backend Contracts

Use the public facade:

from physplot import PhysPlot

pp = PhysPlot()
pp.load("data.csv", loader="auto")
pp.set_roles(x="Time", y="Voltage")
pp.transform("Voltage", "multiply", output="Voltage_mV", factor=1000)
pp.plot_with_module("basic", "scatter")
pp.export("outputs/run")

GUI actions should map to:

Import Data       -> pp.load(...)
Set column role   -> pp.set_roles(...)
Transform column  -> pp.transform(...)
Calculate formula -> pp.calculate(...)
Fit model         -> pp.fit(...)
Generate Plot     -> pp.plot_with_module(...) and Figure Editor for Basic Plotter
Export            -> pp.export(...)
Bulk Run          -> pp.run_bulk(...)

Current built-in loaders:

auto, csv, txt, excel, nanoindentation, dataframe

Current built-in plotter modules and plot types:

basic: scatter, line, scatter_line
histogram: histogram, density_histogram
scatter: scatter
line: line
errorbar: x_y_errorbar, y_errorbar
overlay: overlay_by_group, overlay_by_dataset
subplot_grid: subplots_by_group, subplots_by_dataset
nanoindentation: load_depth, hardness_depth, modulus_depth, stiffness_depth, contact_depth
oliver_pharr: load_depth_with_unloading_fit, unloading_fit, contact_stiffness_fit, area_function, hardness_summary, modulus_summary

Workflow Step Pattern

Each workflow step should be:

  • A small class under physplot/steps/

  • Serializable through to_code()

  • Replayable through apply(physplot, allow_column_number_fallback=False)

  • Safe for headless runs

Example:

TransformColumnStep(
    input_column="Voltage",
    input_column_number=2,
    function_name="multiply",
    output="Voltage_mV",
    output_column_number=5,
    params={"factor": 1000},
)

The column number fields are fallback anchors for headless bulk runs where different files use slightly different column names.

Annotated Protocol Drafts

Draft 1: Generic CSV to Basic Scatter Plot

from __future__ import annotations

"""A minimal PhysPlot protocol.

Run inside a notebook:
    pp = run()

Run from CLI:
    physplot run-workflow sequence.py --input data.csv --output outputs/run
"""

from physplot import PhysPlot
from physplot.steps import LoadDataStep, PlotModuleStep, SetRoleStep

WORKFLOW_STEPS = [
    # The GUI records this when the user imports data. In bulk mode this step is
    # skipped because each input-folder file becomes the active dataset.
    LoadDataStep(
        path="test_data/sample_linear.csv",
        loader="auto",
        dataset_name="sample_linear",
    ),

    # Roles are what plot modules use to find X, Y, errors, groups, and labels.
    SetRoleStep(
        roles={"x": "Time", "y": "Voltage", "yerr": "Error", "group": "Group"},
    ),

    # Plot mode and protocol are defined here, not in the bulk runner UI.
    PlotModuleStep(
        plotter_id="basic",
        plot_type="scatter",
        config={},
    ),
]


def build_workflow():
    return list(WORKFLOW_STEPS)


def run(input_path=None, output_dir="outputs/sequence_run", loader="auto"):
    pp = PhysPlot()
    steps = build_workflow()
    if input_path:
        pp.load(input_path, loader=loader)
        steps = [step for step in steps if not isinstance(step, LoadDataStep)]
    pp.run_workflow(steps, allow_column_number_fallback=True)
    pp.export(output_dir)
    return pp

Draft 2: Transform Then Plot

from physplot.steps import LoadDataStep, PlotModuleStep, SetRoleStep, TransformColumnStep

WORKFLOW_STEPS = [
    LoadDataStep(path="data.csv", loader="auto", dataset_name="experiment"),
    SetRoleStep(roles={"x": "Time", "y": "Voltage"}),

    # Creates a derived column that appears in the table and can become a role.
    TransformColumnStep(
        input_column="Voltage",
        input_column_number=2,
        function_name="normalize_max",
        output="Voltage_norm",
        output_column_number=5,
        params={},
    ),

    # Reassign Y to the derived column before plotting.
    SetRoleStep(roles={"x": "Time", "y": "Voltage_norm"}),
    PlotModuleStep(plotter_id="basic", plot_type="line", config={}),
]

Draft 3: Nanoindentation Visualization

from physplot.steps import LoadDataStep, PlotModuleStep, SetRoleStep

WORKFLOW_STEPS = [
    LoadDataStep(path="nano.csv", loader="csv", dataset_name="nano_test"),

    # These role names are optional for nanoindentation plotters, which also use
    # aliases like "Depth (nm)", "Load (mN)", "h", and "P".
    SetRoleStep(roles={"x": "Depth (nm)", "y": "Load (mN)"}),

    PlotModuleStep(
        plotter_id="nanoindentation",
        plot_type="load_depth",
        config={},
    ),
]

Draft 4: Personal File Loader With Loader-Owned Plotter

Personal file loaders live in config/data_importers/. A loader can define column names, default roles, and allowed plotter metadata. The GUI discovers it automatically.

# config/data_importers/my_instrument_loader.py
import pandas as pd

DISPLAY_NAME = "My Instrument Loader"
COLUMN_NAMES = ["Time", "Signal", "Error", "Sample"]
DEFAULT_COLUMN_ROLES = ["X", "Y", "Y Error", "Group"]


def load_data(path):
    """Return a DataFrame or 2D table-like object."""
    return pd.read_csv(path)


def publication_plot(dataset, plot_type="publication_ready", config=None):
    """Optional loader-owned plotter callable.

    The returned Matplotlib figure is stored as pp.last_figure and exported by
    pp.export(...). Keep custom styling here when a lab or instrument needs a
    fixed publication format.
    """
    import matplotlib.pyplot as plt

    fig, ax = plt.subplots()
    ax.plot(dataset.dataframe["Time"], dataset.dataframe["Signal"])
    ax.set_xlabel("Time")
    ax.set_ylabel("Signal")
    return fig


PLOTTERS = [
    {
        "plotter_id": "my_instrument_publication",
        "name": "My Instrument Publication Plot",
        "plot_types": ["publication_ready"],
        "callable": publication_plot,
    }
]

Testing Checklist

Before marking work production-ready:

python -m pytest
QT_QPA_PLATFORM=offscreen python -m pytest tests/test_protocol_sequence_editor.py
QT_QPA_PLATFORM=offscreen .venv/bin/python -m physplot_gui
rm -rf build dist python_physplot.egg-info
python -m build
python -m twine check dist/*

Expected result:

  • All backend tests pass.

  • GUI smoke tests pass or skip only when PyQt6 is unavailable.

  • Direct GUI launch succeeds on Python 3.12-3.14 (numpy 2.5 and scipy 1.18 need 3.12+); Python 3.12 is the current safe local choice on macOS.

  • twine check passes for both wheel and source distribution.

Release Checklist

  1. Confirm pyproject.toml version and package metadata.

  2. Run tests and package checks.

  3. Confirm dist/python_physplot-*.whl and .tar.gz were freshly built.

  4. Push the release branch.

  5. Tag only after the maintainer approves the exact release commit.