From d5e8b789c4e66c3e08536e2b3d4c8edd2a02cecb Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:27:14 -0600 Subject: [PATCH 1/7] add catalogue-recording pattern documentation --- doc/neuropixels_readers.rst | 176 ++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 doc/neuropixels_readers.rst diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst new file mode 100644 index 00000000..f7c62a37 --- /dev/null +++ b/doc/neuropixels_readers.rst @@ -0,0 +1,176 @@ +Neuropixels format readers: catalogue construction and recording-specific wiring +================================================================================= + +.. currentmodule:: probeinterface + + +The catalogue: :py:func:`build_neuropixels_probe` +------------------------------------------------- + +The foundation of every Neuropixels reader in probeinterface is +:py:func:`build_neuropixels_probe`. Given a probe part number (a specific +stock-keeping unit identifier such as ``"NP1000"``, ``"NP2000"``, ``"NP2014"``), +it returns a :py:class:`Probe` carrying the full silicon geometry for that +part number: every catalogue contact (960 for Neuropixels 1.0, 1280 per shank +for Neuropixels 2.0), the planar contour of the shanks, the contact shapes and +sizes, the analog-to-digital converter (ADC) multiplexer (MUX) routing table, +and the probe-level annotations (``manufacturer``, ``model_name``, +``part_number``, ``description``). + +The numbers behind that geometry come from the +`ProbeTable `_ repository maintained +by `Bill Karsh `_ (author of SpikeGLX). +ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels +probe specifications: contact positions, electrode dimensions, shank geometry, +MUX routing, ADC configuration, all keyed by part number. Probeinterface +mirrors a postprocessed snapshot of that data into the package via +``resources/postprocess_neuropixels_probe_features.py``, which is re-run after +each ProbeTable sync. Without ProbeTable, every reader would have to carry +its own hand-written copy of the manufacturer specs, which is exactly the +situation the catalogue pattern is designed to avoid. + + +The format readers +------------------ + +Four entry points read Neuropixels recordings (or recording configurations) +and produce a probe ready to use with SpikeInterface: + +.. list-table:: + :header-rows: 1 + :widths: 30 30 40 + + * - Reader + - Input + - Where the part number comes from + * - :py:func:`read_spikeglx` + - SpikeGLX ``.ap.meta`` (plus the ``.ap.bin`` it describes) + - ``imDatPrb_pn`` field in the meta file + * - :py:func:`read_openephys_neuropixels` + - Open Ephys ``settings.xml`` (plus the binary stream it describes) + - ``probe_part_number`` attribute in the XML + * - :py:func:`read_imro` + - SpikeGLX IMRO (Imec ReadOut) table file (``.imro``) + - First field of the IMRO header: a part number directly (new SpikeGLX + format) or a legacy numeric type code translated to a part number via + the catalogue mapping (old format). See SpikeGLX issue + `#432 `_ + for the format transition. + * - :py:func:`read_spikegadgets_neuropixels` + - SpikeGadgets ``.rec`` XML header + - Not present in the file; the reader picks a geometry-equivalent stand-in + based on ``(SpikeConfiguration.device, deviceSubType)``: ``NP1000`` for + Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` + for Neuropixels 2.0 4-shank + +The first three readers identify the actual probe stock-keeping unit (SKU) +from the recording metadata. SpikeGadgets is the exception: its ``.rec`` XML +does not carry a part number field, so the reader cannot identify the SKU. +It picks one representative per geometry-equivalent family (all Neuropixels +1.0 staggered variants share contact positions; all Neuropixels 2.0 +single-shank variants share contact positions; all Neuropixels 2.0 4-shank +variants share contact positions) and clears the ``model_name``, +``description``, and ``part_number`` annotations on the returned probe so +downstream code does not read the stand-in as an attribution. + + +From catalogue probe to probe in a recording setup +-------------------------------------------------- + +The catalogue probe is pure geometry, divorced from any session. A real +recording uses only a subset of those contacts: the Neuropixels headstage +acquires 384 channels at a time, and the recording configuration selects +which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels +1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on +Neuropixels 2.0 4-shank). The selection mechanism differs by recording +format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for +Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); the +"Matching catalogue contacts to recorded data" section below covers each +case. On top of the selection, the recording adds session-specific state: +per-contact analog band (AP) and local field potential (LFP) gains, ADC +sample order, reference configuration, and the channel-to-file mapping that +says where each contact's data lives in the saved binary. Probeinterface +calls the result a probe in a recording setup, to distinguish it from the +catalogue. + +Each reader produces the recording-setup probe in the same three steps: + +1. Build the catalogue probe by calling + :py:func:`build_neuropixels_probe(part_number) ` + with the part number obtained from the recording metadata. +2. Slice the catalogue probe to the active electrodes for this session via + :py:meth:`probe.get_slice(active_indices) `. The slice + drops the unrecorded contacts but preserves the probe-level annotations + and the per-contact catalogue annotations (ADC group, sample order) on the + contacts that survive. +3. Attach the recording-specific state: per-contact AP/LFP gains, any + reference annotations, and finally + :py:meth:`probe.set_device_channel_indices(...) ` + to record where each surviving contact's data lives in the saved file. + + +Matching catalogue contacts to recorded data +-------------------------------------------- + +The catalogue-build step is the same across readers; the matching step is +where the formats differ. ``active_indices`` and ``device_channel_indices`` +come from a different field in each metadata source: + +* **SpikeGLX:** ``active_indices`` is the electrode list parsed from the IMRO + table embedded in the ``.ap.meta`` file. ``device_channel_indices`` is + identity (``np.arange(n)``) because SpikeGLX writes one column per active + electrode in IMRO selection order. +* **Open Ephys:** ``active_indices`` is the electrode list parsed from the + ``CHANNELS`` block in ``settings.xml``. ``device_channel_indices`` follows + the order the binary stream uses, which the same XML file describes. +* **IMRO (standalone):** ``active_indices`` is parsed directly from the IMRO + entries. There is no recording to wire to, so :py:func:`read_imro` returns + the sliced probe without setting ``device_channel_indices``; callers that + have a corresponding ``.ap.bin`` use :py:func:`read_spikeglx` instead. +* **SpikeGadgets:** ``active_indices`` is the list of ``SpikeNTrode`` + electrodes from the ``.rec`` XML, remapped from Trodes' ``channelsOn`` bit + order to the catalogue's contact order (an identity remap for Neuropixels + 1.0; a row-major-to-shank-major remap for Neuropixels 2.0 4-shank; a + per-row column swap for Neuropixels 2.0 single-shank). + ``device_channel_indices`` is the ``hwChan`` attribute on each + ``SpikeNTrode``, which happens to coincide with the column index in the + SpikeGadgets datalogger's binary stream because the firmware writes samples + in ``hwChan`` ascending order. + + +What the pattern solves +----------------------- + +Constructing geometry from scratch inside each format reader (the situation +before the catalogue pattern) had three problems: + +* **Geometry drift across readers.** Each reader carried its own copy of the + manufacturer specs. A Neuropixels 2.0 4-shank probe loaded through SpikeGLX + and through SpikeGadgets could return contact positions that disagreed in + the third decimal because the two readers had been updated against + different snapshots of the IMEC spec. Centralising the geometry in + :py:func:`build_neuropixels_probe` and sourcing it from ProbeTable means + every reader returns the same positions for the same part number. +* **Conflated geometry and wiring bugs.** When a saved recording looked wrong + on the probe, it was difficult to say whether the geometry was off + (catalogue issue) or the channel-to-contact mapping was off (wiring issue). + With the two phases separated, a geometry bug is a bug in + :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the reader's + matching step. The two can be diagnosed and fixed independently. +* **Hidden active-electrode selection.** Readers that built a 384-contact + probe directly hid the fact that 576 catalogue contacts were silently + dropped. The explicit ``probe.get_slice(active_indices)`` step makes the + selection visible and inspectable: callers can ask "which catalogue + contacts did this session record?" and get a direct answer. + +The pattern also pays out on the upgrade path. When IMEC ships a new probe +variant, the integration work is "add the part number to ProbeTable, re-run +the postprocess script". + + +Discussion +---------- + +This pattern was proposed and is tracked in issue +`#405 `_; if you have +any discussion point to add please re-open the issue so the mantainers can disucss. From f879ed0370966d205f985f291831abeddf7b1fa5 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:27:29 -0600 Subject: [PATCH 2/7] document --- doc/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/index.rst b/doc/index.rst index aed06dfb..9ca01ef2 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -34,5 +34,6 @@ Here is a schema for the naming used in the package: examples/index.rst format_spec library + neuropixels_readers api release_notes From a4a0d01cecb26cf3285214456eb14ebb865d6f9f Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 22 May 2026 16:30:01 +0000 Subject: [PATCH 3/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/neuropixels_readers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index f7c62a37..1886d46a 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -165,7 +165,7 @@ before the catalogue pattern) had three problems: The pattern also pays out on the upgrade path. When IMEC ships a new probe variant, the integration work is "add the part number to ProbeTable, re-run -the postprocess script". +the postprocess script". Discussion From 5db34fdbf80bf2d128e94076c6e46c7afcb32af2 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:32:45 -0600 Subject: [PATCH 4/7] spelling --- doc/neuropixels_readers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index f7c62a37..fa5c0983 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -173,4 +173,4 @@ Discussion This pattern was proposed and is tracked in issue `#405 `_; if you have -any discussion point to add please re-open the issue so the mantainers can disucss. +any discussion point to add please re-open the issue so the maintainers can discuss. From 852fc67a2aaf402012ee49f4d47e76265f5266cb Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:59:00 -0600 Subject: [PATCH 5/7] title --- doc/neuropixels_readers.rst | 64 +++++++++++-------------------------- 1 file changed, 18 insertions(+), 46 deletions(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index 131755cf..29d9ad10 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -1,5 +1,5 @@ -Neuropixels format readers: catalogue construction and recording-specific wiring -================================================================================= +The Neuropixels catalogue pattern +================================= .. currentmodule:: probeinterface @@ -63,30 +63,31 @@ and produce a probe ready to use with SpikeInterface: Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` for Neuropixels 2.0 4-shank -The first three readers identify the actual probe stock-keeping unit (SKU) -from the recording metadata. SpikeGadgets is the exception: its ``.rec`` XML -does not carry a part number field, so the reader cannot identify the SKU. -It picks one representative per geometry-equivalent family (all Neuropixels -1.0 staggered variants share contact positions; all Neuropixels 2.0 -single-shank variants share contact positions; all Neuropixels 2.0 4-shank -variants share contact positions) and clears the ``model_name``, -``description``, and ``part_number`` annotations on the returned probe so -downstream code does not read the stand-in as an attribution. +The first three readers read the part number directly from the recording +metadata. SpikeGadgets is the exception: its ``.rec`` XML does not carry a +part number field, so the reader cannot know which specific variant produced +the recording. It picks one representative per geometry-equivalent family +(all Neuropixels 1.0 staggered variants share contact positions; all +Neuropixels 2.0 single-shank variants share contact positions; all +Neuropixels 2.0 4-shank variants share contact positions) and clears the +``model_name``, ``description``, and ``part_number`` annotations on the +returned probe so downstream code does not read the stand-in as an +attribution. From catalogue probe to probe in a recording setup -------------------------------------------------- -The catalogue probe is pure geometry, divorced from any session. A real +The catalogue probe is pure geometry, divorced from any recording session. A real recording uses only a subset of those contacts: the Neuropixels headstage acquires 384 channels at a time, and the recording configuration selects which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels 1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on Neuropixels 2.0 4-shank). The selection mechanism differs by recording format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for -Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); the -"Matching catalogue contacts to recorded data" section below covers each -case. On top of the selection, the recording adds session-specific state: +Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); each +reader's docstring covers the specifics for that format. On top of the +selection, the recording adds session-specific state: per-contact analog band (AP) and local field potential (LFP) gains, ADC sample order, reference configuration, and the channel-to-file mapping that says where each contact's data lives in the saved binary. Probeinterface @@ -98,7 +99,7 @@ Each reader produces the recording-setup probe in the same three steps: 1. Build the catalogue probe by calling :py:func:`build_neuropixels_probe(part_number) ` with the part number obtained from the recording metadata. -2. Slice the catalogue probe to the active electrodes for this session via +2. Slice the catalogue probe to the active electrodes for this recording session via :py:meth:`probe.get_slice(active_indices) `. The slice drops the unrecorded contacts but preserves the probe-level annotations and the per-contact catalogue annotations (ADC group, sample order) on the @@ -109,35 +110,6 @@ Each reader produces the recording-setup probe in the same three steps: to record where each surviving contact's data lives in the saved file. -Matching catalogue contacts to recorded data --------------------------------------------- - -The catalogue-build step is the same across readers; the matching step is -where the formats differ. ``active_indices`` and ``device_channel_indices`` -come from a different field in each metadata source: - -* **SpikeGLX:** ``active_indices`` is the electrode list parsed from the IMRO - table embedded in the ``.ap.meta`` file. ``device_channel_indices`` is - identity (``np.arange(n)``) because SpikeGLX writes one column per active - electrode in IMRO selection order. -* **Open Ephys:** ``active_indices`` is the electrode list parsed from the - ``CHANNELS`` block in ``settings.xml``. ``device_channel_indices`` follows - the order the binary stream uses, which the same XML file describes. -* **IMRO (standalone):** ``active_indices`` is parsed directly from the IMRO - entries. There is no recording to wire to, so :py:func:`read_imro` returns - the sliced probe without setting ``device_channel_indices``; callers that - have a corresponding ``.ap.bin`` use :py:func:`read_spikeglx` instead. -* **SpikeGadgets:** ``active_indices`` is the list of ``SpikeNTrode`` - electrodes from the ``.rec`` XML, remapped from Trodes' ``channelsOn`` bit - order to the catalogue's contact order (an identity remap for Neuropixels - 1.0; a row-major-to-shank-major remap for Neuropixels 2.0 4-shank; a - per-row column swap for Neuropixels 2.0 single-shank). - ``device_channel_indices`` is the ``hwChan`` attribute on each - ``SpikeNTrode``, which happens to coincide with the column index in the - SpikeGadgets datalogger's binary stream because the firmware writes samples - in ``hwChan`` ascending order. - - What the pattern solves ----------------------- @@ -161,7 +133,7 @@ before the catalogue pattern) had three problems: probe directly hid the fact that 576 catalogue contacts were silently dropped. The explicit ``probe.get_slice(active_indices)`` step makes the selection visible and inspectable: callers can ask "which catalogue - contacts did this session record?" and get a direct answer. + contacts did this recording session record?" and get a direct answer. The pattern also pays out on the upgrade path. When IMEC ships a new probe variant, the integration work is "add the part number to ProbeTable, re-run From ce65555ae735b7c0f7bda915a706a4a5630ca947 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 11:20:14 -0600 Subject: [PATCH 6/7] bette structure --- .gitignore | 2 + doc/neuropixels_readers.rst | 207 ++++++++++++++++++------------------ 2 files changed, 107 insertions(+), 102 deletions(-) diff --git a/.gitignore b/.gitignore index 52a45ce4..695ffc81 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,7 @@ tests/*.h5 examples/*.prb examples/*.h5 +examples/*.json ressources/*/* build/* @@ -19,6 +20,7 @@ build/* dist/* doc/_* doc/examples/* +doc/sg_execution_times.rst dev_* .coverage diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index 29d9ad10..a5c59a7f 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -4,37 +4,84 @@ The Neuropixels catalogue pattern .. currentmodule:: probeinterface -The catalogue: :py:func:`build_neuropixels_probe` -------------------------------------------------- - -The foundation of every Neuropixels reader in probeinterface is -:py:func:`build_neuropixels_probe`. Given a probe part number (a specific -stock-keeping unit identifier such as ``"NP1000"``, ``"NP2000"``, ``"NP2014"``), -it returns a :py:class:`Probe` carrying the full silicon geometry for that -part number: every catalogue contact (960 for Neuropixels 1.0, 1280 per shank -for Neuropixels 2.0), the planar contour of the shanks, the contact shapes and -sizes, the analog-to-digital converter (ADC) multiplexer (MUX) routing table, -and the probe-level annotations (``manufacturer``, ``model_name``, -``part_number``, ``description``). - -The numbers behind that geometry come from the +Two kinds of Neuropixels probe +------------------------------ + +Probeinterface distinguishes two kinds of Neuropixels probe. + +**Catalogue probe.** The probe as it appears in the IMEC catalogue, with +every contact on the silicon present (960 on Neuropixels 1.0, 1280 per +shank on Neuropixels 2.0). Built via :py:func:`build_neuropixels_probe(part_number) +`. It carries: + +* contact positions +* shank contour and dimensions +* contact shapes and sizes +* the analog-to-digital converter (ADC) multiplexer (MUX) routing on the + silicon +* identity metadata (manufacturer, model name, part number, description) + +A catalogue probe is pure geometry, the same for every recording made with +that variant. Use it to plot the probe layout, compute distances between +contacts, or run any analysis that does not depend on a specific recording. + +**Recording-setup probe.** The catalogue probe specialised for one +recording session: only the contacts actually recorded are present +(typically 384 of the 960 or more catalogue contacts), and per-contact +recording state is attached: + +* per-contact analog band (AP) and local field potential (LFP) gains +* reference configuration +* per-contact sampling order +* probe wiring (the mapping from each contact to the recording channel + that captured its data) + +Use a recording-setup probe to hand the recording to SpikeInterface so +spike sorters see both the geometry and the correct channel mapping, to +convert raw samples to microvolts via the per-contact gains, or to plot +the recorded contacts alongside the recorded traces. + + +How readers connect the two +--------------------------- + +A format reader turns a Neuropixels recording into a recording-setup probe +in three steps: + +1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in + the recording's metadata, then call + :py:func:`build_neuropixels_probe(part_number) `. +2. **Identify the active electrodes.** Read the recording's channel + configuration to find which catalogue contacts were actually recorded, + and slice the catalogue probe down to that subset via + :py:meth:`probe.get_slice(active_indices) `. +3. **Attach the recording-setup metadata.** Add the per-contact gains, + reference settings, and sampling order, and set the probe wiring. + +The sections below cover where the part number comes from per reader +(step 1) and where the catalogue data itself comes from. + + +The catalogue +------------- + +A probe part number (SKU) is an IMEC identifier such as ``"NP1000"``, +``"NP2000"``, or ``"NP2014"``. The part number determines the silicon +geometry: 960 contacts on Neuropixels 1.0, 1280 per shank on Neuropixels +2.0, plus all the per-variant pitch and shank dimensions. + +The data behind the catalogue comes from the `ProbeTable `_ repository maintained by `Bill Karsh `_ (author of SpikeGLX). ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels -probe specifications: contact positions, electrode dimensions, shank geometry, -MUX routing, ADC configuration, all keyed by part number. Probeinterface -mirrors a postprocessed snapshot of that data into the package via -``resources/postprocess_neuropixels_probe_features.py``, which is re-run after -each ProbeTable sync. Without ProbeTable, every reader would have to carry -its own hand-written copy of the manufacturer specs, which is exactly the -situation the catalogue pattern is designed to avoid. +probe specifications, all keyed by part number. -The format readers ------------------- +The readers +----------- -Four entry points read Neuropixels recordings (or recording configurations) -and produce a probe ready to use with SpikeInterface: +Four readers produce a probe from a Neuropixels recording. They differ in +where they look up the part number: .. list-table:: :header-rows: 1 @@ -58,91 +105,47 @@ and produce a probe ready to use with SpikeInterface: for the format transition. * - :py:func:`read_spikegadgets_neuropixels` - SpikeGadgets ``.rec`` XML header - - Not present in the file; the reader picks a geometry-equivalent stand-in - based on ``(SpikeConfiguration.device, deviceSubType)``: ``NP1000`` for - Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` - for Neuropixels 2.0 4-shank - -The first three readers read the part number directly from the recording -metadata. SpikeGadgets is the exception: its ``.rec`` XML does not carry a -part number field, so the reader cannot know which specific variant produced -the recording. It picks one representative per geometry-equivalent family -(all Neuropixels 1.0 staggered variants share contact positions; all -Neuropixels 2.0 single-shank variants share contact positions; all -Neuropixels 2.0 4-shank variants share contact positions) and clears the -``model_name``, ``description``, and ``part_number`` annotations on the -returned probe so downstream code does not read the stand-in as an -attribution. - - -From catalogue probe to probe in a recording setup --------------------------------------------------- - -The catalogue probe is pure geometry, divorced from any recording session. A real -recording uses only a subset of those contacts: the Neuropixels headstage -acquires 384 channels at a time, and the recording configuration selects -which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels -1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on -Neuropixels 2.0 4-shank). The selection mechanism differs by recording -format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for -Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); each -reader's docstring covers the specifics for that format. On top of the -selection, the recording adds session-specific state: -per-contact analog band (AP) and local field potential (LFP) gains, ADC -sample order, reference configuration, and the channel-to-file mapping that -says where each contact's data lives in the saved binary. Probeinterface -calls the result a probe in a recording setup, to distinguish it from the -catalogue. - -Each reader produces the recording-setup probe in the same three steps: - -1. Build the catalogue probe by calling - :py:func:`build_neuropixels_probe(part_number) ` - with the part number obtained from the recording metadata. -2. Slice the catalogue probe to the active electrodes for this recording session via - :py:meth:`probe.get_slice(active_indices) `. The slice - drops the unrecorded contacts but preserves the probe-level annotations - and the per-contact catalogue annotations (ADC group, sample order) on the - contacts that survive. -3. Attach the recording-specific state: per-contact AP/LFP gains, any - reference annotations, and finally - :py:meth:`probe.set_device_channel_indices(...) ` - to record where each surviving contact's data lives in the saved file. + - Not present in the file; the reader picks a geometry-equivalent + stand-in based on ``(SpikeConfiguration.device, deviceSubType)``: + ``NP1000`` for Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 + single-shank, ``NP2014`` for Neuropixels 2.0 4-shank + +The first three readers read the part number directly. SpikeGadgets is the +exception (its ``.rec`` XML does not carry a part number) and falls back to +a geometry-equivalent stand-in; the variants within each Neuropixels family +share identical 2D contact geometry, so any representative produces correct +positions. What the pattern solves ----------------------- -Constructing geometry from scratch inside each format reader (the situation -before the catalogue pattern) had three problems: - -* **Geometry drift across readers.** Each reader carried its own copy of the - manufacturer specs. A Neuropixels 2.0 4-shank probe loaded through SpikeGLX - and through SpikeGadgets could return contact positions that disagreed in - the third decimal because the two readers had been updated against - different snapshots of the IMEC spec. Centralising the geometry in - :py:func:`build_neuropixels_probe` and sourcing it from ProbeTable means - every reader returns the same positions for the same part number. -* **Conflated geometry and wiring bugs.** When a saved recording looked wrong - on the probe, it was difficult to say whether the geometry was off - (catalogue issue) or the channel-to-contact mapping was off (wiring issue). - With the two phases separated, a geometry bug is a bug in - :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the reader's - matching step. The two can be diagnosed and fixed independently. -* **Hidden active-electrode selection.** Readers that built a 384-contact - probe directly hid the fact that 576 catalogue contacts were silently - dropped. The explicit ``probe.get_slice(active_indices)`` step makes the - selection visible and inspectable: callers can ask "which catalogue - contacts did this recording session record?" and get a direct answer. - -The pattern also pays out on the upgrade path. When IMEC ships a new probe -variant, the integration work is "add the part number to ProbeTable, re-run -the postprocess script". +Building geometry from scratch inside each reader (the situation before this +pattern) caused three problems: + +* **Drift across readers.** Each reader carried its own copy of the + manufacturer specs, and the copies could disagree. Centralising the + geometry in :py:func:`build_neuropixels_probe` and sourcing it from + ProbeTable means every reader returns the same positions for the same + part number. +* **Confused bugs.** When a saved recording looked wrong on the probe, it + was hard to tell whether the geometry was off or the channel-to-contact + mapping was off. With the two phases separated, a geometry bug is a bug + in :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the + reader's slicing or wiring step. The two can be diagnosed independently. +* **Hidden electrode selection.** A reader that built a 384-contact probe + directly hid the fact that 576 catalogue contacts were silently dropped. + The explicit slice step makes the selection visible: callers can ask + which catalogue contacts the recording captured and get a direct answer. + +The pattern also helps on the upgrade path. When IMEC ships a new probe +variant, the integration work is to add the part number to ProbeTable; the +readers do not change. Discussion ---------- This pattern was proposed and is tracked in issue -`#405 `_; if you have -any discussion point to add please re-open the issue so the maintainers can discuss. +`#405 `_. +Reopen the issue if you want to discuss changes. From 1c77f7238f55223fbe1fcff90b7fa8b920bf95c9 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 11:22:29 -0600 Subject: [PATCH 7/7] perfect muack done --- doc/neuropixels_readers.rst | 45 ++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index a5c59a7f..2d560b3a 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -42,28 +42,8 @@ convert raw samples to microvolts via the per-contact gains, or to plot the recorded contacts alongside the recorded traces. -How readers connect the two ---------------------------- - -A format reader turns a Neuropixels recording into a recording-setup probe -in three steps: - -1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in - the recording's metadata, then call - :py:func:`build_neuropixels_probe(part_number) `. -2. **Identify the active electrodes.** Read the recording's channel - configuration to find which catalogue contacts were actually recorded, - and slice the catalogue probe down to that subset via - :py:meth:`probe.get_slice(active_indices) `. -3. **Attach the recording-setup metadata.** Add the per-contact gains, - reference settings, and sampling order, and set the probe wiring. - -The sections below cover where the part number comes from per reader -(step 1) and where the catalogue data itself comes from. - - -The catalogue -------------- +The catalogue source +-------------------- A probe part number (SKU) is an IMEC identifier such as ``"NP1000"``, ``"NP2000"``, or ``"NP2014"``. The part number determines the silicon @@ -77,11 +57,24 @@ ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels probe specifications, all keyed by part number. -The readers ------------ +Format readers +-------------- + +A format reader turns a Neuropixels recording into a recording-setup probe +in three steps: + +1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in + the recording's metadata, then call + :py:func:`build_neuropixels_probe(part_number) `. +2. **Identify the active electrodes.** Read the recording's channel + configuration to find which catalogue contacts were actually recorded, + and slice the catalogue probe down to that subset via + :py:meth:`probe.get_slice(active_indices) `. +3. **Attach the recording-setup metadata.** Add the per-contact gains, + reference settings, and sampling order, and set the probe wiring. -Four readers produce a probe from a Neuropixels recording. They differ in -where they look up the part number: +The four readers in probeinterface differ in step 1: where they look up the +part number in the recording's metadata. .. list-table:: :header-rows: 1