Recent developments for the pixelated Anode

.gitignore

@@ -13,4 +13,15 @@ notes.*
 *.hdf
 store
 .snakemake
+*PLAN*
 
+# wogrid scratch / per-machine artifacts (test-full-3d-pixel.sh)
+test/OUTPUT/
+test/bkup_store/
+test/plots/
+test/toy/
+test/review/
+test/log.log
+test/pochoir_stencil_vs_stencil_poisson_torch.log
+toy/*.npy
+env/

Instructions_dataMCcomparison.txt

@@ -0,0 +1,11 @@
+To compare FR between data and MC :
+
+1) Find a straight track with reasonable angle to anode plane in data and simulate exactly (same angle, same distance to anode, same length) through WCWC. Now just look at some metrics like comparing raw WF on several vires, average WF, deconvolved charge between palnes.
+
+Extend it to many angles and large statistics
+
+2) Find blips of enarge from Ar do a salection and simulate similar things. Compare average WF
+
+
+
+If things are different (but transparency condition is 100%) most likely drift velocity is off. That means poor choice of cathode voltage or argon temperature. One can just play with temperature to get exact velocity needed , no reason to resimulate whole drift volume (https://lar.bnl.gov/properties/)

Instructions_runnig.txt

@@ -0,0 +1,129 @@
+there are multiple steps of running. Overall one needs to: 
+1)produce 3D drift field in a small volume 
+2)2D weighting field in large volume 
+3)use 2D wfield as boundary to get 3D weighting field in a smaller volume
+4)using 3D drift field produce velocity map, set electron starting points and drift them
+5)produce large 3D weighting field by combining 3D and 2D versions
+6)produce current in large 3D weighting field by coping paths in smaller volume around (symmetries are important)
+7)use full path map in large volume to get currents
+8)repeat for 3 (or any) planes and pack it in jsonnet for WC (or any format you want)
+
+
+How package does those things : 
+
+example in test-full-3d.sh
+1) one needs to choose domain by choosing step size and number of points in x,y,z directions  
+
+do_domain () {
+    local name=$1 ; shift
+    local shape=$1; shift
+    local spacing=$1; shift
+
+    want domain/$name \
+         pochoir domain --domain domain/$name \
+         --shape=$shape --spacing $spacing
+}
+do_domain drift3d  25,17,2000  '0.1*mm'
+
+# fixme: these weight* identifiers need to split up for N planes.
+do_domain weight2d 1092,2000   '0.1*mm'
+do_domain weight3d 350,32,2000 '0.1*mm'
+
+
+2)Generate a geometry. This step is highly dependent on what geometry you want and usually requires coding it 
+
+do_gen () {
+    local name=$1 ; shift
+    local geom=$1; shift
+    local gen="pcb_$geom"
+    local cfg="example_gen_pcb_${geom}_config.json"
+
+    want initial/$name \
+         pochoir gen --generator $gen --domain domain/$name \
+         --initial initial/$name --boundary boundary/$name \
+         $cfg
+
+}
+do_gen drift3d quarter
+do_gen weight2d 2D
+do_gen weight3d 3D
+
+
+From the example above one needs a generator and config files. generator file in python script and the live in main director. cfg files are in test directory and one can write their own matching generator they have
+
+3)Run FDM. This is most time consuming step, make sure it runs on GPU (should be automatic if engine is set to torch, but if you PyTorch sees only CPU it will use only CPU, double check). Each command has number of epochs (aka 10 in example) and generally you need just 1; number of steps per epoch (20000 or 9000) whose you need in millions range to get reasonable result, to estimate it one have to look at the result and see if for example for collection plane current is actually summed up to one so whole volume got calculate, also one can look at the volume plot and see if there are any gaps -> highly dependent on volume and step; last entry are boundary conditions for x,y,z they can be fixed or periodic make sure to choose once you want for your case as it can be different based on geometry choices, for weighting field though they are generally fixed
+
+weighting field specifically has parameter bc-interp --xcoord '17.5*mm' that tells the code there to stitch 2D to 3D and this one would depend on geoemtry and you spesific coded generator
+
+do_fdm () {
+    local name=$1 ; shift
+    local nepochs=$1 ; shift
+    local epoch=$1 ; shift
+    local prec=$1 ; shift
+    local edges=$1 ; shift
+
+    want potential/$name \
+         pochoir fdm \
+         --nepochs $nepochs --epoch $epoch --precision $prec \
+         --edges $edges \
+         --engine torch \
+         --initial initial/$name --boundary boundary/$name \
+         --potential potential/$name \
+         --increment increment/$name
+}
+do_fdm drift3d  10      20000      0.00002     fix,fix,fix
+do_fdm weight2d 10      20000      0.0002   per,fix
+
+
+# special step to form iva/bva from 2D solution
+want initial/weight3dfull \
+     pochoir bc-interp --xcoord '17.5*mm' \
+     --initial initial/weight3dfull --boundary boundary/weight3dfull \
+     --initial3d initial/weight3d --boundary3d boundary/weight3d \
+     --potential2d potential/weight2d
+
+do_fdm weight3dfull 10      9000      0.00002     fix,fix,fix
+
+
+4) Calculate velocity based on drift field 
+
+
+want velocity/drift3d \
+     pochoir velo --temperature '89*K' \
+     --potential potential/drift3d \
+     --velocity velocity/drift3d
+
+5) Set drift starting points and calculate drift in small volume 
+
+want velocity/drift3d \
+     pochoir velo --temperature '89*K' \
+     --potential potential/drift3d \
+     --velocity velocity/drift3d
+
+#rm -r /Users/sergey/Desktop/ICARUS/LArStand/pochoir/test/store/starts
+
+want starts/drift3d \
+    pochoir starts --starts starts/drift3d \
+    '0.05*mm,0.05*mm,198*mm' 
+
+want paths/drift3d \
+     pochoir drift --starts starts/drift3d \
+     --velocity velocity/drift3d \
+     --paths paths/drift3d '0*us,4250*us,0.1*us'
+    #--paths paths/drift3d '0*us,4250*us,0.1*us'
+
+
+Example in 3view_chain_coll.sh
+6) Get full extended weighting field 
+
+pochoir extendwf -p store/potential/weight2d -P store/potential/weight3dfull -n 10 -o store/potential/weight3dextend
+
+7) Induce current from paths 
+
+pochoir induce -w store/potential/weight3dextend -p store/paths/drift3d -a 11 -n 21 -O store/current/induced_current_avg_coll_ext
+
+
+8) get current from 3 planes int WC format 
+
+pochoir convertfr -u store/all_currents/induced_current_avg_ind1_ext -v store/all_currents/induced_current_avg_ind2_ext -w store/all_currents/induced_current_avg_coll_ext -O store/all_currents/FR_HigherV_89K.json.bz2 example_converter_config_30deg.json
+

Makefile

@@ -0,0 +1,51 @@
+# PLAN_for_production.md §3 — single entry point for the test-full-3d-pixel
+# workflow. Runs pytest and the pipeline, capturing artifacts under a fresh
+# test/review/<date>_<slug>/ folder with a NOTES.md stub.
+#
+# Iteration defaults use the reduced z-extents from the plan (300 instead of
+# 500). Release validation overrides them on the command line:
+#
+#   make test-pixel SLUG=release-validation \
+#        POCHOIR_DRIFT_SHAPE=44,44,500 POCHOIR_WEIGHT_SHAPE=220,220,500
+#
+# Assumes the venv at pochoir/env/ is already activated (per PLAN §"Python
+# environment").
+
+POCHOIR_DRIFT_SHAPE  ?= 44,44,300
+POCHOIR_WEIGHT_SHAPE ?= 220,220,300
+SLUG                 ?= iteration
+DATE                 := $(shell date +%Y-%m-%d)
+REVIEW_DIR           := test/review/$(DATE)_$(SLUG)
+
+.PHONY: test-pixel pytest pipeline
+
+test-pixel: pytest pipeline
+	@echo "Artifacts: $(REVIEW_DIR)"
+
+pytest:
+	@mkdir -p $(REVIEW_DIR)
+	@echo "=== pytest -> $(REVIEW_DIR)/pytest.log ==="
+	cd test && python -m pytest -v 2>&1 | tee ../$(REVIEW_DIR)/pytest.log
+
+pipeline:
+	@mkdir -p $(REVIEW_DIR)/outputs
+	@echo "=== test-full-3d-pixel.sh -> $(REVIEW_DIR)/pipeline.log ==="
+	@echo "    POCHOIR_DRIFT_SHAPE=$(POCHOIR_DRIFT_SHAPE)"
+	@echo "    POCHOIR_WEIGHT_SHAPE=$(POCHOIR_WEIGHT_SHAPE)"
+	cd test && POCHOIR_DRIFT_SHAPE=$(POCHOIR_DRIFT_SHAPE) \
+	           POCHOIR_WEIGHT_SHAPE=$(POCHOIR_WEIGHT_SHAPE) \
+	           bash test-full-3d-pixel.sh 2>&1 | tee ../$(REVIEW_DIR)/pipeline.log
+	@if [ ! -f $(REVIEW_DIR)/NOTES.md ]; then \
+	  printf '%s\n' \
+	    "# $(SLUG) — $(DATE)" "" \
+	    "## What changed" "- " "" \
+	    "## Why" "- " "" \
+	    "## Tests added / updated" "- " "" \
+	    "## How to reproduce" \
+	    "- \`make test-pixel SLUG=$(SLUG) POCHOIR_DRIFT_SHAPE=$(POCHOIR_DRIFT_SHAPE) POCHOIR_WEIGHT_SHAPE=$(POCHOIR_WEIGHT_SHAPE)\`" "" \
+	    "## Result" \
+	    "- pytest: see pytest.log" \
+	    "- pipeline: see pipeline.log" "" \
+	    "## Follow-ups" "- " \
+	    > $(REVIEW_DIR)/NOTES.md ; \
+	fi

README_for_pixelatedReadout.md

@@ -0,0 +1,132 @@
+# Pixelated readout — full 3D pipeline
+
+This document is a focused companion to the main `README.org`. It describes
+what the script `test/test-full-3d-pixel.sh` does, what it needs, and how to
+run it. Use the main README for the package philosophy and the generic
+`pochoir` CLI; use this one when you want to reproduce or modify the
+pixelated-readout field-response simulation.
+
+## What the script does
+
+`test/test-full-3d-pixel.sh` runs the full chain that produces the field
+response of a pixelated LArTPC anode with a PCB shield grid. It chains the
+following `pochoir` subcommands (each step is cached by the `want` helper, so
+re-runs only redo missing artifacts):
+
+1. **Drift domain & boundary** — build a 3D domain
+   (`POCHOIR_DRIFT_SHAPE`, spacing `0.1 mm`), generate the boundary/initial
+   value arrays from `example_gen_pcb_drift_pixel_with_grid.json` using the
+   `pcb_drift_pixel_with_grid` generator (PCB shield plane with rounded
+   square holes + pixel pads with rounded corners).
+2. **Drift potential** — solve Laplace via `pochoir fdm` (torch engine,
+   periodic in x/y, fixed in z).
+3. **Weighting domain & boundary** — larger transverse domain
+   (`POCHOIR_WEIGHT_SHAPE`) using `example_gen_pixel_with_grid.json` and
+   the `pcb_pixel_with_grid` generator.
+4. **Weighting potential** — `pochoir fdm` for the weighting field of the
+   central pixel.
+5. **Velocity field** — `pochoir velo` at 87 K from the drift potential.
+6. **Starting points & drift paths** — a 10×10 grid of starts per pixel
+   (100 points), drifted with `pochoir drift` over `0–210 µs` at `0.05 µs`.
+7. **Induced currents** — `pochoir induce-pixel` over a 4-pixel window using
+   the weighting potential and the drift paths.
+
+Convergence is summarised by `parse_maxerr.py`, which produces
+`store/maxerr_*.png` and `store/summary_log_*.pdf`.
+
+## Requirements
+
+### System
+
+- Linux (the script has been used on Debian-class hosts).
+- Python 3.9+ with a working virtual environment.
+- A CUDA-capable GPU is **strongly** recommended — the `fdm` step uses the
+  `torch` engine and the weighting domain in particular is large
+  (a typical run uses `396×396×1500` ≈ 235 M voxels).
+- `bash`, `make`, and standard coreutils.
+
+### Python
+
+Install pochoir with the extras needed by this pipeline:
+
+```bash
+python3 -m venv env
+source env/bin/activate
+pip install -e .[torch,plots,hdf5]
+```
+
+(See `README.org` for the canonical install, including optional `numba`,
+`cupy`, and `vtk` extras.)
+
+### Configuration files
+
+Two JSON configs in `test/` drive the geometry generators. The fields most
+relevant to the pixelated readout are:
+
+- `example_gen_pcb_drift_pixel_with_grid.json` — drift-field geometry
+  (cathode, PCB shield plane, pixel plane).
+- `example_gen_pixel_with_grid.json` — weighting-field geometry.
+
+Common fields (units: mm except where noted):
+
+| field | meaning |
+| --- | --- |
+| `pixelSize` | pixel side length |
+| `pixelGap` | gap between adjacent pixels |
+| `chamfer_r` | rounded-corner radius (passed to `trimCorner`) |
+| `Npixels` | number of pixels along one axis |
+| `pixelPlaneLowEdgePosition` | z of the bottom of the pixel plane |
+| `pixelPlaneWidth` | thickness of the pixel plane |
+| `PcbWidth` | thickness of the PCB shield |
+| `GridHoleShape` | `"None"`, `"circular"`, or `"square"` |
+| `GridPotential` | shield-plane potential (V) |
+| `CathodePotential` | cathode potential (V) |
+| `LArPermittivity`, `FR4Permittivity` | dielectric constants |
+
+The two configs should agree on `pixelSize`, `pixelGap`, `chamfer_r`, and
+`Npixels`, otherwise the drift and weighting geometries describe different
+detectors.
+
+## Usage
+
+From the `test/` directory, after activating the venv:
+
+```bash
+cd test
+bash test-full-3d-pixel.sh [STORE_DIR]
+```
+
+`STORE_DIR` defaults to `store/`. The two domain shapes are defined at the
+top of the script and may be overridden via environment variables. A
+typical full-z-extent run uses:
+
+```bash
+POCHOIR_DRIFT_SHAPE=44,44,1500 \
+POCHOIR_WEIGHT_SHAPE=396,396,1500 \
+bash test-full-3d-pixel.sh store
+```
+
+Defaults in the script (`44,44,300` / `396,396,300`) are tuned for fast
+iteration; use the larger z-extent (`1500`) for production runs.
+
+## Outputs
+
+All artifacts land under `STORE_DIR` (default `test/store/`):
+
+- `domain/{drift3d,weight3d}` — domain definitions.
+- `initial/*`, `boundary/*` — generator outputs.
+- `potential/{drift3d,weight3d}` — Laplace solutions.
+- `velocity/drift3d` — drift velocity field.
+- `starts/drift3d`, `paths/drift3d_tight` — drift starts and paths.
+- `current/induced_current` — induced currents on the 4-pixel window.
+- `pochoir_driftfield.log`, `pochoir_weightingfield.log` — solver logs.
+- `maxerr_*.png`, `summary_log_*.pdf` — convergence diagnostics.
+
+## Tweaking the geometry
+
+The rounded-corner shape is implemented in `trimCorner` in
+`pochoir/gen_pcb_drift_pixel_with_grid.py` and
+`pochoir/gen_pcb_pixel_with_grid.py`. Both carve a quarter-disk of radius
+`chamfer_r` (in grid-index units after dividing by the domain spacing) at
+each inner corner. Increase `chamfer_r` in the JSON configs to round the
+corners more strongly; set it to `0` for sharp 90° corners.