Parameter support by Transurgeon · Pull Request #42 · SparseDifferentiation/SparseDiffEngine

Transurgeon · 2026-02-10T16:51:12Z

This PR adds support for parameters entering in affine functions.
The semantics of parameter creation are similar to that of variables.
We take the parameter offset data from the InverseData in cvxpy and create equivalent parameters in C.
Then we form a parameter vector theta which stacks all parameters, this theta can be updated on every iteration.

After calling update_params we just need to recompute the values of the jacobian, since the sparsity pattern will be fixed. This pattern is computed once using the exact same technique as before.
One thing that is nice with our abstraction is that all affine operators require no changes; they simply manipulate the arrays from their children, which are irrespective of if the expression has parameters or not.

The main change ends up adding new functionality in bivariate affine operators where parameters can appear.
These include scalar multiplication, vector (elementwise) multiplication and matrix multiplication.
I have checked carefully the first two implementations, but I must admit I don't fully get the matrix mult one (but this also probably stems from me not understanding the original implementation). @dance858 please verify that this looks good.

Overall I think this is a very clean and simple addition to support affine parameters in the diff-engine, I am happy to hear any feedback on the design and implementation.

Add parameter node type and parameter-aware variants of scalar mult, vector mult, and left matmul. Parameters store an offset into a global theta vector and can be updated via problem_update_params without rebuilding the expression tree. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

n_params (total scalar parameter count) can be computed from n_param_nodes and each node's size, making the field redundant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

These dimensions are always equal to param_node->d1 and param_node->d2, which are set during make_parameter. Read them from the node directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Precompute total parameter size in problem_register_params, mirroring how total_constraint_size is computed in new_problem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Exercises param_scalar_mult, param_vector_mult, and left_param_matmul with problem_register_params/problem_update_params to verify objective, gradient, constraint, and Jacobian values update correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The retained param_node was never released because free_type_data was NULL. Add free_param_type_data to match const_vector_mult and left_matmul. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

dance858 · 2026-02-12T14:47:38Z

This looks great! Two comments:

There's one conceptual thing that bothers me. left_matmul represents A @ f(x) where f(x) is a vector-valued or matrix-valued expression. When A is a constant, we treat it as a CSR matrix. When A is a parameter, we treat it as a dense matrix stored in column order. Let's discuss this point over phone.
Also, I think we must update AT inside eval_wsum_hess, since f(x) is not necessarily just x.

Sync parameter-support with main's left matmul 100x perf improvements (PR #44) and right matmul refactor (PR #46). Simplify param matmul to store only the small A matrix instead of block-diagonal — block_left_multiply_* functions handle the rest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge the separate Constant and Parameter leaf nodes into a unified parameter_expr with PARAM_FIXED sentinel (-1) for constants. This eliminates duplicate code paths and consolidates 7 bivariate constructors into 3 unified ones: - new_const_scalar_mult / new_param_scalar_mult -> new_scalar_mult - new_const_vector_mult / new_param_vector_mult -> new_vector_mult - new_left_matmul (CSR) / new_left_param_matmul -> new_left_matmul (param node) Key changes: - Add PARAM_FIXED define and extend new_parameter() to accept initial values - Delete constant.c (absorbed by parameter.c) - Remove direct value storage (double a, double *a) from scalar/vector mult structs; always read from param_source - left_matmul builds sparse CSR for fixed params (preserving sparsity) and dense CSR for updatable params - right_matmul internally creates a fixed parameter node from transposed A - problem_update_params skips PARAM_FIXED nodes - Update all test callers to use new_parameter with PARAM_FIXED Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ents Post-refactor cleanup after the Constant→Parameter merge: - Remove dead NULL assignments in free_type_data (quad_form, hstack, index, linear_op, left_matmul, scalar_mult, vector_mult) - Rename const_scalar_mult/const_vector_mult source and test files to drop const_ prefix - Rename test_variable_constant.h → test_variable_parameter.h and test_constant → test_fixed_parameter - Update stale comments in multiply.c and bivariate.h - Rename const_scalar_mult_expr/const_vector_mult_expr structs to scalar_mult_expr/vector_mult_expr - Update left_matmul param_source comment Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon

@dance858 some progress on the parameter feature.
I also renamed tests and functions to remove the const_ text, I hope that's okay.
For example, now we have scalar_mult instead of const_scalar_mult.

Transurgeon · 2026-02-15T23:09:06Z

src/affine/linear_op.c

        free(lin_node->b);
-        lin_node->b = NULL;
    }
-
-    lin_node->A_csr = NULL;
-    lin_node->A_csc = NULL;


claude noticed that these NULL assignments were not necessary. please confirm if that's accurate.

src/affine/index.c

Transurgeon · 2026-02-15T23:17:45Z

src/affine/parameter.c

+    init_expr(&pnode->base, d1, d2, n_vars, forward, jacobian_init, eval_jacobian,
+              is_affine, wsum_hess_init, eval_wsum_hess, NULL);


is this correct declaration for a parameter expression? I am not sure about the first argument being &pnode->base but it seems to form a base parameter expression?

src/bivariate/left_matmul.c

tests/jacobian_tests/test_left_matmul.h

…anching - Change signature to (expr *param_node, expr *child, const CSR_Matrix *A) - Constructor copies A with new_csr() instead of rebuilding from dense values - Remove src_m/src_n fields from left_matmul_expr (use A->m directly) - Allow param_node=NULL for fixed constants (no-op in refresh_param_values) - Update all tests to pass CSR directly; fixed-constant tests use NULL param Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

src/bivariate/left_matmul.c

- left_matmul: replace col-major loop in refresh_param_values with memcpy of nnz doubles (values now arrive in CSR data order) - right_matmul: pass AT->x directly to new_parameter(nnz, 1, ...), remove col-major round-trip allocation - test_param_prob: update theta arrays to CSR data order Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon · 2026-02-16T05:24:45Z

src/bivariate/right_matmul.c

+    /* Parameter stores CSR data order (same as AT->x) */
    expr *u_transpose = new_transpose(u);
-    expr *left_matmul = new_left_matmul(u_transpose, AT);
-    expr *node = new_transpose(left_matmul);
+    expr *param_node = new_parameter(AT->nnz, 1, PARAM_FIXED, u->n_vars, AT->x);
+    expr *left_matmul_node = new_left_matmul(param_node, u_transpose, AT);
+    expr *node = new_transpose(left_matmul_node);


I also changed right matmul to use parameters now.

Transurgeon

@dance858 I might have gone a bit crazy tonight, but good progress nonetheless.
Let me know what you think about supporting sparse parameters with a fixed pattern.
I feel like the PR is in a good state, but definitely want you to take your time with reviewing and asking me questions about the implementation.
Eventually we can make new constructors for dense Parameters (and constants).. and have some super fast BLAS operations done there to separate the two code paths.

P.S. you might need to try this code on this DNLP branch with this SparseDiffPy PR.

src/bivariate/left_matmul.c

Transurgeon and others added 12 commits February 9, 2026 01:29

Remove redundant n_params from problem struct

fea0708

n_params (total scalar parameter count) can be computed from n_param_nodes and each node's size, making the field redundant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove redundant A_m, A_n params from new_left_param_matmul

36ec239

These dimensions are always equal to param_node->d1 and param_node->d2, which are set during make_parameter. Read them from the node directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add total_parameter_size field to problem struct

4f93f7c

Precompute total parameter size in problem_register_params, mirroring how total_constraint_size is computed in new_problem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Inline get_scalar helper in const_scalar_mult.c

cf695a2

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Inline get_vector helper in const_vector_mult.c

bd4f3b0

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Simplify refresh_param_values: fill one block, memcpy the rest

27acb7d

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Skip AT recomputation in param refresh; param matmul is always affine

00f7732

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Run clang-format on parameter support files

dc56c90

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Clean up comments in bivariate.h and subexpr.h

01c6f82

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix memory leak in new_param_scalar_mult

b8cf436

The retained param_node was never released because free_type_data was NULL. Add free_param_type_data to match const_vector_mult and left_matmul. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon and others added 3 commits February 13, 2026 16:22

Run clang-format on merge-resolved files

bf8a55c

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon force-pushed the parameter-support branch from 528e1c1 to 9700344 Compare February 15, 2026 21:09

Transurgeon and others added 2 commits February 15, 2026 17:55

Run clang-format on cleanup changes

71dddf1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon commented Feb 15, 2026

View reviewed changes

Transurgeon force-pushed the parameter-support branch from e13ce04 to 36864d4 Compare February 16, 2026 00:45

Transurgeon commented Feb 16, 2026

View reviewed changes

src/bivariate/left_matmul.c Show resolved Hide resolved

Transurgeon commented Feb 16, 2026

View reviewed changes

src/bivariate/left_matmul.c Show resolved Hide resolved

dance858 added 3 commits February 17, 2026 08:37

small edits

b3e2304

fix AT workspace in left_matmul and add has_been_refreshed

0edf928

add parameter support for right matmul

b985028

dance858 added 3 commits February 17, 2026 12:01

clean up

4e25da7

add back setting ptrs to null after freeingmake

4b58b1c

added free and null macro

1c3ebee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parameter support#42

Parameter support#42
Transurgeon wants to merge 25 commits intomainfrom
parameter-support

Transurgeon commented Feb 10, 2026 •

edited

Loading

Uh oh!

dance858 commented Feb 12, 2026

Uh oh!

Transurgeon left a comment

Uh oh!

Transurgeon Feb 15, 2026

Uh oh!

Uh oh!

Transurgeon Feb 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Transurgeon Feb 16, 2026 •

edited

Loading

Uh oh!

Transurgeon left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		init_expr(&pnode->base, d1, d2, n_vars, forward, jacobian_init, eval_jacobian,
		is_affine, wsum_hess_init, eval_wsum_hess, NULL);

Conversation

Transurgeon commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dance858 commented Feb 12, 2026

Uh oh!

Transurgeon left a comment

Choose a reason for hiding this comment

Uh oh!

Transurgeon Feb 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Transurgeon Feb 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Transurgeon Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Transurgeon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Transurgeon commented Feb 10, 2026 •

edited

Loading

Transurgeon Feb 16, 2026 •

edited

Loading

Transurgeon left a comment •

edited

Loading