chore: remove dangerous From implementation on guarded strings#7065
chore: remove dangerous From implementation on guarded strings#7065benjamin-stacks wants to merge 15 commits intostacks-network:developfrom
From implementation on guarded strings#7065Conversation
When testing something, I got surprised by a `try_from` call panicking, which should never happen. It *did* happen because the `guarded_string!` macro implements `From<&str>`, which isn't only invalid, but also causes an automatic implementation of `TryFrom`, calls to which then look safe but actually aren't. This feels like a giant footgun: `ClarityName::try_from::<String>` is perfectly safe and works as intended, but `ClarityName::try_from::<&str>` will panic instead of returning an error. Luckily, all calls with truly external data used the former version, but using the latter would be a very easy mistake to make. Initially I wanted to completely remove this implementation, but it's used a loooot with hardcoded strings (both in tests and in release code). For such strings, it feels safe enough, so for now I just changed it to require `&'static str`. Happy to discuss this approach though. I still think that all else being equal, fully removing it would be safer. The current approach is a compromise between blast radius and safety.
My local toolchain insisted I do them, but CI insists that I don't 🤷 Since CI has the last work, I disabled my pre-commit hook for this.
|
changing to draft mode until I've fixed the build errors (which the local |
|
I think personally I would probably opt for a full removal, maybe via a |
That may not be worth it, I think the vast majority of issues are all one type (namely Another option is to remove the |
jcnelson
left a comment
There was a problem hiding this comment.
This looks fine to me, modulo the compile errors.
… `MustInto` Instead of the previous change, where we just restricted the source types for `.from()`, this change now outright removes that trait from guarded strings. Instead, it adds a new trait `MustInto` that allows convenient creation of e.g. clarity names from strings (something that is used a lot in both test and non-test code), while making it obvious that the caller has a responsibility to guarantee that the conversion will work.
I've done that now (see updated PR description), there wasn't really much of a choice 😅 |
Coverage Report for CI Build 24347808425Coverage increased (+0.003%) to 85.715%Details
Uncovered Changes
Coverage Regressions1797 previously-covered lines in 58 files lost coverage.
Coverage Stats
💛 - Coveralls |
There was a problem hiding this comment.
Pull request overview
This PR removes the unsafe From<&str> conversion generated by guarded_string! (which previously caused surprising panics via an auto-derived TryFrom) and replaces it with explicit, intention-revealing “must” conversions for hardcoded literals.
Changes:
- Remove
From<&str>for guarded strings; addTryFrom<&str>plusmust_from(&'static str)andMustInto::must_into()for explicit “this may panic” conversions. - Update call sites across the codebase from
.into()/::from(...)totry_from(...),must_from(...), or.must_into(). - Add a regression test to ensure
try_*conversions return errors instead of panicking, and add a changelog entry.
Reviewed changes
Copilot reviewed 93 out of 93 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| stackslib/src/util_lib/signed_structured_data.rs | Switch tuple keys to explicit .must_into() conversions. |
| stackslib/src/net/tests/convergence.rs | Replace .into() with explicit ContractName::try_from(...). |
| stackslib/src/net/stackerdb/config.rs | Replace .into() with .must_into() for Clarity name literals in type signatures. |
| stackslib/src/net/p2p.rs | Update test URL literal conversion to .must_into(). |
| stackslib/src/net/mod.rs | Remove rusqlite import and adjust imports; introduces use url;. |
| stackslib/src/net/db.rs | Update test literals to .must_into() for UrlString / contract identifiers. |
| stackslib/src/net/codec.rs | Avoid temporary as_str() by passing owned String into UrlString::try_from. |
| stackslib/src/net/chat.rs | Update test literals to .must_into() and adjust util imports. |
| stackslib/src/net/api/tests/mod.rs | Update contract-name literals to .must_into() and adjust util imports. |
| stackslib/src/net/api/tests/getmapentry.rs | Update map-name parsing/assertions to .must_into(). |
| stackslib/src/net/api/tests/getistraitimplemented.rs | Update trait-name literals to .must_into(). |
| stackslib/src/net/api/tests/getdatavar.rs | Update var-name literals to .must_into(). |
| stackslib/src/net/api/tests/getconstantval.rs | Update constant-name literals to .must_into(). |
| stackslib/src/net/api/tests/fastcallreadonly.rs | Update read-only function name literal to .must_into(). |
| stackslib/src/net/api/tests/callreadonly.rs | Update read-only function name literal to .must_into(). |
| stackslib/src/net/api/tests/blocksimulate.rs | Use must_from and pass owned ContractName/ClarityName into helper. |
| stackslib/src/net/api/tests/blockreplay.rs | Use must_from for contract/function names in test payload. |
| stackslib/src/cost_estimates/tests/fee_scalar.rs | Add license header; update contract/function literals to .must_into(). |
| stackslib/src/cost_estimates/tests/fee_medians.rs | Add license header; update contract/function literals to .must_into(). |
| stackslib/src/cost_estimates/tests/cost_estimators.rs | Add license header; update conversions to try_into() / .must_into(). |
| stackslib/src/core/tests/mod.rs | Update contract-name literal conversion to .must_into(). |
| stackslib/src/core/test_util.rs | Make contract/function names explicit (try_from / must_from) and tighten helper signatures. |
| stackslib/src/clarity_vm/tests/forking.rs | Update contract identifier name literal conversion to .must_into(). |
| stackslib/src/clarity_vm/tests/events.rs | Update asset-name literals to .must_into(). |
| stackslib/src/clarity_vm/tests/costs.rs | Update contract identifier name literals to .must_into(). |
| stackslib/src/clarity_vm/tests/contracts.rs | Update tuple key literals to .must_into(). |
| stackslib/src/clarity_vm/tests/analysis_costs.rs | Update contract identifier name literals to .must_into(). |
| stackslib/src/clarity_vm/clarity.rs | Update transaction contract name/function literals to .must_into(). |
| stackslib/src/chainstate/tests/runtime_analysis_tests.rs | Update contract identifier name literal to .must_into(). |
| stackslib/src/chainstate/stacks/transaction.rs | Update test contract-name literals to .must_into(). |
| stackslib/src/chainstate/stacks/tests/mod.rs | Replace ContractName::from with ContractName::try_from in test helpers. |
| stackslib/src/chainstate/stacks/tests/block_construction.rs | Update tuple key literals to .must_into(). |
| stackslib/src/chainstate/stacks/mod.rs | Update test contract-name literals to .must_into(). |
| stackslib/src/chainstate/stacks/db/transactions.rs | Replace guarded string from/into uses with must_from/try_from/.must_into(). |
| stackslib/src/chainstate/stacks/db/mod.rs | Update tuple key literals to .must_into() for genesis/namespace/name props. |
| stackslib/src/chainstate/stacks/boot/signers_tests.rs | Update boot contract/function name literals to .must_into() and use ContractName::try_from where dynamic. |
| stackslib/src/chainstate/stacks/boot/pox_4_tests.rs | Update tuple key literals to .must_into() for PoX-4 tests/helpers. |
| stackslib/src/chainstate/stacks/boot/pox_2_tests.rs | Update tuple key literals to .must_into() for PoX-2 tests/helpers. |
| stackslib/src/chainstate/stacks/boot/mod.rs | Update boot function name conversions to .must_into(). |
| stackslib/src/chainstate/stacks/boot/contract_tests.rs | Update contract identifier tuple keys to .must_into(). |
| stackslib/src/chainstate/stacks/address.rs | Update PoX address tuple keys to .must_into(). |
| stackslib/src/chainstate/nakamoto/tests/mod.rs | Update function name literals to .must_into() in signer voting tests. |
| stackslib/src/chainstate/nakamoto/signer_set.rs | Update tuple key literals and function-name comparisons to .must_into(). |
| stackslib/src/chainstate/nakamoto/coordinator/tests.rs | Update contract-name handling to avoid .into() and use ContractName explicitly. |
| stackslib/src/chainstate/coordinator/tests.rs | Update contract/function name literals to .must_into(). |
| stacks-signer/src/client/stacks_client.rs | Use ClarityName::must_from for hardcoded function names; update tuple keys to .must_into(). |
| stacks-signer/src/cli.rs | Update tuple keys to .must_into() for vote digest generation. |
| stacks-node/src/tests/stackerdb.rs | Update contract-name literals to .must_into(). |
| stacks-node/src/tests/neon_integrations.rs | Use must_from for hardcoded contract/function names; adjust helper usage. |
| stacks-node/src/tests/nakamoto_integrations.rs | Use must_from/.must_into() for hardcoded Clarity identifiers. |
| stacks-node/src/tests/mempool.rs | Use must_from for hardcoded contract names. |
| stacks-node/src/tests/integrations.rs | Update tuple keys and contract/function name literals to .must_into(). |
| stacks-node/src/tests/epoch_23.rs | Update contract-name literals to .must_into() in tests. |
| stacks-node/src/tests/epoch_21.rs | Update contract name comparisons to .must_into() and adjust imports. |
| stacks-node/src/tests/epoch_205.rs | Use ContractName::must_from in payload matching. |
| stacks-node/src/event_dispatcher/tests.rs | Use must_from for hardcoded contract/function names. |
| stacks-common/src/util/mod.rs | Introduce MustInto trait for explicit panic-on-failure conversions from literals. |
| stacks-common/src/util/macros.rs | Add must_from, remove From<&str>, add TryFrom<&str>, and implement MustInto for guarded strings. |
| pox-locking/src/pox_4.rs | Update tuple key literals to .must_into() in tests. |
| libstackerdb/src/tests/mod.rs | Update contract-name literal to .must_into(). |
| libsigner/src/v0/messages.rs | Update tuple key literals to .must_into() in mock proposal encoding. |
| contrib/stacks-inspect/src/main.rs | Replace UrlString::from with try_from(...).unwrap() and use must_from where intended. |
| clarity/src/vm/types/serialization.rs | Update tuple key literals to .must_into() in tests. |
| clarity/src/vm/types/mod.rs | Update type signature tuple keys to .must_into(). |
| clarity/src/vm/tests/simple_apply_eval.rs | Update names/atoms in tests to .must_into(). |
| clarity/src/vm/tests/proptest_utils.rs | Update contract-name literals to .must_into(). |
| clarity/src/vm/tests/principals.rs | Update tuple keys and contract-name construction to .must_into() / try_into(). |
| clarity/src/vm/tests/post_conditions.rs | Update contract-name literals to .must_into(). |
| clarity/src/vm/tests/datamaps.rs | Update tuple key literals to .must_into(). |
| clarity/src/vm/tests/assets.rs | Update contract/asset name literals to .must_into(). |
| clarity/src/vm/test_util/mod.rs | Update tuple key literals to .must_into(). |
| clarity/src/vm/mod.rs | Update transient contract identifier name to .must_into() in test-only helpers. |
| clarity/src/vm/functions/principals.rs | Update tuple key literals to .must_into() in principal construction helpers. |
| clarity/src/vm/functions/post_conditions.rs | Update wildcard asset name to .must_into(). |
| clarity/src/vm/functions/mod.rs | Update atoms in tests to .must_into(). |
| clarity/src/vm/functions/define.rs | Update atoms and trait name literal to .must_into() in tests. |
| clarity/src/vm/functions/database.rs | Update tuple key literals to .must_into() for burn block info response. |
| clarity/src/vm/docs/mod.rs | Update tuple key literals to .must_into() in docs tests. |
| clarity/src/vm/costs/mod.rs | Update cost tuple keys to .must_into(); ensure cost function name atom is validated. |
| clarity/src/vm/contexts.rs | Update asset/trait/function name literals to .must_into() in tests. |
| clarity/src/vm/callables.rs | Update names in tests to .must_into() / must_from. |
| clarity/src/vm/ast/sugar_expander/mod.rs | Require 'static atom literals in tests and use .must_into(). |
| clarity/src/vm/ast/parser/v1.rs | Require 'static atom literals in tests and use .must_into(). |
| clarity/src/vm/analysis/types.rs | Update analysis test identifiers to .must_into(). |
| clarity/src/vm/analysis/type_checker/v2_1/tests/mod.rs | Use must_from for trait names in test expectations. |
| clarity/src/vm/analysis/type_checker/v2_1/tests/conversions.rs | Use must_from for tuple key names in tests. |
| clarity/src/vm/analysis/type_checker/v2_1/tests/contracts.rs | Use must_from for tuple key names in tests. |
| clarity/src/vm/analysis/type_checker/v2_1/natives/mod.rs | Update tuple key literals to .must_into() in type signatures. |
| clarity-types/src/tests/types/signatures.rs | Update tuple key literals to .must_into() in type tests. |
| clarity-types/src/tests/types/serialization.rs | Update tuple key literals to .must_into() in serialization vectors/tests. |
| clarity-types/src/tests/types/mod.rs | Update tuple key literals to .must_into() in constructor tests. |
| clarity-types/src/tests/representations.rs | Add regression test ensuring try_* conversions do not panic on invalid strings. |
| changelog.d/7065-guarded-string-from.changed | Add changelog entry describing the guarded-string conversion refactor. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cylewitruk-stacks
left a comment
There was a problem hiding this comment.
Haven't gotten all the way through yet, but just dropping a few comments in the meantime :)
stacks-common/src/util/macros.rs
Outdated
| /// at a glance that the conversion will succeed. | ||
| /// | ||
| /// For values only known at runtime, use `try_from()` and deal with errors. | ||
| pub fn must_from(value: &'static str) -> Self { |
There was a problem hiding this comment.
from_static, from_str or from_literal might be some naming alternatives? (possibly with an _unchecked suffix, indicating that there are invariants for the caller to uphold; but may be a bit overstated since _unchecked often indicates UB if the invariants aren't upheld..)
There was a problem hiding this comment.
Yeah I did consider something with unchecked, but didn't like it because of that very reason. unchecked seems to have established semantics in the Rust world that don't quite apply here. On the other hand, the word "must" for this pattern has precedence (my inspiration came from Go's Regexp MustCompile.
On the other hand I wanted to keep the symmetry for from/into and try_from/try_into.
stacks-common/src/util/mod.rs
Outdated
|
|
||
| pub trait MustInto<T> { | ||
| /// Like `.into()`, but for cases where it's not a perfect conversion, and the | ||
| /// caller must guarantee that the conversion will succeed, because the method | ||
| /// will panic otherwise. This is made for converting `&str` into things | ||
| /// like `ClarityName`s, where the source value is hardcoded and thus it's visible | ||
| /// at a glance that the conversion will succeed. | ||
| /// | ||
| /// For values only known at runtime, use `try_into()` and deal with errors. | ||
| fn must_into(&'static self) -> T; | ||
| } |
There was a problem hiding this comment.
must_into() should probably just take self instead of &'static self, and let the impl block dictate if self is a static ref.
There was a problem hiding this comment.
Naming-wise, maybe IntoUnchecked and into_unchecked() or something similar might be nicer?
There was a problem hiding this comment.
must_into()should probably just takeselfinstead of&'static self, and let theimplblock dictate ifselfis a static ref.
Hmmm. I'm willing to be convinced otherwise, but doing it this way was a conscious decision. Here's my reasoning:
This functionality is kinda dangerous. It's fine for simple hardcoded strings (the way it's used in hundreds of places), but I really want to discourage future us from turning it into a more generic convenience functionality (like, arguably, was the case before this change). Changing the implementation type feels like a simple tweak somebody might make innocuously. Changing the trait itself on the other hand feels like a bigger deal, and that's pretty much by design.
Let me know what you think.
There was a problem hiding this comment.
Yeah, I get the "why", there's just something that's semantically bugging me; I think it's the naming and "Like .into() ..." comment, because we're implying Into-like semantics but deviate from them.
Tbh if it were me, I would probably drop both this trait and the macro's must_from() impl, and force all callsites to use Foo::try_from(bar), either handling the Result properly or using .unwrap() (tests)/.expect(..) (prod code for the sad functions which haven't been made fallible). Perfect, mechanical task for an LLM :)
I think that would remove all of the "smell" around this -- what do you think?
There was a problem hiding this comment.
Tbh if it were me, I would probably drop both this trait and the macro's
must_from()impl, and force all callsites to useFoo::try_from(bar), either handling theResultproperly or using.unwrap()(tests)/.expect(..)(prod code for the sad functions which haven't been made fallible).
I'm not a fan of that, quite frankly, because it loses the distinction between hardcoded (i.e. &'static) and other strings. In cases like this:
stacks-core/stackslib/src/chainstate/stacks/db/mod.rs
Lines 1610 to 1613 in fcca97b
(these into()s are before this PR; they're must_into()s in the current version of the PR) I think it's reasonable and useful that there's a succinct way to say "this is obviously ok". Adding expect()s all over the place for this just puts a strain on the reader of the code, because every time you see that, your brain should yell "danger!". These two are not the same:
Foo::try_from("reasonable-identifier").expect("...")
Foo::try_from(some_function_parameter).expect("...")
but you're training the reader to glance over that.
I think it's the naming and "Like
.into()..." comment, because we're implyingInto-like semantics but deviate from them.
What I mean by "Like .into()" is the Rust-ism of saying "Compiler, please convert this to the thing you need. You'll figure it out". But if that's not clear, I'm happy to phrase it differently, and as I said, I'm also not necessarily married to the name.
What's important to me is that there's an API that makes it non-annoying to do the obviously correct thing, but harder to accidentally cause a vulnerability because everything looks the same, but in the 1% case things are different.
I like the .into() thing (not the actual name, but the method-style call) because I think it reads quite well and removes the need for boilerplate because the type system figures out the details.
Perfect, mechanical task for an LLM :)
Nah, once we've come to a conclusion here, I'll make whatever change we've agreed on with deterministic refactoring tools 😅 That's faster, cheaper, safer, and not against the rules 😁
There was a problem hiding this comment.
Yeah, I get the distinction you're trying to preserve, but I still don't think we should introduce a generically-named trait which strongly implies standard Into semantics when the API is really about constructing a T from a &'static foo, for which TryFrom/TryInto are typically the right tools.
Yeah, it's used in a thousand places, but mostly in test code where a common practice would be to have a #[cfg(test)]-gated impl which can do the "panicky" and ergonomic .into() if you prefer to avoid .expect()s.
For the handful of literals in non-test code, we could use e.g. a ClarityName::from_literal(name: &'static str) panicking function, which yes is more verbose, but clear (and could carry its own # Panics doc-comment). Or, alternatively, implement e.g. namespace(), name(), etc. as associated functions on their respective types so you could use e.g. ClarityName::namespace() instead of "namespace".into() (since we can't use real constants due to String).
There was a problem hiding this comment.
I we're willing to live with [cfg(test)]-only stuff, I can get onboard with that! I'll make that change.
For the handful of literals in non-test code, we could use e.g. a
ClarityName::from_literal(name: &'static str)panicking function, which yes is more verbose, but clear (and could carry its own# Panicsdoc-comment). Or, alternatively, implement e.g.namespace(),name(), etc. as associated functions on their respective types so you could use e.g.ClarityName::namespace()instead of"namespace".into()(since we can't use real constants due toString).
I like that approach! It's a little more than a handful (I'm counting 29 at a glance nah it's a lot more), but this still sounds like a reasonable direction to think in. Thanks!
There was a problem hiding this comment.
if you prefer to avoid
.expect()s.
I just stumbled over this phrase when re-reading -- to be clear, it's the existing code that is trying to avoid them. This original .into() is used about 600 times in existing code, and it doesn't just strongly imply standard Into semantics, it literally implements that trait (while violating the semantics).
I'm just trying to fix the issue while still as much as possible respecting the choices made when creating that existing code.
There was a problem hiding this comment.
Ok, 87b9b08 makes must_into a test-only thing and uses Thing::from_literal for all non-test code.
The remaining question now is if we want to retain some sort of "extension method"-ish (I know you have a .NET history, so you know what I mean) shortcut for the test code, or also use Thing::from_literal for those instances. There are more than 500 such calls, and it would add quite some visual noise, but I can also see the benefits.
I'm good either way, let me know what you think.
There was a problem hiding this comment.
Just to close the loop here -- Cyle and I discussed this more and decided we'll just use ::from_literal everywhere, including tests.
Also make `must_into` a test-only thing (pending discussion on the PR as to whether we might want to remove it completely).
As per https://stackslabs.slack.com/archives/C0A7GGFQNKC/p1775828249032919?thread_ts=1775826350.099919&cid=C0A7GGFQNKC, this was probably missed by accident.
When testing something, I got surprised by a
try_fromcall panicking, which should never happen. It did happen because theguarded_string!macro implementsFrom<&str>, which isn't only invalid, but also causes an automatic implementation ofTryFrom, calls to which then look safe but actually aren't.This feels like a giant footgun:
ClarityName::try_from::<String>is perfectly safe and works as intended, butClarityName::try_from::<&str>will panic instead of returning an error.Luckily, all calls with truly external data used the former version, but using the latter would be a very easy mistake to make.
Initially I wanted to completely remove this implementation, but it's used a loooot with hardcoded strings (both in tests and in release code). For such strings, it feels safe enough, so for now I just changed it to require&'static str.Happy to discuss this approach though. I still think that all else being equal, fully removing it would be safer. The current approach is a compromise between blast radius and safety.I ended up fully removing it, and instead creating
Foo::from_literal()and making things explicit everywhere.