Review of Augur and Some Questions

[ismaelsadeeq]

Posting feedback here since discussions aren't enabled in the repo and no post in a technical development forum

The tool provides fairly accurate estimates for "as soon as possible" targets. The benchmarks in the blog post https://engineering.block.xyz/blog/augur-an-open-source-bitcoin-fee-estimation-library are quite impressive.

Questions on Methodology:

• How did you benchmark these fee rate estimators? Are the benchmarks reproducible?
• How reliable is historical mempool data in predicting potential inflow? Did you validate this empirically? For example, why did you choose to immediately add half of the short-term inflow to the latest mempool snapshot to construct current weights with a buffer?
  I think based on past observations, user behavior is highly dynamic, and inflow metrics are often noisy for predicting long-term targets (especially for targets > 3 blocks).

Inaccurate data point

• Some transaction fee rates do not reflect their effective chunk fee rates (The effective mining score). For example, a high-fee parent transaction using CPFP to pay for a low-fee child transaction may not be properly represented here because you take the base fee of the transactions as the chunk fee rate which is not accurate hence the current mining algorithm in Augur may have skewed data points toward the high end, However this issue is hard to resolve with the current Bitcoin Core API, and Bitcoin Core estimator also has this limitation, an approach is to only consider individual transactions alone aka singletons see https://delvingbitcoin.org/t/package-aware-fee-estimator-post-cluster-mempool/312/10?u=ismaelsadeeq It will be easy to solve this issue after cluster mempool work and possibly Bitcoin Core will expose an API that returns the full linearization of the mempool with chunk fee rate groupings corresponding to weights.

[kevkevinpal]

Hey @ismaelsadeeq

fyi I'm not a maintainer of this repo but from inspection of https://github.com/block/bitcoin-augur-reference

But it seems like they fetch the mempool data via the getrawmempool rpc and then convert the transactions into singletons

in the getrawmempool rpc the attributes I see
descendantcount,
descendantsize,
descendantfees(DEPRECATED),
ancestorcount,
ancestorsize,
and ancestorfees(DEPRECATED)

do you think we can use those attributes to form non-singleton transactions to then make an estimate on?
Ideally, switching to cluster mempool linearizations after the cluster mempool work is finished.

[laurenshareshian]

Hello and apologies for the late reply... @zpv and I were traveling for a conference this week. Firstly, thanks for this feedback! I just turned discussions on and we're working on a Delving post as well.

• Re:benchmarking...
  @zpv created a Python benchmarking tool that we're also planning on sharing with the community so that anyone can reproduce the results and work on improving the algorithm. Sharing the benchmarking script is simple, but the less straightforward part that we're currently trying to figure out is how to share the data that the script uses, as it is located on proprietary datastores.

• Re: halving short term flow and predicting long term targets...
  Our initial v1 plan for this project was to follow Felix Weis' WhatTheFee algorithm closely, as we found that these estimates were the most accurate among the providers we explored. To that end, we halved the short term flow as he does here, as well as weighted the shorter and longer block target estimates closer to the 30 minute and 24 hour inflow calculations, respectively, as observed here. Totally agree that these choices can be explored for improvements.

• Really appreciate you sharing the cluster mempool context. I'm admittedly pretty new to the Bitcoin community so any insight and resources are much appreciated.

[ismaelsadeeq]

@kevkevinpal

Do you think we can use those attributes to form non-singleton transactions to then make an estimate on?

I believe you meant filtering to get only singletons. Yes, I think you can do that by considering only transactions whose ancestor score and descendant score are the same as the base fee. This indicates that the transaction is a singleton cluster.
In getrawmempool, this is where descendant, ancestor, and fees are equal.

However, I did some research and found that although singletons dominate the mempool, you can still find a significant number of non-singletons, and in some cases, they are not even CPFP. Each transaction pays for itself, so you may reduce data point by just considering singletons. Technically I think that’s what prevents us from completely ignoring non-singletons and awaiting cluster mempool in core because you have to detect and ignore ONLY CPFPs, or include them but with their effective mining score, not their individual fee rates.
See: bitcoin/bitcoin#30079 (comment) Although I personally think it’s better to have fewer data points than incorrect ones.
So @kevkevinpal, I’m happy to collaborate on or review a PR for this.

@laurenshareshian

Re: Benchmarking...

It would be really great to see a tool that helps evaluate fee rate forecasters in a reproducible way. Open-sourcing the tool and its data source could be highly valuable and helpful for the community (a work I and other contributors started but did not complete so it is still a valuable TODO). It could potentially serve as a foundation for making informed decisions on which forecaster to use.
The potential path for Core now is to support multiple fee rate forecasters, with a forecaster manager that selects the appropriate one based on heuristics and observed mempool and block conditions. ^[0]

Re: Halving short-term flow and predicting long-term targets...

Yes, the choice matters it determines how far off the inflow can make an estimate.
The current method I’ve implemented for mempool-based ASAP fee rate forecasting uses percentile fee rates specifically the 50th and 75th percentiles a generated block template from node's mempool. ^[1]

The idea is that some inflow is expected after an estimate (Because of high demand, or confirmation delay), and using a higher percentile makes it less likely your transaction gets bumped out of the block template. (However, this choice was arbitrary but backed with empirical data we’ve observed that it’s highly unlikely to get bumped with those percentiles. ^[1])

So, Augur’s approach differs from mine in that it uses historical inflow to decide which percentile to select in the generated block template/s.
This might potentially be better than choosing blindly, but it can also be misleading and lead to overshooting especially if the mempool was recently cleared but past inflow data suggests otherwise, or in another case there was no inflow in past and then you select a low percentile, but block get delayed a bit effectively getting bumped (although undershooting is not much of a concern because you can easily fee bump, overshooting is the issue), we can also prevent this from Augur by not picking the last tx to get in runSimulation when addedWeightsInOneBlock is not much, go a bit higher up to account for potential confirmation time delay (hence I was asking the rationale for some constants I see).

Personally, I’m more focused on ASAP fee rate estimates, as I haven’t yet seen strong utility in long-term forecasts most use cases seem to rely on ASAP estimates. ^[2]
For long-term forecasting, I do like the idea of the LND budget sweeper ^[3].

Really appreciate you sharing the cluster mempool context. I'm admittedly pretty new to the Bitcoin community so any insight and resources are much appreciated.

And we're working on a Delving post as well.

That’s good to hear. I believe it would be great to get the opinions of other contributors there too. I guess you’ll beat me to it then I’ve been working on one as well.
See draft: https://hackmd.io/@kEyqkad6QderjWKtcBF5Hg/SkCdR-E4gl
I’ve also experiment porting the ideas of augur into the mempool-based forecaster I’ve implemented (will switch after evaluating the effectiveness of the forecaster):
bitcoin/bitcoin@master...ismaelsadeeq:bitcoin:new-fee-estimator-with-inflow

• [0] Add Fee Rate Forecaster Manager – Bitcoin Core PR Review Club Notes
• [1] Mempool-Based Fee Estimator on Bitcoin Core
• [2] Utility of Long-Term Fee Estimation
• [3] LND Smart Deadline-Aware Budget Sweeper

---

Other relevant links:

• Package-Aware Fee Estimator
• bitcoind Block Policy Modes Analysis
• Empirical Data Analysis of Fee Rate Forecasters for ASAP Fee Estimation

[laurenshareshian]

That’s good to hear. I believe it would be great to get the opinions of other contributors there too. I guess you’ll beat me to it then I’ve been working on one as well.
See draft: https://hackmd.io/@kEyqkad6QderjWKtcBF5Hg/SkCdR-E4gl

Wow, this is cool! Our Delving post coming later this week won't be as detailed as yours - it'll be pretty similar to the original post here. If you'd like to follow it up with a more detailed post then feel free to do so. In the meantime, I'm looking forward to digging into the relevant links you've generously included here, thanks!

[laurenshareshian]

Hey @ismaelsadeeq just in case you didn't see it, @zpv released his benchmarking tool in the delving replies: https://delvingbitcoin.org/t/augur-block-s-open-source-bitcoin-fee-estimation-library/1848/5

[ismaelsadeeq]

Great, I reproduced the benchmarks locally.

Thanks for open sourcing it 🦂

[laurenshareshian]

No problem. If you have any suggestions for what to add, assumptions to improve, etc., please let us know.