The Bitcoin Mempool: Why We Have Filters

In my prior article on the mempool, I laid out a simple conceptual framework to reason about the basic functionality of the mempool, and how it was used by different kinds of users of the Bitcoin network. In this piece I will be looking at the differences between relay policy and consensus rules, and why by default Bitcoin nodes do not relay some types of bitcoin transactions despite being consensus valid. First and foremost, regardless of the peer-to-peer network refusing to relay certain kinds of consensus valid transactions, if those transactions were to find up in a miner’s mempool and be selected for inclusion in a block, they will be received and downloaded by nodes when they receive that block. Nothing can prevent this short of consensus changes to make those classes of transactions invalid under consensus rules. There are different types of filters for different reasons. The three general types of filters are those protecting nodes (and therefore the network) from Denial of Service (DoS), those protecting upgrade hooks for future softforks, and those gently discouraging things that Bitcoiners might not like but otherwise present no serious harm to individual nodes or the network. Denial of Service Vectors Bitcoin nodes are computer programs running on computers. This means they have all the technical constraints of any programming running on any computer, limitations for storage, memory, processing power, etc. This is the root of why the blocksize limit was introduced and maintained, so as to create a global constraint keeping the verification costs reasonable for normal devices. This class of filters is designed specifically to ensure that even with the blockspace limit individual transactions that can be created that can consume too much of a node’s resources do not do so. The simplest example of such a filter is the minimum feerate needed for a transaction to propagate, and the Replace-By-Fee (RBF) rules dictating when a different version of the same transaction can replace the previous one, i.e. only when it pays a higher fee than the last version. Once you sign a transaction with a fee, you are on the hook. Unless you doublespend it, any miner who gets that transaction can mine it and collect that fee. There is no way to escape paying that cost other than spending your UTXO in a different transaction first (which also requires a fee). The reason for this is DoS protection. Without having to put themselves on the hook for a fee that they can’t escape paying, a user could simply create infinite variations of a single transaction and spam the mempools of every node on the network, eating bandwidth and memory in the process. Nothing would be stopping them from doing this forever. Nodes on the network would outright crash, or bandwidth costs become so exorbitantly high that users couldn’t afford them. Another example of transactions filtered by relay policy are expensive to validate transactions. It is possible to create transactions that are incredibly expensive to verify. Some blocks can be created that will take a Bitcoin node running on normal consumer hardware over an hour to verify. This is done by creating large custom scripts that are designed to create the maximum amount of signature checks that can be and stuffing a block full of nothing but these transactions. Such script structures have been constructed before and verification times tested on different types of machines, but the exact structure of those scripts has not been publicly revealed by the developers who did so for obvious reasons. These are transactions that could literally stall the entire network. A last example of DoS protection would be the dust limit. Transactions creating UTXOs with a satoshi value below the dust limit are not relayed because the fee to spend that UTXO would be higher than the satoshi value of the output. This makes it uneconomical and unlikely that it would ever be spent, meaning that the UTXO set would have to store these outputs forever. This could create a bloating UTXO set that makes block validation more computationally intensive. Future Softforks All major upgrades to the Bitcoin protocol have been done with softforks, an upgrade mechanism that allows new script functionality to be added to the protocol in a way that un-upgraded nodes will still accept as valid. This is possible because Bitcoin script includes “undefined” opcodes, meaning that any use of them automatically is considered valid because no verification rules are currently defined for them. When people upgrade their nodes to enforce the new rules, upgraded nodes will apply the new rules against that opcode, and older ones will simply accept any use of them. As long as miners do not mine transactions violating the new rules before the network of nodes all upgrade, everyone stays on the same blockchain and everything is backwards compatible. Transactions using these undefined opcodes are filtered by relay policy. This is done in order to preserve the upgradeability of the Bitcoin protocol in the future. If users were to make UTXOs using such undefined opcodes, say in combination with a defined ones so that they weren’t spendable by anyone, if that undefined opcode were given verification rules in a softfork that UTXO would become unspendable. The structure of the script would not be able to meet the new verification rules applied during the softfork. Allowing these to propagate and be confirmed could allow UTXOs using undefined opcodes to turn any potential softfork upgrade in the future into a philosophical dilemma of not upgrading or rendering some user’s coins unspendable. Discouragement There are some types of transactions that while causing no actual harm to nodes on the network, i.e. crashing nodes, using excessive memory or resources, a large segment of network users find undesirable or contrary to the primary purpose of Bitcoin. Examples of such transactions would be those making use of large OP_RETURN outputs, or Inscriptions making use of the Witness field, to write arbitrary information to the blockchain. These are discouraged because they are not seen as a primary use case of the Bitcoin network. Not Everything Is The Same These different classes of filters in relay policy are very clearly distinctly different things. Not all relay filters exist for the same reason, not all of them involve the same incentives for miners to mine (or not mine) them. Each of them exists for a specific purpose to protect your node, or the blockchain, from different types of things that are either legitimately damaging or just undesirable. All filters are not the same, and the difference between the things they are filtering is massive. Everything from problematic transactions that could crash the network (which should be fixed at the consensus level), to just discouraging harmless transactions that people find undesirable. It’s important to realize the difference between these things. For instance, a miner might mine a simply undesirable transaction if a user pays for it, but no rational miner would construct and mine a block full of transactions that would crash the entire network. That would undermine their investment. Source: https://bitcoinmagazine.com/technical/the-bitcoin-mempool-why-we-have-filters