<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Anatolii Zhadan]]></title><description><![CDATA[Anatolii Zhadan]]></description><link>https://anatoliizhadan.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6a4c06e28a429c1defdba5ba/1073c591-f1c1-40bc-9f21-f641a27ce4f1.png</url><title>Anatolii Zhadan</title><link>https://anatoliizhadan.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 13:17:21 GMT</lastBuildDate><atom:link href="https://anatoliizhadan.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Reading Postgres EXPLAIN ANALYZE when the planner expects 300 rows and gets 15,000]]></title><description><![CDATA[A bad plan starts at the lowest node where expected and actual rows split, and every node above it is built on that wrong number.]]></description><link>https://anatoliizhadan.hashnode.dev/reading-postgres-explain-analyze-when-the-planner-expects-300-rows-and-gets-15-000</link><guid isPermaLink="true">https://anatoliizhadan.hashnode.dev/reading-postgres-explain-analyze-when-the-planner-expects-300-rows-and-gets-15-000</guid><category><![CDATA[PostgreSQL]]></category><category><![CDATA[Explain Analyze]]></category><category><![CDATA[query-optimization]]></category><category><![CDATA[database]]></category><category><![CDATA[SQL]]></category><dc:creator><![CDATA[Anatolii Zhadan]]></dc:creator><pubDate>Tue, 17 Nov 2026 23:09:08 GMT</pubDate><enclosure url="https://miro.medium.com/v2/1*QoJhgUlzFHHCW0y6FXhf7g.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>Slow Postgres, part 2 of 8. The SQL and raw numbers for every test are <a href="https://github.com/ImmuneFOMO/research/tree/main/postgres/slow-postgres/02-reading-explain-analyze">on GitHub</a>.</em></p>
<p><img src="https://miro.medium.com/v2/1*1RDFj1zWfj7_Fd1EKaE22g.png" alt="The users scan expected 300 rows and got 15,000" />
<em>Expected and actual rows per node, read from the bottom up.</em></p>
<p>Postgres guessed that 300 users live in Lisbon. There are 15,000. That one line of the plan explains the rest: a Nested Loop that ran an index lookup 15,000 times, a sort that spilled to disk, and a query that takes 159 ms with the data in memory and 2.15 seconds when it comes off disk.</p>
<p><a href="https://hackernoon.com/postgres-not-using-your-index-its-counting-pages-not-rows">Part 1</a> was about how the planner prices a plan. This one reads what happened when it ran, and how much EXPLAIN ANALYZE itself adds.</p>
<p>One note before the numbers: the AlphaVPS link at the end is a referral link, so I may get a commission at no extra cost to you, and the netcup voucher I offer there comes from my own account.</p>
<p>The short version:</p>
<ul>
<li>Compare estimated and actual rows on every node, from the bottom up. Here a guess of 300 users picked a Nested Loop that won in memory (159 ms against 344) and lost from disk (2.15 s against 455 ms).</li>
<li><code>actual time</code> and <code>rows</code> are per loop: an index scan that says <code>rows=0 loops=15000</code> returned 3,071 rows.</li>
<li><code>read</code> in <code>Buffers</code> means "not in shared_buffers", not "from disk".</li>
<li>EXPLAIN ANALYZE made <code>count(*)</code> 2.4 to 3.0 times slower on all four of my machines, so compare plans with <code>TIMING OFF</code> or the plain query.</li>
</ul>
<h2>The lab: a million users, five million orders</h2>
<p>PostgreSQL 17.12 in Docker, on the same 4-vCPU, 8 GB cloud VM as part 1. Every user has five orders, and <code>orders.user_id</code> has an index. The city decides the country: 15,000 users live in Lisbon, 5,000 in Porto, and those 20,000 are all of <code>PT</code>. The rows come from <code>generate_series</code> and <code>setseed</code>, so yours match mine. <code>shared_buffers</code> is 1 GB so everything fits in memory, and JIT and parallel workers are off to keep the plans short.</p>
<p>The report counts orders of Lisbon users by status:</p>
<pre><code class="language-sql">SELECT o.status, count(*) AS orders, sum(o.amount_cents) AS cents
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.country = 'PT' AND u.city = 'Lisbon'
GROUP BY o.status
ORDER BY orders DESC;
</code></pre>
<h2>EXPLAIN guesses, EXPLAIN ANALYZE runs</h2>
<p><code>EXPLAIN</code> prints the plan with the planner's forecast for each node, <code>cost</code> and <code>rows</code>. <code>EXPLAIN ANALYZE</code> runs the query and adds what happened: <code>actual time</code>, actual <code>rows</code> and <code>loops</code>. <code>BUFFERS</code> adds the pages each node touched, and PostgreSQL 18 turns it on by default with <code>ANALYZE</code>. The result rows are thrown away, so <code>Execution Time</code> leaves out sending them to the client and turning values into text (the <code>SERIALIZE</code> option, new in PostgreSQL 17, measures the second part).</p>
<p>Writes run too, so wrap them in a transaction:</p>
<pre><code class="language-sql">BEGIN;
EXPLAIN (ANALYZE, BUFFERS) DELETE FROM orders WHERE status = 'pending';
ROLLBACK;
</code></pre>
<p><code>ROLLBACK</code> brought back all 149,701 pending orders, but the work still happened: the DELETE dirtied 36,469 pages that still get written to disk. Sequences don't roll back either. After a rolled-back <code>EXPLAIN ANALYZE INSERT</code> of 1,000 rows, the next real row got id 1001.</p>
<h2>Read it from the inside out</h2>
<p>The report's plan, with some lines cut (the full output is in <code>02_plans.out</code>):</p>
<pre><code class="language-text">Sort  (cost=31759.19..31759.20 rows=4 width=21) (actual time=209.555..209.560 rows=4 loops=1)
  Buffers: shared hit=129440, temp read=205 written=205
  -&gt;  GroupAggregate  (cost=31744.11..31759.15 rows=4 width=21) (actual time=201.392..209.511 rows=4 loops=1)
        -&gt;  Sort  (cost=31744.11..31747.86 rows=1500 width=9) (actual time=201.037..205.049 rows=75000 loops=1)
              Sort Method: external merge  Disk: 1640kB
              -&gt;  Nested Loop  (cost=0.43..31664.98 rows=1500 width=9) (actual time=0.030..189.952 rows=75000 loops=1)
                    -&gt;  Seq Scan on users u  (cost=0.00..24434.00 rows=300 width=4) (actual time=0.013..47.068 rows=15000 loops=1)
                          Filter: ((country = 'PT'::text) AND (city = 'Lisbon'::text))
                          Rows Removed by Filter: 985000
                    -&gt;  Index Scan using orders_user_id_idx on orders o  (cost=0.43..24.05 rows=5 width=13) (actual time=0.003..0.009 rows=5 loops=15000)
                          Index Cond: (user_id = u.id)
                          Buffers: shared hit=120000
Planning Time: 0.247 ms
Execution Time: 209.990 ms
</code></pre>
<p>A plan prints top-down, but rows flow bottom-up. The users scan read a million rows and kept 15,000. For each of them, the Nested Loop ran the index scan on orders. The inner Sort ordered the 75,000 orders by status, GroupAggregate collapsed them into four rows, and the top Sort ordered those.</p>
<p>Each node has two sets of parentheses. <code>cost=31744.11..31747.86 rows=1500</code> is the forecast, in planner units that are priced in pages and don't convert to milliseconds. <code>actual time=201.037..205.049 rows=75000</code> is what happened: milliseconds until the node's first row and until its last. The inner Sort's first row came at 201 ms because a sort returns nothing until it has read all its input.</p>
<p>Times are cumulative. The Nested Loop finished at 190 ms, 47 ms of which was the users scan, so the 15,000 index lookups took about 143 ms. To find the slow node, subtract each node's children from it and look for the biggest remainder. <code>Buffers</code> add up the same way.</p>
<h2>Estimated rows against actual rows</h2>
<p>The users scan expected 300 rows and returned 15,000. The 300 comes from <code>pg_stats</code>: <code>PT</code> is 2% of users, <code>Lisbon</code> is 1.5%, and the planner treats the two conditions as independent and multiplies them, 1,000,000 × 0.02 × 0.015 = 300. Every Lisbon user is also in <code>PT</code>, so the country condition removes nobody.</p>
<p>Go up one node. The Nested Loop was planned for 300 index lookups returning 1,500 orders and ran 15,000 lookups returning 75,000. The Sort above it was planned for 1,500 rows, got 75,000 and spilled to disk. So walk up from the leaves comparing the two <code>rows=</code> and start at the lowest node where they differ by 10x or more. Under a <code>LIMIT</code> a node stops early, so fewer actual rows there is expected.</p>
<p>Did the wrong guess hurt? Drop the redundant <code>country = 'PT'</code> and the planner estimates 74,999 joined rows against 75,000 actual, and switches to a Hash Join: read all five million orders once and match them against a hash of the 15,000 users. I timed both plans without EXPLAIN, with the data in three places:</p>
<p><img src="https://miro.medium.com/v2/1*5ukP9cMUA6iCVCYk_eaU6A.png" alt="Nested Loop against Hash Join: 159 vs 344 ms in memory, 2.15 s vs 455 ms from disk" />
<em>Plain query time, median of 7 warm runs and of 5 runs after each restart, on the 4-vCPU cloud VM. The OS cache was dropped before every disk run.</em></p>
<p>In memory the plan built on the wrong guess won, 159 ms against 344, because 15,000 index lookups in RAM cost less than reading five million rows. From disk it lost, 2.15 s against 455 ms. The planner can't see what's cached, and its page costs put 15,000 scattered lookups far above one pass over the table, which is how they behaved cold. A warm EXPLAIN ANALYZE shows only one side of that trade. Where estimates come from and why they drift is part 3.</p>
<h2>Loops: the numbers are per loop</h2>
<p>The index scan on orders says <code>rows=5 loops=15000</code>: the Nested Loop ran it once per user. <code>actual time</code> and <code>rows</code> are averages per loop, like the estimate next to them: 5 × 15,000 = 75,000 rows in about 0.009 × 15,000 = 135 ms, close to the 143 ms from the subtraction above.</p>
<p><code>Buffers</code> is a total over all loops. Its 120,000 pages are 8 per lookup: three levels of the B-tree plus five heap pages for five orders spread through the table. PostgreSQL 18 adds an <code>Index Searches</code> line to index scans, also a total.</p>
<p>Per-loop averages are rounded, too. The same join, keeping only failed orders:</p>
<pre><code class="language-text">Nested Loop  (cost=0.43..31656.73 rows=62 width=20) (actual time=0.240..147.007 rows=3071 loops=1)
  -&gt;  Index Scan using orders_user_id_idx on orders o  (cost=0.43..24.07 rows=1 width=24) (actual time=0.006..0.007 rows=0 loops=15000)
        Index Cond: (user_id = u.id)
        Filter: (status = 'failed'::text)
        Rows Removed by Filter: 5
</code></pre>
<p><code>rows=0</code> doesn't mean the scan found nothing. 3,071 rows over 15,000 loops is 0.2 per loop, which PostgreSQL 17 rounds to 0 and 18 prints as <code>rows=0.20</code>. <code>Rows Removed by Filter: 5</code> is per loop too, about 72,000 rows in all. Parallel workers count as loops: with them on, a Parallel Seq Scan on orders showed <code>rows=1666667 loops=3</code>, the leader and two workers.</p>
<h2>Rows Removed by Filter</h2>
<p>In the report, <code>Rows Removed by Filter: 985000</code> on the users scan means Postgres fetched a million rows and checked each one to keep 15,000. The orders scan has an <code>Index Cond</code> instead: the index found the rows, and nothing was thrown away.</p>
<p>On a Seq Scan, a large count is where an index might help (part 1 covers when it won't). On an index scan, it means the index covers only part of the <code>WHERE</code> clause and the rest is checked row by row. In the failed-orders join, the index found all five orders of each user and the filter threw away 96% of them. Which index fits a query like that is part 4.</p>
<h2>BUFFERS: hit, read, and a cold cache</h2>
<p><code>shared hit</code> is a page found in shared_buffers. <code>read</code> is a page Postgres asked the OS for, which came from the OS page cache or from disk, and <code>BUFFERS</code> can't tell which. The counts aren't distinct pages: a page visited twenty times is twenty hits. The report's Nested Loop plan in all three states:</p>
<pre><code class="language-text">                          Buffers                    I/O wait    query time
in shared_buffers         hit=129440                 -           159 ms
restart, OS cache warm    hit=87639  read=41801      164 ms      304 ms
restart, OS cache empty   hit=87639  read=41801      2,011 ms    2.15 s
</code></pre>
<p>I/O wait is the median from the EXPLAIN runs, query time from the plain ones. Both restarted runs read the same 41,801 pages, and the second was seven times slower because those reads went to disk.</p>
<p>The I/O wait comes from <code>track_io_timing</code>, which is off by default and which a superuser can turn on for the session. Every node then gets an <code>I/O Timings</code> line. From the median cold run of each plan, costs cut:</p>
<pre><code class="language-text">Nested Loop plan
-&gt;  Index Scan using orders_user_id_idx on orders o  (actual time=0.032..0.141 rows=5 loops=15000)
      Buffers: shared hit=87633 read=32367
      I/O Timings: shared read=1965.141

Hash Join plan
-&gt;  Seq Scan on orders o  (actual time=0.125..291.938 rows=5000000 loops=1)
      Buffers: shared read=37101
      I/O Timings: shared read=65.418
</code></pre>
<p>The index scan waited almost 2 seconds for 32,367 pages, about 60 µs each, as its lookups jumped around the table. The Seq Scan read 37,101 pages of the same table front to back in 65 ms, under 2 µs each. All 87,633 hits are repeat visits inside the index scan, like the top of the B-tree on every lookup.</p>
<h2>Sort and Hash: memory or disk</h2>
<p>Two kinds of lines say an operation didn't fit in memory. A sort gets <code>work_mem</code>, 4 MB by default, and a hash twice that, since <code>hash_mem_multiplier</code> defaults to 2. The report's inner Sort shows <code>Sort Method: external merge  Disk: 1640kB</code>: it wrote sorted runs to temp files, the <code>temp read=205 written=205</code> in its <code>Buffers</code> (205 pages of 8 kB). A Hash shows it as batches. This one joins all 200,120 failed orders to users, sorted by amount:</p>
<pre><code class="language-text">Hash  (cost=19434.00..19434.00 rows=1000000 width=26) (actual time=186.969..186.971 rows=1000000 loops=1)
  Buckets: 131072  Batches: 16  Memory Usage: 4695kB
</code></pre>
<p><code>Batches: 16</code> means the hash table was split into 16 parts and 15 of them waited in temp files. The final sort spilled too, 7,448 kB.</p>
<p>At 64 MB of <code>work_mem</code> (128 MB for the hash) both fit in memory, the hash in one 66,782 kB batch, and on the cloud VM the query got slower: 425 ms to 578 ms with <code>TIMING OFF</code>, median of seven. On my M3 Max laptop it got faster, 418 ms to 341 ms. I haven't found out why. CPU cache was my first guess, since a 4 MB batch fits in it and a 66 MB table doesn't, but that doesn't explain the laptop, so it stays unconfirmed.</p>
<p>The report's own 1.6 MB spill cost nothing I could measure: 158.6 ms at 4 MB, 158.0 ms at 64 MB. A <code>Disk:</code> line says a node spilled, and only a rerun with more <code>work_mem</code>, on the machine that serves the query, says whether the spill is slow. Each sort and hash in each connection can take that much, so raise it per session or per role.</p>
<h2>Planning time and execution time</h2>
<p>The report spent 0.25 ms planning and 210 ms executing, the usual split for a slow query. <code>Planning Time</code> counts on queries that run in microseconds, like a primary-key lookup on users right after a restart:</p>
<pre><code class="language-text">first run in a new connection:  Planning Time: 0.430 ms   Execution Time: 0.078 ms
                                Planning: Buffers: shared hit=52 read=21
second run:                     Planning Time: 0.013 ms   Execution Time: 0.006 ms
</code></pre>
<p>The first plan in a new connection reads table and index details from the system catalogs, 73 pages here, and took five times longer than the query. After that, planning still took twice as long as execution. If a lookup like this runs thousands of times a second over short-lived connections, a pooler that keeps connections open removes that first cost.</p>
<h2>What EXPLAIN ANALYZE costs</h2>
<p>To time each node, EXPLAIN ANALYZE reads the clock before and after every call into it, twice for every row each node returns. If millions of rows pass through a cheap node, the clock reads can cost more than the work.</p>
<p>On my laptop, the cloud VM, an AlphaVPS server in Los Angeles and an ARM server at netcup, I ran <code>pg_test_timing</code>, which ships with Postgres and times a loop around one clock read, then <code>SELECT count(*)</code> over 10 million rows, seven times each way.</p>
<p><img src="https://miro.medium.com/v2/1*TGOOcxgCW-BpqU_-n0j7eg.png" alt="count(*) over 10 million rows, plain against EXPLAIN ANALYZE, on four machines" />
<em>Median of 7 runs per variant, data in memory, one backend, PostgreSQL 17.7 on the laptop and 17.12 on the three servers.</em></p>
<p><code>pg_test_timing</code> gave 18 to 40 ns per loop, and EXPLAIN ANALYZE still made <code>count(*)</code> 2.4 to 3.0 times slower everywhere. The extra time per row followed the clock: 24 ns on the laptop, 44 on the VM, 59 on the Xeon, 74 on the ARM server, or 1.3 to 1.9 times the <code>pg_test_timing</code> figure. On the VM, switching the clock source from <code>tsc</code> to <code>kvm-clock</code> raised both, to 28 and 50 ns, and the slowdown to 3.2 times. A nested loop with a million index lookups does more work per row, and there EXPLAIN ANALYZE added 4% to 15%. The report lost 7%, 159 ms to 170.</p>
<p>I expected a bigger gap between machines, but all four clocks were fast. The <a href="https://www.postgresql.org/docs/17/pgtesttiming.html">pg_test_timing docs</a> show a slow one, <code>acpi_pm</code>, where a loop took 723 ns and EXPLAIN ANALYZE turned a 9.8 ms count into 115.9 ms. The AlphaVPS server offers <code>acpi_pm</code> too, but I didn't switch a live server's clock.</p>
<p>The overhead also skews comparisons between plans. Under EXPLAIN ANALYZE the Hash Join version of the report took 591 ms and the Nested Loop 170 ms, 3.5 times apart. Run plain, they took 344 ms and 159 ms, 2.2 times apart. The Hash Join pushes five million orders through its Seq Scan node, ten million clock reads, while no node in the Nested Loop plan returns more than 75,000 rows.</p>
<p><code>TIMING OFF</code> keeps rows, loops and buffers and drops the per-node times. It still counts every row, so <code>count(*)</code> ran 16% to 19% slower than plain even then. Inside one plan the per-node times are the best map you have, but the nodes passing the most rows carry the most overhead.</p>
<h2>The order I read a plan in</h2>
<ol>
<li>Run <code>EXPLAIN (ANALYZE, BUFFERS)</code>, inside <code>BEGIN</code> and <code>ROLLBACK</code> if it writes.</li>
<li>From the innermost nodes up, find the lowest one where estimated and actual <code>rows</code> differ by 10x or more.</li>
<li>Multiply per-loop numbers by <code>loops</code>, then subtract each node's children to see where the time went.</li>
<li>Compare <code>Rows Removed by Filter</code> with the rows kept.</li>
<li>Look for <code>Disk:</code> and <code>Batches:</code> above 1, and rerun with more <code>work_mem</code> before blaming the spill.</li>
<li>Read <code>Buffers</code> with <code>track_io_timing</code> on to see which node waited for reads.</li>
<li>Before comparing two plans, time the plain queries, warm and cold.</li>
</ol>
<h2>Run it on your own server</h2>
<p>The walkthrough plans come from one 4-vCPU AMD EPYC cloud VM with 8 GB of RAM and PostgreSQL 17.12 in Docker, so your milliseconds will differ while the row counts match. The <code>work_mem</code> check also ran on my M3 Max laptop (Homebrew PostgreSQL 17.7), and the overhead test ran on the laptop, the VM and an AlphaVPS server in Los Angeles, a 4-vCPU Xeon Platinum 8176 with 8 GB of RAM (PostgreSQL 17.12 from conda-forge), and a netcup server with 6 ARM Neoverse-N1 cores and 8 GB of RAM (PostgreSQL 17.12 in Docker). The SQL, scripts, raw output and chart code are <a href="https://github.com/ImmuneFOMO/research/tree/main/postgres/slow-postgres/02-reading-explain-analyze">on GitHub</a>, and <code>bench/overhead.sh</code> runs the overhead test on a new Linux box with one command. Run it on your own server, and if you get a different picture or think I got something wrong, open an issue or a PR.</p>
<p>My own projects run on netcup (VPS from €2.21 a month) and <a href="https://alphavps.com/clients/aff.php?aff=1157">AlphaVPS</a> (VPS from €2.54 a month). For netcup, message me on <a href="https://www.linkedin.com/in/anatolii-zhadan/">LinkedIn</a> and I'll send you a €5 voucher for new customers.</p>
<p>Good luck with your plans. Part 3 is next: why Postgres statistics go stale.</p>
<h2>Finish</h2>
<p>I hope you understood the instructions correctly. If you have any edits to this article, I will be waiting for your comments. If I really helped you, I kindly ask you to subscribe to my LinkedIn and provide feedback on this article.</p>
<p>Link to follow me:</p>
<p><a href="https://www.linkedin.com/in/anatolii-zhadan/">LinkedIn</a></p>
<p>All my articles and links: <a href="https://fomo.moe">fomo.moe</a></p>
]]></content:encoded></item><item><title><![CDATA[Postgres not using your index? It's counting pages, not rows]]></title><description><![CDATA[You get a Seq Scan even with an index because the planner prices a query in pages, and a small table is cheaper to read whole.]]></description><link>https://anatoliizhadan.hashnode.dev/postgres-not-using-your-index-it-s-counting-pages-not-rows</link><guid isPermaLink="true">https://anatoliizhadan.hashnode.dev/postgres-not-using-your-index-it-s-counting-pages-not-rows</guid><category><![CDATA[PostgreSQL]]></category><category><![CDATA[database]]></category><category><![CDATA[SQL]]></category><category><![CDATA[performance]]></category><category><![CDATA[Backend Development]]></category><dc:creator><![CDATA[Anatolii Zhadan]]></dc:creator><pubDate>Fri, 13 Nov 2026 14:08:42 GMT</pubDate><enclosure url="https://miro.medium.com/v2/1*eBtvvTUPOaxq-N6E-VnR9A.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>Slow Postgres, part 1 of 8. The SQL and raw numbers for every test are <a href="https://github.com/ImmuneFOMO/research/tree/main/postgres/slow-postgres/01-index-vs-seq-scan">on GitHub</a>.</em></p>
<p><img src="https://miro.medium.com/v2/1*2eE46bh5fvKKaJ0KJ6YAig.png" alt="Seq Scan reads heap pages in order; Index Scan reads the B-tree, then one heap page per TID" />
<em>A Seq Scan reads each page once, in order. An index lookup walks the B-tree and then reads one heap page per TID.</em></p>
<p>You add an index on <code>status</code>, run <code>EXPLAIN</code>, and Postgres reads the whole table anyway. <code>ANALYZE</code> doesn't change the plan. Someone suggests <code>SET enable_seqscan = off</code>, the plan finally shows your index, and the query gets slower.</p>
<p>Usually the planner is right. It prices a plan by the 8 KB pages it has to read and by whether it reads them in order or jumps around between them, and once you count pages the same way, you can answer most "why is it ignoring my index" tickets yourself.</p>
<p>This series started as my notes from an <a href="https://www.youtube.com/watch?v=AZF9QZeJ-v8">open lesson on slow Postgres queries</a> by balun.courses. The tests and numbers are my own.</p>
<p>One note before the numbers: the AlphaVPS link at the end is a referral link, so I may get a commission at no extra cost to you, and the netcup vouchers come from my own account.</p>
<p>The short version:</p>
<ul>
<li>Postgres prices a plan in pages, and in my test table a status with 0.9% of the rows sits on 94% of them, so its index saves 12 page reads out of 221.</li>
<li>On 5 million rows in RAM, the index beat a full scan up to 10% of the rows when the matches were scattered, and at every share I tried (up to 50%) when they were stored together.</li>
<li>Read cold from disk, the scattered lookups at 0.5% and 1% of rows ran 2 to 5 times slower than a full scan, and Postgres still picked them.</li>
<li>A 200-row table gets a Seq Scan even when 2 rows match, so a plan from a tiny test copy says nothing about production.</li>
<li>Keep <code>enable_seqscan</code> out of app code, read <code>pg_stats</code> before <code>CREATE INDEX</code>, and if you only query the rare values, index only those rows.</li>
</ul>
<h2>What a B-tree index stores</h2>
<p>A regular B-tree index keeps only keys, the values it is sorted by, and next to each key a TID: the row's address in the heap, which is the table's main storage.</p>
<p>A TID is a page number plus a slot on that page. You can see it as the hidden <code>ctid</code> column:</p>
<pre><code class="language-sql">SELECT ctid, id, status FROM orders WHERE status = 'chargeback' ORDER BY id LIMIT 3;
</code></pre>
<pre><code class="language-text">  ctid   |  id  |   status
---------+------+------------
 (7,48)  | 1000 | chargeback
 (14,96) | 2000 | chargeback
 (22,8)  | 3000 | chargeback
</code></pre>
<p>Row 1000 is slot 48 on page 7. An index lookup for <code>'chargeback'</code> returns a list of addresses like these, and Postgres then visits each of those heap pages to fetch the row and check that your transaction can see it.</p>
<p>The second step is the one people forget. To fetch one row, Postgres has to read the whole 8 KB page it sits on.</p>
<h2>The lab: 30,000 orders on 221 pages</h2>
<p>The lab runs on PostgreSQL 17.11 in Docker with default settings. My numbers come from a 4-vCPU, 8 GB cloud VM, but any small VPS or a laptop will do.</p>
<pre><code class="language-sql">CREATE TABLE orders (
  id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  created_at   timestamptz NOT NULL,
  amount_cents int NOT NULL,
  status       text NOT NULL
);

INSERT INTO orders (created_at, amount_cents, status)
SELECT now() - (30000 - i) * interval '1 minute',
       (random() * 10000)::int,
       CASE
         WHEN i % 1000 = 0 THEN 'chargeback'  -- 30 rows, 0.1%
         WHEN i % 100  = 0 THEN 'failed'      -- 270 rows, 0.9%
         WHEN i % 10   = 0 THEN 'created'     -- 2,700 rows, 9%
         ELSE 'paid'                          -- 27,000 rows, 90%
       END
FROM generate_series(1, 30000) AS i;

CREATE INDEX orders_status_idx ON orders (status);
ANALYZE orders;

SELECT relpages, reltuples FROM pg_class WHERE relname = 'orders';
</code></pre>
<pre><code class="language-text"> relpages | reltuples
----------+-----------
      221 |     30000
</code></pre>
<p>136 rows fit on a page, so 30,000 rows take 221 pages. I picked 30,000 because <code>ANALYZE</code> samples 300 × <code>default_statistics_target</code> rows, which is 30,000 at the default. On this table it reads every row, so the statistics are exact and your plans should come out the same as mine.</p>
<h2>What a Seq Scan costs</h2>
<p>Ask for the most common value and the index goes unused:</p>
<pre><code class="language-text">EXPLAIN SELECT * FROM orders WHERE status = 'paid';

Seq Scan on orders  (cost=0.00..596.00 rows=27000 width=25)
  Filter: (status = 'paid'::text)

221 pages  × seq_page_cost     1.0    = 221
30000 rows × cpu_tuple_cost    0.01   = 300
30000 rows × cpu_operator_cost 0.0025 =  75
                                        ---
                                        596
</code></pre>
<p>Cost has no unit you can convert to milliseconds. It only means something next to another plan's cost.</p>
<p>The value you filter on appears nowhere in that formula. Ask for <code>'chargeback'</code> with the index paths switched off and the Seq Scan still costs 596. Only the index path's price moves with the value, so the planner is choosing between a fixed price and a sliding one.</p>
<p>For <code>'paid'</code>, every one of the 221 pages holds paid rows, so the index path would read the whole heap plus the index. Force it with <code>enable_seqscan = off</code> and you get a Bitmap Heap Scan at cost 872.</p>
<h2>How many pages each status sits on</h2>
<p>Selectivity is the share of rows that pass the filter. The usual rule of thumb goes: one row, use the index; 1%, usually; 20%, it depends; 70%, Seq Scan. Here is the same table counted by pages as well as by rows:</p>
<pre><code class="language-sql">SELECT status,
       count(*)                               AS rows,
       count(DISTINCT (ctid::text::point)[0]) AS pages
FROM orders
GROUP BY status
ORDER BY rows DESC;
</code></pre>
<p><img src="https://miro.medium.com/v2/1*moaoxnMGWcmLWhXq64ojaQ.png" alt="Bar chart: paid 90% of rows on 100% of pages, created 9% on 100%, failed 0.9% on 94%, chargeback 0.1% on 14%" />
<em>Share of rows and share of pages for each status in the 30,000-row table. failed is 0.9% of the rows but sits on 94% of the pages.</em></p>
<p>The rule of thumb says index <code>failed</code>, and Postgres agrees: Index Scan at cost 235 against 596. Run both with <code>EXPLAIN (ANALYZE, BUFFERS)</code>, though, and the index path touches 209 buffers to the Seq Scan's 221, a saving of twelve page reads.</p>
<p>It still finished faster, 0.19 ms versus 0.93, because checking 270 rows is less CPU work than filtering 30,000. For <code>'chargeback'</code>, 30 rows on 30 pages, the index path needs 32 buffers instead of 221.</p>
<p>Two things decide how rows land on pages. Row width sets how many fit on one, and at 136 rows per page any value much more common than 1 in 136 ends up on nearly every page. Physical order is the other. I spread <code>failed</code> evenly on purpose. If your failed orders came from one bad deploy on one afternoon, they'd sit on a few neighbouring pages and the index would look great.</p>
<h2>Timing it on 5 million rows in RAM</h2>
<p>On 221 pages everything finishes in a couple of milliseconds, so for timings I built a bigger table: 5 million rows, 326 MB, about 120 rows per page.</p>
<p>A <code>bucket</code> column holds a random number from 0 to 999,999, so <code>WHERE bucket &lt; 10000</code> matches 1% of the rows. I loaded the same rows twice, once in random order and once sorted by <code>bucket</code>, and forced each plan with the <code>enable_*</code> settings at shares from 0.01% to 50%. <code>shared_buffers</code> was 2 GB, which kept the whole table in memory.</p>
<p><img src="https://miro.medium.com/v2/1*IPf38_tjy2BdHK2UbrDPdA.png" alt="Line chart of query time against share of matching rows: Seq Scan between 114 and 159 ms, scattered bitmap scan crossing it between 10% and 25%, clustered Index Scan below it all the way to 50%" />
<em>Median of 5 warm runs on 5 million rows, one process, JIT off. Dots mark the plan Postgres chose on its own. Grey dotted line: Seq Scan on the clustered copy.</em></p>
<p>For scattered rows Postgres picked a bitmap scan, which collects all the TIDs first and then reads each heap page once, in page order. Up to 10% of the rows it beat the Seq Scan, 98 ms to 124. At 25% it lost, 174 against 139, and the planner still chose it. A plain Index Scan, one heap visit per row, lost from 5% on.</p>
<p>By 10% the scattered matches sit on every single page, and the bitmap scan still won. It reads all 41,667 pages but checks only the 499,000 rows the bitmap points to, while the Seq Scan checks all 5 million. With the table in RAM a page read costs next to nothing, and the time goes into checking rows.</p>
<p>For clustered rows the index won at every share I tried, even 50%: 123 ms against 142.</p>
<h2>The same queries with a cold cache</h2>
<p>RAM is the friendly case. For the cold runs I restarted Postgres and dropped the OS page cache before every run, so every page came off the VM's disk.</p>
<p><img src="https://miro.medium.com/v2/1*FRLx1-1bbwdaZ7zg4O1EuA.png" alt="Line chart of the same queries with a cold cache: Seq Scan flat around 250 to 350 ms, scattered bitmap scan peaking at 1.27 s at 1% of rows, clustered Index Scan below the Seq Scan everywhere" />
<em>Same tables and queries read cold from disk, median of 5. Dots mark the plan Postgres chose. Grey dotted line: Seq Scan on the clustered copy.</em></p>
<p>The Seq Scan got about twice as slow, 250 to 350 ms at every share. The clustered index won everywhere again, 23 ms versus 263 at 5%.</p>
<p>The scattered bitmap scan took 748 ms at 0.5% of rows, to the Seq Scan's 349. At 1% it took 1.27 seconds against 255 ms, five times slower, and that's the plan Postgres picks.</p>
<p>The bitmap scan at 2% read more heap pages than at 1%, 37,939 versus 29,143, and still finished more than six times faster, 190 ms against 1,275. From 2% to 10% it beat the Seq Scan again.</p>
<p>The 2% run touched 91% of the pages, so it was close to one long sequential read. The 1% run touched 70%, with holes in between. My best guess is OS readahead, which prefetches reads that look sequential and gives up when they're full of gaps. I haven't confirmed it yet.</p>
<p>The planner has no way to see that difference. All it has is <code>random_page_cost = 4</code>, a single guess at how much a jump costs next to a sequential read, and it used the same guess for both runs.</p>
<h2>Seq Scans on a 200-row table</h2>
<p>A 200-row copy shows the other end of the scale, where the table is too small for any index to win.</p>
<pre><code class="language-sql">CREATE TABLE orders_small AS SELECT * FROM orders WHERE id &lt;= 200;
CREATE INDEX ON orders_small (status);
ANALYZE orders_small;

EXPLAIN SELECT * FROM orders_small WHERE status = 'failed';
</code></pre>
<pre><code class="language-text">Seq Scan on orders_small  (cost=0.00..4.50 rows=2 width=25)
  Filter: (status = 'failed'::text)
</code></pre>
<p>Only 2 of the 200 rows match, but reading two pages and checking 200 rows costs just 2 + 2 + 0.5 = 4.50. Turn off <code>enable_seqscan</code> and the cheapest index plan costs 6.19, because its one index page alone is priced as a random read at <code>random_page_cost = 4</code>. If you check plans on a 200-row test fixture, this is the arithmetic you're looking at, and it tells you nothing about the same query on 30 million rows in production.</p>
<p>"We're on SSDs, set it to 1.1." Fair, and on this table that flips the plan to an Index Scan at cost 2.77. Both versions finish in 10 microseconds or less, so on a table this size it makes no difference.</p>
<h2>A partial index instead of enable_seqscan</h2>
<p>The one place for <code>enable_seqscan = off</code> is a scratch session, where it shows you what the other plan would cost. It applies to every query in the session, so in app code it also overrides the plans that were right, like 596 against 872 for <code>'paid'</code>.</p>
<p>If the queries you care about only ask for the rare values, say a failed-payments dashboard or a chargeback queue, index only those rows:</p>
<pre><code class="language-sql">CREATE INDEX orders_status_not_paid_idx ON orders (status)
  WHERE status &lt;&gt; 'paid';
</code></pre>
<p>It holds 3,000 entries instead of 30,000 and takes 5 pages, where the full index takes 28. Paid orders, 90% of the inserts here, never touch it.</p>
<p>Postgres only uses it when it can prove your filter implies the index's <code>WHERE</code>, and a literal <code>status = 'chargeback'</code> does. With only the partial index in place, a prepared <code>status = $1</code> under a generic plan falls back to a Seq Scan, because the planner can't prove anything about a value it hasn't seen. Generic plans get their own part later in the series.</p>
<h2>The planner's statistics in pg_stats</h2>
<p>Before adding an index, look at the column in <code>pg_stats</code>:</p>
<pre><code class="language-sql">SELECT null_frac, n_distinct, most_common_vals, most_common_freqs
FROM pg_stats
WHERE schemaname = 'public' AND tablename = 'orders' AND attname = 'status';
</code></pre>
<pre><code class="language-text"> null_frac | n_distinct |         most_common_vals         |   most_common_freqs
-----------+------------+----------------------------------+------------------------
         0 |          4 | {paid,created,failed,chargeback} | {0.9,0.09,0.009,0.001}
</code></pre>
<p>That's what the planner believes about <code>status</code> as of the last <code>ANALYZE</code>. If most of your queries filter on <code>paid</code>, at 0.9, they get a Seq Scan, and no extra index will change that.</p>
<p>A negative <code>n_distinct</code> means a fraction of the row count. On tables bigger than the 30,000-row sample these numbers are estimates; you can raise the sample with <code>ALTER TABLE ... ALTER COLUMN ... SET STATISTICS</code>.</p>
<p>The same view has a <code>correlation</code> column, the planner's idea of physical order. On my 5-million-row tables it was 0.01 for the scattered copy and 1 for the clustered one, and that difference in physical order is what separates the scattered and clustered lines in both charts.</p>
<p>Before <code>CREATE INDEX</code>, run the <code>pg_stats</code> query, <code>relpages</code> and the rows-versus-pages count on your own table. If the value you care about sits on most of the pages and the table doesn't stay in memory, the index won't save you reads.</p>
<h2>Run it on your own server</h2>
<p>All the numbers above come from one 4-vCPU AMD EPYC cloud VM with 8 GB of RAM, so yours will differ. The SQL, the scripts, the raw output and the chart code are <a href="https://github.com/ImmuneFOMO/research/tree/main/postgres/slow-postgres/01-index-vs-seq-scan">on GitHub</a>. Run them on your own server, and if you get a different picture or think I got something wrong, open an issue or a PR.</p>
<p>My own projects run on netcup (VPS from €2.21 a month) and <a href="https://alphavps.com/clients/aff.php?aff=1157">AlphaVPS</a> (VPS from €2.54 a month). If you need a box for this, either works.</p>
<p>For netcup I have a few €5 vouchers for new customers. Each one works once, so take one and leave the rest for the next person:</p>
<pre><code class="language-text">36nc179418264998
36nc179418264997
36nc179418264996
36nc179418264995
36nc179418264994
36nc179418264993
</code></pre>
<p>Paste it into the "Redeem voucher" field at the bottom of the netcup cart. If they're all gone, message me on <a href="https://www.linkedin.com/in/anatolii-zhadan/">LinkedIn</a> and I'll send you a new one.</p>
<p>Good luck with your plans. Part 2 is next: how to read Postgres EXPLAIN ANALYZE.</p>
<h2>Finish</h2>
<p>I hope you understood the instructions correctly. If you have any edits to this article, I will be waiting for your comments. If I really helped you, I kindly ask you to subscribe to my LinkedIn and provide feedback on this article.</p>
<p>Link to follow me:</p>
<p><a href="https://www.linkedin.com/in/anatolii-zhadan/">LinkedIn</a></p>
<p>All my articles and links: <a href="https://fomo.moe">fomo.moe</a></p>
]]></content:encoded></item><item><title><![CDATA[How to Deploy a Self-Hosted 2-of-3 MPC Signing Cluster (Without Losing Everyone's Keys)]]></title><description><![CDATA[A restored MPC cluster can boot up, report [READY] on all three nodes, pass every health check — and have silently lost every private key]]></description><link>https://anatoliizhadan.hashnode.dev/how-to-deploy-a-self-hosted-2-of-3-mpc-signing-cluster-without-losing-everyone-s-keys</link><guid isPermaLink="true">https://anatoliizhadan.hashnode.dev/how-to-deploy-a-self-hosted-2-of-3-mpc-signing-cluster-without-losing-everyone-s-keys</guid><category><![CDATA[Web3]]></category><category><![CDATA[software development]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Cryptography]]></category><category><![CDATA[Smart Contracts]]></category><dc:creator><![CDATA[Anatolii Zhadan]]></dc:creator><pubDate>Tue, 14 Jul 2026 11:00:01 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a4c06e28a429c1defdba5ba/ad23395c-243b-4996-a998-ececc50f5650.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<img src="https://miro.medium.com/v2/1*NQ6YYzNEjCJk_Bpw_OKjFA.png" alt="A production 2-of-3 MPC signing cluster: three isolated nodes, a message bus, service discovery, and a backend that only ever sends a digest" style="display:block;margin:0 auto" />

<p><em>Three independent MPC nodes, each holding one key share. The backend never holds a key. It sends a 32-byte digest and gets a signature back.</em></p>
<p>A restored MPC cluster can boot up, report <code>[READY]</code> on all three nodes, pass every health check — and have silently lost every private key it was supposed to hold. Ours did exactly that, once, during a disaster-recovery drill. A signature check caught it, with about thirty minutes to spare. In a real emergency, on a bad night, we might not have.</p>
<p>That's the whole difference between a deploy guide and a production playbook, and it's why this one spends as much time restoring the cluster as standing it up.</p>
<p>This is the hands-on half of my previous article, <a href="https://medium.com/@Anatolii_Zhadan/how-to-store-your-users-crypto-wallets-so-nobody-can-steal-them-b0dd840211f6"><em>How to Store Your Users' Crypto Wallets So Nobody Can Steal Them</em></a>. There I argued <em>why</em> the private key should never exist as a whole. Here we build the thing: a self-hosted 2-of-3 MPC signing cluster you can run on your own infrastructure, initialize with a distributed key-generation ceremony, back up, and recover from a real disaster.</p>
<p>Everything below is from an actual deployment. The tools are open source, the numbers are measured, and the gotchas cost real hours. I've kept the architecture generic, a custodial Web3 backend that generates wallets and signs transactions for users, so you can map it onto your own stack.</p>
<p>If you've deployed distributed systems before and just want the landmines, jump to <strong>Section 5</strong> (the startup-order trap that can force you to rebuild your message bus) and <strong>Section 8</strong> (the silent restore bug from the opening). Everything else builds up to those two.</p>
<h2>Table of Contents</h2>
<ol>
<li><p>Architecture overview</p>
</li>
<li><p>The stack</p>
</li>
<li><p>Docker Compose — nine services</p>
</li>
<li><p>Node configuration</p>
</li>
<li><p>The initialization ceremony (DKG)</p>
</li>
<li><p>One master wallet, infinite users</p>
</li>
<li><p>The backup model</p>
</li>
<li><p>The disaster-recovery drill</p>
</li>
<li><p>Deploying across separate hosts</p>
</li>
<li><p>Production hardening</p>
</li>
<li><p>Scaling — splitting hosts and adding nodes</p>
</li>
<li><p>Troubleshooting cheat sheet</p>
</li>
<li><p>Build it so you don't need the luck</p>
</li>
</ol>
<hr />
<h2>Section 1: Architecture overview</h2>
<p>The whole point is one structural property: no single machine can produce a signature alone, and no machine ever holds the whole private key.</p>
<p>The shape that delivers it:</p>
<ul>
<li><p>Three MPC nodes, each on its own isolated host, each holding one key share</p>
</li>
<li><p>A 2-of-3 threshold: any two nodes can sign, one can be down or compromised without breaking anything</p>
</li>
<li><p>A message bus for the nodes to exchange protocol messages</p>
</li>
<li><p>A service-discovery / KV store for peer routing and wallet metadata</p>
</li>
<li><p>A backend that holds zero keys and only ever sends a 32-byte digest</p>
</li>
</ul>
<p>The trust boundary is the security model. The recommended topology puts the three nodes on three separate hosts over a private encrypted network, so they fail independently. A local Docker Compose file mirrors the same wiring for development — same services, same connections — and the local layout maps straight onto the multi-host production one in Section 9.</p>
<blockquote>
<p><strong>Why not one box with three containers?</strong> Because the isolation <em>is</em> the security. Three containers share one kernel and one root, so host compromise = all three shares = the master key reconstructed. Co-located nodes give you the security of one machine, not true 2-of-3.</p>
<p>There's an honest middle ground, because you won't always get three hosts on day one: run the nodes as separate users on a single memory-encrypted, attested VM, a Confidential VM (AMD SEV-SNP) or an enclave (AWS Nitro). Even host root can't read the nodes' memory there, and attestation proves the running image wasn't tampered with. It's a legitimate compensating control for a soft launch. Just be honest about what it is: it hardens the one box, it does not give you independent-failure 2-of-3. Treat multi-host as the target you graduate to.</p>
</blockquote>
<hr />
<h2>Section 2: The stack</h2>
<p>I didn't want to build an MPC server from scratch. The cryptography is the easy part now; the hard part is the server around it, networking, retries, encrypted storage, backups, health checks, a client SDK. That's easily a quarter of an engineering budget before you sign your first transaction.</p>
<p>So the stack sits on <a href="https://github.com/fystack/mpcium"><strong>Mpcium</strong></a> (fystack), the most turnkey open-source MPC server there is, Apache-2.0, written in Go. It hands you a full t-of-n cluster out of the box:</p>
<table>
<thead>
<tr>
<th>Component</th>
<th>What it is</th>
<th>Role here</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Mpcium</strong></td>
<td>Open-source MPC server (Go)</td>
<td>Runs the nodes, the signing/keygen lifecycle, the SDK</td>
</tr>
<tr>
<td><a href="https://github.com/fystack/tss-lib"><strong>tss-lib</strong></a> (<code>fystack/tss-lib v2.0.3</code>)</td>
<td>Threshold-signature library (GG18/GG20)</td>
<td>The cryptographic core, backing Mpcium</td>
</tr>
<tr>
<td><a href="https://github.com/dgraph-io/badger"><strong>BadgerDB</strong></a></td>
<td>Embedded KV store, AES-256 encrypted</td>
<td>Each node's local key-share storage</td>
</tr>
<tr>
<td><a href="https://docs.nats.io/nats-concepts/jetstream"><strong>NATS JetStream</strong></a></td>
<td>Message bus</td>
<td>Carries keygen/signing request + result events</td>
</tr>
<tr>
<td><a href="https://developer.hashicorp.com/consul"><strong>Consul KV</strong></a></td>
<td>Service discovery + KV</td>
<td>Peer routing and wallet metadata</td>
</tr>
<tr>
<td><strong>PostgreSQL</strong></td>
<td>Relational DB</td>
<td>User rows, including each user's derivation index</td>
</tr>
</tbody></table>
<p>Out of the box you also get ECDSA (secp256k1) + EdDSA (Ed25519), Ed25519 mutual auth on every message, ECDH-encrypted peer channels, incremental encrypted backups, and a TypeScript client (<a href="https://www.npmjs.com/package/@fystack/mpcium-ts"><code>@fystack/mpcium-ts</code></a>) that drops into a NestJS backend.</p>
<p>Two things it deliberately doesn't have, and how we deal with them:</p>
<ul>
<li><p><strong>No trusted-dealer key import.</strong> You can't import an existing private key. Section 6 makes that a non-issue by deriving all user wallets from one generated master instead of importing anything.</p>
</li>
<li><p><strong>No REST/gRPC API</strong>, it's NATS pub/sub + SDK only. Fine for a backend already on a message bus; wrap it if you need HTTP.</p>
</li>
</ul>
<p>One thing worth doing before you trust any of this. Mpcium runs GG18/GG20, the family with the <a href="https://www.fireblocks.com/blog/gg18-and-gg20-paillier-key-vulnerability-technical-report"><em>BitForge</em></a> vulnerability class from the last article. It's acceptable here for two reasons. BitForge needs a malicious <em>signing party</em>, which is irrelevant when you run all three nodes. And more importantly, I checked the patch is actually in the code path. The tss-lib v2 line carries the two hardening proofs, Paillier-Blum in <code>crypto/modproof/</code> and no-small-factor in <code>crypto/facproof/</code>, generated in GG18 keygen round 2 and verified in round 3. Mpcium pins the <code>fystack/tss-lib v2.0.3</code> fork through a <code>go.mod</code> replace, so I grepped its whole tree for the flags that would disable those proofs (<code>SetNoProofMod</code>, <code>SetNoProofFac</code>) and found them called nowhere in the keygen path, only in the library's own tests. That's the level of due diligence a signing stack earns: don't trust the protocol name, read the code path.</p>
<p>(Starting fresh and want the newest protocol? <a href="https://github.com/LFDT-Lockness/cggmp21"><code>cggmp24</code></a> (the <code>cggmp21</code> crate, which now implements CGGMP24 — patched, <code>v0.7.0-alpha.2+</code>) and Coinbase's Cure53-audited <a href="https://github.com/coinbase/cb-mpc"><code>cb-mpc</code></a> are the forward-looking cores. But they're libraries, not servers, so you're back to building the server yourself.)</p>
<hr />
<h2>Section 3: Docker Compose — nine services</h2>
<p>The full stack is nine services, eight long-running plus a one-shot bootstrap. Here's how they fit, and the two traps that will bite you.</p>
<table>
<thead>
<tr>
<th>Service</th>
<th>Image / build</th>
<th>Role</th>
</tr>
</thead>
<tbody><tr>
<td><code>app</code></td>
<td>your backend (NestJS)</td>
<td>Custodial backend + MPC client. Starts last.</td>
</tr>
<tr>
<td><code>postgres</code></td>
<td><code>postgres:15-alpine</code></td>
<td>User rows, including each user's derivation index</td>
</tr>
<tr>
<td><code>redis</code></td>
<td><code>redis:7-alpine</code></td>
<td>Cache (not part of MPC state)</td>
</tr>
<tr>
<td><code>nats</code></td>
<td><code>nats:2.10-alpine</code> (<code>-js</code>)</td>
<td>Message bus (JetStream enabled)</td>
</tr>
<tr>
<td><code>consul</code></td>
<td><code>hashicorp/consul:1.15</code></td>
<td>Service discovery + routing KV</td>
</tr>
<tr>
<td><code>mpcium-bootstrap</code></td>
<td><code>hashicorp/consul:1.15</code></td>
<td>One-shot: seeds peer IDs into Consul, then exits</td>
</tr>
<tr>
<td><code>mpcium-node0/1/2</code></td>
<td>your Mpcium node image</td>
<td>The three MPC nodes</td>
</tr>
</tbody></table>
<img src="https://miro.medium.com/v2/1*IXWoFvU8T0IEyeQQoJ2uxQ.png" alt="The Compose dependency graph: bootstrap seeds the peers, the three nodes come up, and the backend starts last" style="display:block;margin:0 auto" />

<p><code>depends_on</code> <em>enforces the order: Consul healthy → the one-shot bootstrap completes → nodes start → the backend connects last.</em></p>
<h3>The bootstrap pattern</h3>
<p>The nodes need to know about each other before they start. Rather than bake that in, a one-shot <code>mpcium-bootstrap</code> container writes the three peer UUIDs into Consul and exits <code>0</code>. Every node then depends on it with <code>service_completed_successfully</code>:</p>
<pre><code class="language-yaml">services:
  mpcium-bootstrap:                 # one-shot: seed peer UUIDs into Consul, then exit 0
    image: hashicorp/consul:1.15
    environment: [CONSUL_HTTP_ADDR=consul:8500]
    command:
      - sh
      - -c
      - |
        consul kv put mpc_peers/node0 &lt;node0-uuid&gt; &amp;&amp;
        consul kv put mpc_peers/node1 &lt;node1-uuid&gt; &amp;&amp;
        consul kv put mpc_peers/node2 &lt;node2-uuid&gt;
    depends_on: { consul: { condition: service_healthy } }
    restart: "no"

  mpcium-node0:
    build: ./mpc/node                # distroless image, runs as UID 65532
    command: ["start", "-n", "node0"]
    working_dir: /cluster/node0      # mpcium reads identity/ + peers.json CWD-relative
    volumes:
      - ./mpc/node0:/cluster/node0                              # host dir: chown 65532:65532
      - ./mpc/config/node0.yaml:/cluster/node0/config.yaml:ro
    depends_on:
      mpcium-bootstrap: { condition: service_completed_successfully }
      nats:   { condition: service_started }
      consul: { condition: service_healthy }
    restart: unless-stopped
  # node1, node2 are identical. The `app` service depends on all three and starts LAST.
</code></pre>
<p>A few load-bearing lines in there:</p>
<ul>
<li><p><code>restart: "no"</code> on <code>mpcium-bootstrap</code> — it's a one-shot: seed the peer IDs, exit <code>0</code>, done. Restart it and it re-seeds on every boot.</p>
</li>
<li><p><code>condition: service_completed_successfully</code> — the nodes block until that clean exit, so peers always exist before a node reads them.</p>
</li>
<li><p>the <code>:ro</code> config mount — each node's <code>config.yaml</code> is read-only; nothing at runtime should mutate it.</p>
</li>
<li><p><code>working_dir</code> and the volume's owner are both load-bearing enough to be their own traps, below.</p>
</li>
</ul>
<p>(As of v0.3.5 there's a lighter option: pass <code>--peers=peers.json</code> to each node and it syncs any new peers into Consul on startup, skipping ones already there — so you can drop the <code>mpcium-bootstrap</code> container entirely, or run <code>mpcium-cli register-peers --peers peers.json</code> once. The manual <code>consul kv put mpc_peers/…</code> still works; it's just no longer required.)</p>
<h3>Trap #1 — the distroless UID</h3>
<p>The Mpcium node image is <a href="https://github.com/GoogleContainerTools/distroless"><code>FROM gcr.io/distroless/base-debian12</code></a> with <code>USER nonroot:nonroot</code>, which is UID/GID 65532. Your host-mounted node dirs are owned by your user, so the moment a node tries to write its BadgerDB it hits permission-denied and crash-loops. One line fixes it, and you'll forget it exactly once:</p>
<pre><code class="language-bash">chown -R 65532:65532 ./mpc/node0 ./mpc/node1 ./mpc/node2
</code></pre>
<h3>Trap #2 — <code>working_dir</code> is load-bearing</h3>
<p>Mpcium resolves <code>identity/</code> and <code>peers.json</code> relative to the current working directory, not to a config path. Miss <code>working_dir</code> (or <code>WorkingDirectory</code> in a systemd unit) and the node starts in <code>/</code> and can't find its identity files. Pin it.</p>
<hr />
<h2>Section 4: Node configuration</h2>
<p>Each node gets its own <code>config.yaml</code>. Most of it is boring. Two fields will bite you.</p>
<pre><code class="language-yaml">nats:   { url: "nats://nats:4222" }
consul: { address: "consul:8500" }

mpc_threshold: 1                   # tss-lib degree t -&gt; t+1 = 2 signers (this is 2-of-3)
environment: "production"

badger_password: "&lt;32 hex chars = 32-byte AES-256 key, DIFFERENT per node&gt;"
event_initiator_algorithm: "ed25519"
event_initiator_pubkey:  "&lt;64 hex, SAME on all nodes + backend&gt;"
chain_code:              "&lt;64 hex, SAME on all nodes + backend&gt;"

db_path: "./data"                  # mpcium appends /nodeN — remember this in Section 8
backup_enabled: true
backup_period_seconds: 300
max_concurrent_keygen: 2
max_concurrent_signing: 10
</code></pre>
<p>What the important fields do:</p>
<ul>
<li><p><code>mpc_threshold</code> — the availability/security knob; <code>1</code> means 2-of-3 (see the gotcha below).</p>
</li>
<li><p><code>db_path</code> — where the shards live; Mpcium appends <code>/nodeN</code> to it, which will matter enormously at restore time (Section 8).</p>
</li>
<li><p><code>backup_enabled</code> / <code>backup_period_seconds</code> — Mpcium's own incremental encrypted backup, every 300 s.</p>
</li>
<li><p><code>max_concurrent_keygen</code> / <code>max_concurrent_signing</code> — per-node concurrency caps.</p>
</li>
<li><p>the three identity fields (<code>badger_password</code>, <code>event_initiator_pubkey</code>, <code>chain_code</code>) have a shared-vs-per-node rule of their own, right below.</p>
</li>
</ul>
<h3>The threshold gotcha (read this twice)</h3>
<blockquote>
<p>In tss-lib, <code>mpc_threshold</code> is the polynomial degree <code>t</code>, and the number of signers required is <code>t + 1</code>. So a 2-of-3 cluster uses <code>mpc_threshold: 1</code>, not <code>2</code>.</p>
</blockquote>
<p>And here's the part that makes it dangerous: the Mpcium dev <code>config.yaml.template</code> ships with <code>mpc_threshold: 2</code> by default (the <code>config.prod.yaml.template</code> correctly ships <code>1</code> — but the dev one is what most people copy first). Copy it unthinkingly and you've built a 3-of-3 cluster, which throws away the entire availability benefit of MPC, because now <em>every</em> node must be online to sign. Set it to <code>1</code>.</p>
<p>One more thing: tss-lib only enforces that the degree is at least 1 and no larger than the node count minus one. For three nodes that leaves 2-of-3 (<code>mpc_threshold: 1</code>) or 3-of-3 (<code>mpc_threshold: 2</code>) — a 1-of-3 where a single node could sign alone is degenerate and rejected. (GG18 tolerates a dishonest majority, so the code allows any <code>1 ≤ t ≤ n−1</code>; Mpcium's README still <em>recommends</em> an honest-majority <code>t ≥ ⌊n/2⌋+1</code>, and for three nodes there's no sensible quorum below 2-of-3 anyway.)</p>
<h3>Shared vs. per-node</h3>
<p>This catches people, so let me be blunt:</p>
<ul>
<li><p><strong>Different on every node:</strong> <code>badger_password</code> (each node encrypts its own BadgerDB with its own key).</p>
</li>
<li><p><strong>Identical on every node and the backend:</strong> <code>chain_code</code>, <code>event_initiator_pubkey</code>.</p>
</li>
</ul>
<p><code>event_initiator_pubkey</code> pins who's allowed to ask the cluster to sign, the Ed25519 public key of your backend's initiator identity. <code>chain_code</code> is what makes deterministic derivation work (Section 6).</p>
<h3>Identities and peers</h3>
<p>Each node dir has an <code>identity/</code> folder with its own private key (<code>node0_private.key</code>, mode <code>0600</code>) plus every node's public identity (<code>nodeX_identity.json</code>, mode <code>0644</code>). And every node carries the same <code>peers.json</code>, whose UUIDs must exactly match what <code>mpcium-bootstrap</code> seeded into Consul:</p>
<pre><code class="language-json">{
  "node0": "00000000-0000-0000-0000-000000000000",
  "node1": "11111111-1111-1111-1111-111111111111",
  "node2": "22222222-2222-2222-2222-222222222222"
}
</code></pre>
<hr />
<h2>Section 5: The initialization ceremony (DKG)</h2>
<p>This is the ceremony that generates the master key without ever assembling it. It runs once, on a fresh cluster, with the backend stopped.</p>
<h3>The one rule that governs everything: startup order</h3>
<p>The single most important operational rule in this whole deployment:</p>
<blockquote>
<p>The MPC nodes must create the JetStream streams before anything else connects to NATS.</p>
</blockquote>
<p>Here's why. The Go nodes create their <code>mpc-keygen</code> and <code>mpc-signing</code> streams with JetStream's <strong>default (limits)</strong> retention, but the TypeScript client (<code>@fystack/mpcium-ts</code>) creates those same streams with <strong>workqueue</strong> retention (and the <code>mpc</code> result stream the other way round — <code>interest</code> on the client, <code>workqueue</code> on the nodes). If the client wins the race and creates either one first, the nodes come up, refuse to reconcile the retention, and immediately crash-loop with:</p>
<pre><code class="language-plaintext">err_code=10052 ... can not change retention policy to/from workqueue
</code></pre>
<p>And JetStream won't let you add or remove <code>workqueue</code> retention on an existing stream, so no config change or API call fixes it in place. The simplest remedy is to recreate the NATS container (or drop just the two offending streams and let the nodes recreate them), either way rebuilding JetStream state you never planned to touch. So the sequence is non-negotiable:</p>
<img src="https://miro.medium.com/v2/1*leG081h4waWG7wSDS6cNyg.png" alt="Bring up infrastructure and the three nodes first; wait for the streams; only then connect the backend" style="display:block;margin:0 auto" />

<p><em>Get this order wrong and JetStream's fixed workqueue retention forces you to recreate NATS from scratch.</em></p>
<pre><code class="language-bash"># 1. Bring up infra + the three nodes (NOT the app yet)
docker compose up -d postgres redis nats consul \
  mpcium-bootstrap mpcium-node0 mpcium-node1 mpcium-node2

# 2. Wait until each node logs that it owns its streams, e.g.:
#    "SigningConsumer: Subscribed to signing events"
#    "Subscription created successfully"
docker compose logs -f mpcium-node0   # watch for the stream-subscription logs above

# 3. ONLY NOW run the init ceremony / start the app
</code></pre>
<p>There's a sneakier version of this trap too: a running backend steals the DKG result. If the app is up during init, its consumer grabs the keygen-result event off the queue before the init script can read it, and init hangs forever on "waiting for keygen result." Stop the app before you initialize.</p>
<h3>What the init script actually does</h3>
<p>The init is a one-time bootstrap script. Don't underestimate it, ours is 1,160 lines, and that's normal. A one-time bootstrap for a distributed system is never a ten-liner. Four phases:</p>
<ol>
<li><p><strong>Validate</strong> — the compose file, every node config, the initiator key, the backend <code>.env</code>, and that all three nodes plus NATS and Consul are actually running.</p>
</li>
<li><p><strong>DKG the master wallet</strong> — connect to NATS, create the Mpcium client, call <code>createWallet(masterWalletId)</code>, wait for the keygen-result event (60-second timeout). Take the ECDSA public key, compressed to secp256k1.</p>
</li>
<li><p><strong>Write env + back up</strong> — persist <code>MPC_MASTER_WALLET_ID</code>, <code>MPC_MASTER_PUBLIC_KEY</code>, and <code>MPC_CHAIN_CODE</code> to the backend's <code>.env</code>; then copy each node's data dir, export the Consul KV, write a <code>metadata.json</code> (<code>nodeCount: 3</code>, <code>threshold: "2-of-3"</code>). Move this backup offline immediately, it's hot-key-equivalent material.</p>
</li>
<li><p><strong>Verify</strong> — derive addresses at indexes 0 and 1, sign a test 32-byte digest at each (30-second timeout, up to four attempts on transient errors), print <code>=== INITIALIZATION COMPLETE ===</code>.</p>
</li>
</ol>
<h3>Timeouts and the slow part of boot</h3>
<p>Keygen needs all three nodes at once, so give it the most generous timeout (we use 60 s for keygen, 30 s for signing) — a transient hiccup shouldn't abort a ceremony. The one real cost at boot is Paillier preparameter generation, the bulk of a cold start. Given enough cores the three nodes generate their preparams in parallel; on a smaller host they contend for CPU, so start them serially to avoid starving each other. Either way you pay it once: the preparams are persisted to BadgerDB and reused on every restart, so restarts are cheap.</p>
<hr />
<h2>Section 6: One master wallet, infinite users</h2>
<p>Give every user their own MPC wallet the obvious way, and every signup runs its own DKG across all three nodes — and your backups grow without bound: ten thousand users is ten thousand ceremonies and ten thousand more sets of shards you can never lose.</p>
<p>We don't do that. We run DKG exactly once, for a single master wallet, and derive every user wallet from it with BIP-32 child key derivation (CKD):</p>
<ul>
<li><p>Each user gets an integer <code>wallet_index</code>, stored as a column in Postgres</p>
</li>
<li><p>Their address is derived at path <code>m/44/60/0/0/{wallet_index}</code> (all levels non-hardened — hardened derivation needs the private key, which never exists here) — pure math, zero MPC nodes required online</p>
</li>
<li><p>To sign, each node applies its own derivation delta to its master share; the child key, like the master, is never assembled</p>
</li>
</ul>
<p>This is what makes the operational model sane, and it's the reason the backup story in the next section is so simple. One derivation path supports about 2.1 billion users (indexes 0 through 2³¹ − 1), so you'll run out of users long before you run out of key space. I covered CKD's security equivalence in the previous article; the short version is that it's <em>identical</em> to per-user DKG under every node-compromise scenario.</p>
<hr />
<h2>Section 7: The backup model — back up three things, not one</h2>
<p>The most common way to lose an MPC cluster is to back up the key shares, feel safe, and find out mid-restore that shares alone are useless. You have to back up three things:</p>
<img src="https://miro.medium.com/v2/1*sdcVgz0sSs4ZrTD_VERaBQ.png" alt="Three things to back up: the encrypted shares, the routing metadata, and the derivation metadata" style="display:block;margin:0 auto" />

<p><em>Restore only BadgerDB and the cluster boots but can't find any wallet — signing fails with</em> <code>ERROR_KEY_NOT_FOUND</code><em>.</em></p>
<table>
<thead>
<tr>
<th>#</th>
<th>Item</th>
<th>Where</th>
<th>Why it's mandatory</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>BadgerDB shards</td>
<td>each node's <code>data/</code> dir</td>
<td>the encrypted key-share state</td>
</tr>
<tr>
<td>2</td>
<td>Consul KV (<code>threshold_keyinfo/*</code>, <code>mpc_peers/*</code>)</td>
<td>Consul export</td>
<td>wallet routing; without it every sign fails <code>ERROR_KEY_NOT_FOUND</code></td>
</tr>
<tr>
<td>3</td>
<td>Metadata (<code>chain_code</code>, master wallet ID, master pubkey, node configs, identity keys)</td>
<td>configs + backend <code>.env</code></td>
<td>needed to derive addresses and boot the nodes</td>
</tr>
</tbody></table>
<h3>One backup, forever</h3>
<p>Here's the quiet superpower of CKD. The master shares, the <code>threshold_keyinfo/*</code> routing entries, and the peer metadata never change after the initial DKG. New users are just new rows in Postgres. So your MPC backup strategy is: take one full MPC backup right after init, then back up Postgres normally from then on. That's it. No incremental shard sync, no window where a user created between backups is unrecoverable. Per-user DKG would need a fresh MPC backup after <em>every</em> signup.</p>
<p>And this is the whole disaster-recovery story in one sentence: lose the entire cluster and you still lose nothing. Stand up fresh nodes, restore that single encrypted master backup, and every user wallet comes back, because no user wallet was ever stored on its own. Each one is a deterministic derivation of the master, and its <code>wallet_index</code> lives safely in your Postgres rows. The master backup plus your normal Postgres backups reconstruct every address and every signing key by index, on brand-new hardware, with nothing lost.</p>
<h3>The copy-size trap</h3>
<p>How you copy BadgerDB matters more than you'd think. A <em>running</em> node's BadgerDB is ~2.2 GB — SSTables, value logs, and the WAL, mostly churn. What you actually ship is Mpcium's own cold backup: with <code>backup_enabled: true</code>, each node writes a pruned, encrypted export to <code>backup_dir</code> (the <code>backup-*.enc</code> files), a few tens of KB apiece. In the drill the full <code>age</code>-wrapped bundle — all three nodes' exports plus every config, identity, and the Consul KV dump — came to <strong>164 KB</strong>, four orders of magnitude below the live directory. Two consequences: never copy a live BadgerDB (and never stage a hot copy in a small <code>tmpfs</code> — it can exceed 6 GB and blow out a ~2 GB <code>/tmp</code>); ship the <code>backup_dir</code> export instead.</p>
<h3>Restore is a specific sequence</h3>
<pre><code class="language-bash"># 1. Start fresh nodes, then STOP them
docker compose up -d &amp;&amp; docker compose stop mpcium-node0 mpcium-node1 mpcium-node2
# 2. For each node, decrypt + restore its export INTO the exact db_path/nodeN dir:
#    mpcium-cli recover --backup-dir &lt;node's backup-*.enc&gt; \
#      --recovery-path &lt;db_path&gt;/nodeN --force   # the nodeN subdir is load-bearing — Section 8
#    (recover prompts for the badger_password; drive it with `expect` in automation)
# 3. Import the Consul KV dump  (skip this and signing fails ERROR_KEY_NOT_FOUND)
consul kv import @backup/consul-kv.json
# 4. Fix ownership, delete stale BadgerDB LOCK files
chown -R 65532:65532 ./mpc &amp;&amp; find ./mpc -name LOCK -delete
# 5. Start the nodes, wait for all three to log [READY]
docker compose start mpcium-node0 mpcium-node1 mpcium-node2
# 6. Verify with a signing smoke test — do NOT trust "[READY]" alone (Section 8)
</code></pre>
<hr />
<h2>Section 8: The disaster-recovery drill</h2>
<p>You don't have a backup until you've restored it. So before go-live we ran the full thing: destroy the cluster, rebuild it from backups on fresh hardware, and prove the recovered cluster signs with the same master key. The pass criterion was strict, recover a signature and confirm via <code>ecrecover</code> that its master public key was byte-for-byte identical to the original.</p>
<p>It passed. It also surfaced eleven runbook and config bugs in a single session. A clean re-run took 45–60 minutes; the first pass took about two hours, precisely because of what it caught. Three of them are worth the price of this whole article.</p>
<h3>The silent empty-database restore (the one from the opening)</h3>
<p>This is the bug that would have quietly destroyed every user wallet in a real emergency.</p>
<p>Mpcium reads BadgerDB from <code>${db_path}/nodeN/</code>, it auto-appends the <code>nodeN</code> subdirectory. Drop your restore into <code>.../data</code> instead of <code>.../data/nodeN</code> and the files land one level too high. Mpcium then looks in <code>.../data/nodeN/</code>, finds it empty, and, this is the dangerous part, silently creates a brand-new, empty database.</p>
<p>The cluster boots. All three nodes report <code>[READY]</code>. Health checks are green. But the shares are gone — every existing wallet is orphaned, and the first real signing request fails with <code>ERROR_KEY_NOT_FOUND</code> (newer v0.3.5 builds report the same thing as <code>ERROR_NOT_MAJORITY</code>, <em>"not enough peers to process signing request"</em>). Worse, the empty cluster will happily run a fresh DKG and mint a brand-new master key that has nothing to do with your users' funds. You find out on that first signature, long after the green health check lied to you.</p>
<img src="https://miro.medium.com/v2/1*OsP1c327I_5KQJfbtsZypA.png" alt="A restore into the wrong directory boots a healthy-looking cluster that has silently lost every key" style="display:block;margin:0 auto" />

<p><em>"[READY]" is not "recovered." Restore into the exact</em> <code>db_path/nodeN</code> <em>subdirectory, then verify by signing, never by status alone.</em></p>
<p>The drill burned about 30 minutes on exactly this. The fix is trivial once you know it, restore into the explicit <code>.../data/nodeN</code> subdir and <code>chown</code> afterward, but you'd never guess it from a green health check. The lesson generalizes: <code>[READY]</code> means "a database is present," not "the <em>right</em> database is present." Verify recovery by signing and checking the recovered public key, never by status.</p>
<h3>The documented command that didn't exist</h3>
<p>The first-draft runbook confidently referenced an <code>mpcium-cli init-dkg</code> command that doesn't exist in the shipped binary, with a flag set to match. There is no keygen subcommand at all: DKG is triggered by publishing a signed keygen-request message onto NATS (subject <code>mpc.keygen_request.*</code>). Recovery really uses <code>--backup-dir</code>, <code>--recovery-path</code>, and <code>--force</code>, and the binary prompts for its password interactively via <code>term.ReadPassword</code>, so you drive it with <code>expect</code> in automation. Dry-run every command in your runbook against the actual binary before you trust it in an emergency.</p>
<h3><code>openssl rand</code> will hand you an invalid key</h3>
<p>BadgerDB's encryption key must be exactly 16, 24, or 32 bytes. The instinctive command produces neither:</p>
<pre><code class="language-bash">openssl rand -base64 32     # WRONG: 44 ASCII chars -&gt; node fatals opening the encrypted BadgerDB:
                            # the key must be exactly 16, 24, or 32 bytes.
openssl rand -hex 16        # RIGHT: 32 hex characters = 32 bytes = AES-256
</code></pre>
<p><code>-base64 32</code> turns 32 random <em>bytes</em> into a 44-<em>character</em> string, and the password is used as raw key bytes — so that's a 44-byte key, not a valid AES length. <code>-hex 16</code> gives 32 characters = 32 bytes: an AES-256-length key carrying 128 bits of entropy, ample for a local DB key.</p>
<h3>Five more the drill taught us</h3>
<ul>
<li><p><strong>Never regenerate</strong> <code>badger_password</code> <strong>after first start.</strong> The KEYREGISTRY is encrypted with the <em>original</em> password; change it and the database is permanently unreadable.</p>
</li>
<li><p><strong>A wrong or regenerated</strong> <code>chain_code</code> <strong>silently orphans every wallet.</strong> The cluster boots and signs perfectly happily, but it now derives a completely different address space, so none of your users' existing addresses come back. Like the empty-DB bug, it sails past <code>[READY]</code> and only a sign-and-compare catches it. Treat <code>chain_code</code> as recovery-critical metadata, not a rotatable secret.</p>
</li>
<li><p><strong>Regenerating node identity keys or peer UUIDs is a re-key, not a recovery.</strong> Fresh identities make the restored shares reject each other, forcing a brand-new DKG, which produces a <em>new</em> master public key and abandons every address derived from the old one. Same limit if a node is permanently lost: you can't just drop in a replacement, because a reshare needs the surviving shares as input. Mpcium ships that as <code>Resharing()</code> (Section 11) — it rebuilds the replacement from the survivors while keeping the master key. A recovered 2-of-3 cluster missing its third node has zero remaining fault tolerance until you re-share, so plan node replacement rather than improvising it.</p>
</li>
<li><p><strong>Consul KV doesn't contain the master public key.</strong> It stores only <code>{participant_peer_ids, threshold, version}</code>. So you can't verify a restore by reading KV, you sign and <code>ecrecover</code>.</p>
</li>
<li><p><strong>Your cold bundle must include the supporting config</strong> (<code>nats.conf</code>, CA templates), or the restored NATS/services crash-loop and you never reach the signing test.</p>
</li>
</ul>
<hr />
<h2>Section 9: Deploying across separate hosts</h2>
<p>Everything so far ran from one Docker Compose file — one host, one kernel, one root. That's the right way to <em>learn</em> the stack, and fine while you're testing, but it isn't 2-of-3: own the box and you own all three shares. Production means each node on its own machine, so a single host compromise costs the attacker exactly one share. None of the cryptography changes — same three nodes, same DKG, same backups. What changes is the wiring: instead of a Compose network, the nodes live on separate hosts joined by a private encrypted overlay, and they reach the bus over that overlay instead of a Docker bridge.</p>
<h3>The shape</h3>
<ul>
<li><p><strong>Three node hosts</strong> — one Mpcium node each. Ideally memory-encrypted, attested VMs (AMD SEV-SNP, AWS Nitro), each with <strong>no public IP</strong> and a scoped machine identity. Run the node as its own locked-down user with a <code>0700</code> data dir, so a compromised node process still can't read another node's BadgerDB.</p>
</li>
<li><p><strong>One control host</strong> — NATS + Consul (and a small internal CA, below). The bus and the KV store live here; every node and the backend point at it.</p>
</li>
<li><p><strong>The backend</strong> — sends digests, holds zero keys, also on the overlay.</p>
</li>
</ul>
<p>The nodes never talk to each other directly — all protocol traffic goes through NATS. So the only reachability requirement is: every node and the backend can reach the control host's NATS and Consul over the private network. That keeps the network model a simple star, not a full mesh.</p>
<h3>The private overlay (WireGuard)</h3>
<p>Put every host on one <a href="https://www.wireguard.com/">WireGuard</a> network so nothing MPC-related ever touches a public interface. On each node host, one peer pointing at the control host:</p>
<pre><code class="language-plaintext"># /etc/wireguard/wg0.conf  (a node host)
[Interface]
Address    = 10.10.0.11/24
PrivateKey = &lt;this node's private key&gt;
ListenPort = 51820
MTU        = 1380          # provider MTU minus WireGuard's 80-byte header

[Peer]                     # control host (NATS + Consul)
PublicKey           = &lt;control host public key&gt;
AllowedIPs          = 10.10.0.1/32          # just the control host — nodes reach each other via NATS, not directly
Endpoint            = &lt;control host public ip&gt;:51820
PersistentKeepalive = 25
</code></pre>
<p>The control host runs the mirror image — one <code>[Peer]</code> block per node, each with that node's <code>/32</code> — and forwards nothing else. Only public keys cross the wire; private keys never leave their host. Then lock the provider firewall so UDP/51820 is the <em>only</em> inbound port and only from the overlay's peers, deny the rest, and bind SSH to the WG address instead of the public one. On a cloud MPC VM that means launching it with <strong>no external IP at all</strong> and reaching it purely over WG.</p>
<h3>Per-node config</h3>
<p>Each node's <code>config.yaml</code> is identical to the single-host one, except the addresses now point at the control host over WG and the shared secrets stay shared:</p>
<pre><code class="language-yaml">nats:   { url: "nats://10.10.0.1:4222" }   # control host, over WG
consul: { address: "10.10.0.1:8500" }
mpc_threshold: 1                            # still 2-of-3
badger_password: "&lt;per-node, 32 hex&gt;"       # distinct per node
chain_code:             "&lt;same on all nodes + backend&gt;"
event_initiator_pubkey: "&lt;same on all nodes + backend&gt;"
db_path: "/var/lib/mpcium/data"
</code></pre>
<p>Lock the bus itself down with <strong>mTLS</strong>: issue a client certificate per node from a small internal CA (<a href="https://smallstep.com/docs/step-ca/">step-ca</a> is plenty), so only your nodes and backend can speak to NATS — a stray connection that lands on the overlay still can't.</p>
<h3>Bringing it up</h3>
<p>The startup order from Section 5 still holds, just spread across hosts: bring up NATS + Consul on the control host, <code>register-peers</code> (or start the nodes with <code>--peers</code>) so the UUIDs land in Consul, start the three nodes on their own hosts (streams first — the retention trap doesn't care that the nodes are remote now), wait for <code>[READY]</code> on all three, then run the DKG exactly as before. The backend connects last.</p>
<p>This is the topology every backup, restore, and reshare in this article already assumes — the wiring is all that stood between you and it.</p>
<hr />
<h2>Section 10: Production hardening</h2>
<p>Once the cluster works, the remaining work is isolation and discipline.</p>
<ul>
<li><p><strong>Split the nodes across hosts</strong> — the from-scratch recipe is Section 9, migrating an existing single-box cluster is Section 11. Three separate machines mean a single host compromise yields at most one share; anything less is theater. Can't split yet? Fall back to a memory-encrypted attested VM (<a href="https://www.amd.com/en/developer/sev.html">AMD SEV-SNP</a>, <a href="https://aws.amazon.com/ec2/nitro/nitro-enclaves/">AWS Nitro</a>) — the Section 1 note covers exactly what that buys you and what it doesn't. One caveat that lands here: SEV-SNP VMs generally can't live-migrate, so an abrupt host termination becomes your disaster-recovery trigger — exactly why the drill in Section 8 isn't optional.</p>
</li>
<li><p><strong>Private encrypted network.</strong> Put inter-node and node-to-backend traffic on a private mesh (<a href="https://www.wireguard.com/">WireGuard</a>). The message bus and KV store should never be exposed publicly.</p>
</li>
<li><p><strong>Egress firewall.</strong> MPC nodes have no business making arbitrary outbound connections. Lock egress to what the protocol needs.</p>
</li>
<li><p><strong>Encrypt every backup with</strong> <a href="https://github.com/FiloSottile/age"><code>age</code></a> before it leaves the host, and store it on physically separate, offline media. A stolen backup is all shares plus metadata, equivalent to stealing the funds.</p>
</li>
<li><p><strong>Plan for the memory profile.</strong> Each node is lightweight at rest, on the order of tens of MB idle, but spikes during Paillier preparameter generation, so provision for the spike, not the idle state, and remember the serial-start caveat on smaller hosts. For reference, the entire dev stack (backend, three nodes, Postgres, Consul, NATS, Redis) fits comfortably in 3.8 GiB.</p>
</li>
<li><p><strong>Make init lazy, not fatal.</strong> The backend shouldn't crash if MPC is briefly unavailable at boot; log a warning and retry on first real use. And handle the result-before-request race: register the pending request <em>before</em> you check for an early result, because a fast cluster can return a signature before your handler is registered.</p>
</li>
<li><p><strong>Retry the transient errors.</strong> Treat these as retryable with a short backoff (<code>[0, 500, 2000] ms</code>): <code>symmetrickey key not found</code>, <code>failed to decrypt message</code>, <code>encrypt tss message error</code>, <code>not enough peers to process signing request</code>, <code>timed out waiting for signing result</code>. They're normal under load, not failures.</p>
</li>
</ul>
<hr />
<h2>Section 11: Scaling the cluster — splitting hosts, and adding nodes</h2>
<p>Two operations get conflated here, and they cost wildly different amounts. Moving your three nodes onto separate hosts keeps the same key and costs only a maintenance window. <em>Adding</em> a node changes the committee and needs a resharing ceremony — which Mpcium does ship, just not as a one-command move. Know which one you're doing before you touch production.</p>
<h3>Splitting one host into three (same key, just downtime)</h3>
<p>This is the common "started on one box, now you want real 2-of-3" move, and it is <em>not</em> a re-key. The identities, the shares, the <code>chain_code</code>, and the peer UUIDs all stay byte-for-byte the same — you're only changing where each node runs. Because none of the key material changes, the master public key and every derived address survive untouched. Mechanically it's the Section 8 restore, aimed at three hosts instead of one:</p>
<ol>
<li><p>Stand up the private network first — a <a href="https://www.wireguard.com/">WireGuard</a> mesh across the three hosts and the backend. Nodes talk through NATS, not node-to-node, so what actually has to be reachable is one NATS and one Consul over that private network.</p>
</li>
<li><p>Stop the cluster. Your downtime starts here, and it's a hard stop: nodes down, no signing.</p>
</li>
<li><p>Move each node's data dir, its <code>identity/</code> folder, and its <code>config.yaml</code> onto its own host. Keep the peer UUIDs and identity keys identical — regenerating them is a re-key, not a move (Section 8), and it abandons every address.</p>
</li>
<li><p>Point each node's config at the relocated <code>nats.url</code> and <code>consul.address</code>, re-import the Consul KV dump, <code>chown -R 65532:65532</code>, and delete stale <code>LOCK</code> files.</p>
</li>
<li><p>Start the nodes, wait for three <code>[READY]</code>, then <em>verify by signing</em> and <code>ecrecover</code> that the master public key is byte-identical to before. Never trust <code>[READY]</code> alone — that's the whole lesson of Section 8.</p>
</li>
</ol>
<p>Downtime is the stop-to-verified-restart window — minutes if you've rehearsed it. Rehearse it exactly like the DR drill, because mechanically it is one.</p>
<h3>Adding a node (2-of-3 → 2-of-4, or 3-of-5): resharing</h3>
<p>Reshaping the committee — a fourth node, a higher threshold — can't be a plain <code>docker run</code>: a new node has no share, and the existing shares were generated for a three-party polynomial. The primitive that fixes this is <strong>key resharing</strong> — the current shareholders jointly re-distribute the secret to a new set of parties, producing fresh shares for the new committee <em>while keeping the same master public key</em>, so every user address is preserved and nothing moves on-chain.</p>
<p>Mpcium ships resharing as a first-class, signed operation (it landed in v0.3.0 and was hardened for new-node joins in v0.3.5). You publish a <code>ResharingMessage</code> and wait for the result:</p>
<pre><code class="language-js">mpcClient.Resharing({
  sessionID, walletID, keyType,
  nodeIDs:      newCommitteePeerIDs,  // include a brand-new peer ID to add a node
  newThreshold,                       // change t here
});
mpcClient.OnResharingResult((r) =&gt; { /* new shares are written */ });
</code></pre>
<p>Results arrive on <code>mpc.mpc_reshare_result.*</code>; there's a runnable <code>examples/reshare/main.go</code> and it's e2e-tested for both ECDSA and EdDSA. The redistribution stores the new shares under the same <code>walletID</code> with only an internal share-version bump, so the master public key — and every CKD-derived address — survives untouched. Add a node by including its peer ID in <code>nodeIDs</code>; remove one or raise the threshold by editing <code>nodeIDs</code> / <code>newThreshold</code>.</p>
<p>Two honest caveats, because this is where "read the code, not the pitch" earns its keep:</p>
<ul>
<li><p><strong>It's a two-part move, not one command.</strong> There's no <code>mpcium-cli reshare</code>. A full membership change is: register the new peer first (generate its <code>identity.json</code>, then <code>mpcium-cli register-peers</code>), <em>then</em> the <code>Resharing()</code> call that actually hands it a share. <a href="https://github.com/fystack/mpcium/issues/84">Issue #84</a> is still open — but for a turnkey add/remove-node CLI and end-to-end docs, not because the crypto is missing.</p>
</li>
<li><p><strong>A reshare rotates the master shares</strong> (new version, new committee in <code>threshold_keyinfo</code>), so it breaks Section 7's "one backup, forever": re-take the MPC backup after every reshare.</p>
</li>
</ul>
<p>Still, pick your <code>n</code> and threshold before the master DKG when you can — a reshare is real coordinated work, and the fewer you run against a cluster holding real funds, the better.</p>
<hr />
<h2>Section 12: Troubleshooting cheat sheet</h2>
<p>The failure modes you're most likely to hit, and the one-line fix for each:</p>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>Cause</th>
<th>Fix</th>
</tr>
</thead>
<tbody><tr>
<td><code>err_code=10052 ... retention policy</code> crash-loop</td>
<td>App/client created NATS streams before the nodes (wrong retention)</td>
<td>Recreate NATS; always start nodes first</td>
</tr>
<tr>
<td>Node permission-denied on <code>data/</code></td>
<td>Distroless UID 65532 vs. host ownership</td>
<td><code>chown -R 65532:65532</code> the node dirs</td>
</tr>
<tr>
<td>Signing fails <code>ERROR_KEY_NOT_FOUND</code> (<code>Key info not found</code>)</td>
<td>Consul KV (<code>threshold_keyinfo/*</code>) not restored</td>
<td><code>consul kv import</code> the KV dump</td>
</tr>
<tr>
<td>Cluster <code>[READY]</code> but signing fails <code>ERROR_KEY_NOT_FOUND</code> (<code>ERROR_NOT_MAJORITY</code> on v0.3.5)</td>
<td>Backup restored one dir too high → empty DB, shares gone</td>
<td>Restore into the exact <code>.../data/nodeN</code> subdir; verify by signing</td>
</tr>
<tr>
<td>DKG hangs on "waiting for keygen result"</td>
<td>The backend consumed the result event</td>
<td>Stop the app before running init</td>
</tr>
<tr>
<td>DKG times out</td>
<td>Not all three nodes were ready</td>
<td>Keygen needs all nodes; wait for three <code>[READY]</code></td>
</tr>
<tr>
<td>Node fatals opening the encrypted BadgerDB</td>
<td><code>badger_password</code> is not 16/24/32 bytes</td>
<td>Generate with <code>openssl rand -hex 16</code></td>
</tr>
<tr>
<td>Nodes misbehave after re-running <code>docker compose up</code></td>
<td>An override recreated a stateful service (NATS/Consul), wiping its state</td>
<td>Restart the nodes after any infra bring-up; treat NATS/Consul as stateful</td>
</tr>
</tbody></table>
<p>Look at the pattern. Not one of these is a cryptography failure. They're orchestration, filesystem, and config failures. That's the real lesson of running MPC in production: the math is the part that works, and the operations are where you earn your keep.</p>
<hr />
<h2>Section 13: Build it so you don't need the luck</h2>
<p>A self-hosted 2-of-3 MPC signing cluster isn't exotic infrastructure. It's a handful of open-source components wired together with a few load-bearing rules: get the startup order right, get the threshold semantics right, back up three things instead of one, and drill your disaster recovery before you need it. Because the failure that actually hurts you isn't a dramatic one. It's a cluster that looks perfectly healthy while it has silently lost every key.</p>
<p>So if you take one thing from all of this, take the opening. Run the drill. A cluster that boots <code>[READY]</code> with every key silently gone isn't a hypothetical, it's the first thing that goes wrong the day you finally need your backup, and it stays invisible until you sign and check. We found ours with half an hour to spare and a lot of luck. Build the thing so you don't need the luck.</p>
<hr />
<p><em>If this was useful, I write about backend, crypto infrastructure, and shipping distributed systems to production.</em></p>
<p><strong>Follow me:</strong></p>
<ul>
<li><p>LinkedIn: <a href="https://www.linkedin.com/in/anatolii-zhadan/">https://www.linkedin.com/in/anatolii-zhadan/</a></p>
</li>
<li><p>Hackernoon: <a href="https://hackernoon.com/u/zhadan">https://hackernoon.com/u/zhadan</a></p>
</li>
<li><p>Github: <a href="https://github.com/ImmuneFOMO">https://github.com/ImmuneFOMO</a></p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[How to Store Your Users' Crypto Wallets So Nobody Can Steal Them]]></title><description><![CDATA[How to Store Your Users' Crypto Wallets So Nobody Can Steal Them]]></description><link>https://anatoliizhadan.hashnode.dev/how-to-store-your-users-crypto-wallets-so-nobody-can-steal-them</link><guid isPermaLink="true">https://anatoliizhadan.hashnode.dev/how-to-store-your-users-crypto-wallets-so-nobody-can-steal-them</guid><category><![CDATA[mpc]]></category><category><![CDATA[Crypto Wallet Security]]></category><category><![CDATA[Web3]]></category><category><![CDATA[custody]]></category><category><![CDATA[Cryptography]]></category><dc:creator><![CDATA[Anatolii Zhadan]]></dc:creator><pubDate>Tue, 07 Jul 2026 09:30:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a4c06e28a429c1defdba5ba/8ac2b9c0-9916-451c-a398-4a70d1ec053f.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><img src="https://miro.medium.com/v2/1*J6DnowIGW8LXv57FwrJgvg.png" alt="Encrypting the private key still leaves one decryptable key behind one server — the honeypot that MPC later removes." />
<em>Encrypting the private key still leaves one decryptable key behind one server — the honeypot that MPC later removes.</em></p>
<p>Your backend can decrypt every user's private key. So can your <code>ENCRYPTION_KEY</code>. So can whoever gets far enough into your servers: a leaked cloud token, a poisoned CI job, one unlucky RCE.</p>
<p>You encrypted the keys. You feel safe. You've quietly built the biggest honeypot in the company.</p>
<p>I know, because that's how we started: <code>Wallet.createRandom()</code>, AES-256-GCM, an encrypted blob in Postgres, one <code>ENCRYPTION_KEY</code> in the env. It ships fast and it passes the demo. It also means one stolen secret unlocks every user at once. That's the exact failure that drained Ronin (<del>$625M) and Harmony (</del>$100M). Nobody cracked the cryptography in either case. They found the one place that could.</p>
<p>This article is the fix I wish I'd read first: MPC-based threshold signing, where the private key is never assembled anywhere. We'll cover what it actually changes, what it does <em>not</em> solve, which protocols and libraries are safe to build on in 2026, and how to migrate an existing "encrypted key in a database" system without a big-bang cutover.</p>
<p>If you already run custodial Web3 infra and know how MPC-TSS works, skip to <strong>Section 4</strong> (the protocol reality check) and <strong>Section 7</strong> (one master wallet for a million users). That's where the real decisions live.</p>
<h2>Table of Contents</h2>
<ol>
<li><a href="https://medium.com/p/b0dd840211f6#4e40">The real problem — it's not encryption, it's centralization</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#2c00">What MPC-TSS actually changes</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#0200">MPC in 60 seconds, for engineers</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#da00">The protocol reality check (2026)</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#5540">The open-source options: build your own vs. a turnkey server</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#c300">Licensing traps</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#b900">One master wallet for a million users (CKD)</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#f400">Who actually runs MPC in production</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#5f00">Migrating from encrypted keys to MPC</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#0000">The mistakes teams make (and what MPC won't save you from)</a></li>
<li><a href="https://medium.com/p/b0dd840211f6#9a00">One strong primitive, not the whole system</a></li>
</ol>
<h2>Section 1: The real problem — not encryption, but centralization</h2>
<p>Here's the custodial flow almost every team ships first. It works, and that's the trap:</p>
<ol>
<li>Generate a wallet, a <code>secp256k1</code> private key</li>
<li>Encrypt it with AES-256-GCM</li>
<li>Store the encrypted blob in PostgreSQL</li>
<li>Decrypt it in the backend whenever you need to sign</li>
</ol>
<p>This protects you against exactly one thing: a database-only leak. Steal the Postgres dump and nothing else, and all you have is ciphertext.</p>
<p>But your backend has to decrypt to function. So an attacker who gets enough control of it, through RCE, stolen cloud credentials, a CI/CD compromise, or a bad insider, doesn't need to break any cryptography. They use your own signing path. The AES does nothing, because your server was always going to decrypt.</p>
<p>This keeps draining real money, and it always looks the same:</p>
<ul>
<li><strong><a href="https://www.elliptic.co/blog/540-million-stolen-from-the-ronin-defi-bridge">Ronin Bridge</a> (2022, ~$625M):</strong> attackers got 5 of 9 validator keys. Control was too concentrated; a Lazarus phishing campaign was enough to reach them.</li>
<li><strong><a href="https://www.halborn.com/blog/post/explained-the-harmony-horizon-bridge-hack">Harmony Horizon Bridge</a> (2022, ~$100M):</strong> a 2-of-5 multisig where the attacker collected just 2 keys and drained it.</li>
</ul>
<p>Nobody cracked ECDSA. They collected enough key material from too few places. So the goal isn't "encrypt the private key better." It's this:</p>
<blockquote>
<p>Remove the possibility that any single machine, process, or operator can unilaterally use the key.</p>
</blockquote>
<p>That's what multi-party computation gives you.</p>
<h2>Section 2: What MPC-TSS actually changes</h2>
<p>In threshold MPC signing (MPC-TSS), the private key is never assembled in one place. Not in memory, not on disk, not for a microsecond.</p>
<p>The key is split into <strong>shares</strong>, each share lives on an independent node, and a threshold of nodes cooperates to produce a signature. The blockchain sees a completely normal ECDSA signature and can't tell the difference.</p>
<p>The canonical setup is three nodes with a 2-of-3 threshold:</p>
<p><img src="https://miro.medium.com/v2/1*vJYq8D4QipDC4OmU5Ur9sg.png" alt="Any two of three nodes can jointly sign. One node, or the backend, is never enough." />
<em>Any two of three nodes can jointly sign. One node, or the backend, is never enough.</em></p>
<p>Walk the attack scenarios and the difference is structural, not cosmetic:</p>
<ul>
<li><strong>The database dump</strong> — Traditional (encrypted key): Ciphertext (safe, for now) · MPC 2-of-3: Nothing useful</li>
<li><strong>Database + encryption key</strong> — Traditional (encrypted key): All keys stolen · MPC 2-of-3: Nothing useful</li>
<li><strong>Full backend / RCE</strong> — Traditional (encrypted key): All keys stolen · MPC 2-of-3: No key exists to steal</li>
<li><strong>One MPC node</strong> — Traditional (encrypted key): — · MPC 2-of-3: Not enough to sign</li>
<li><strong>One disk snapshot of one node</strong> — Traditional (encrypted key): — · MPC 2-of-3: Not enough to sign</li>
</ul>
<p><em>One honest caveat on that last row: an attacker who owns the backend can't exfiltrate a key, because there isn't one. But they can still ask the cluster to sign while they hold the box. MPC removes key theft, not a hijacked signing path. We come back to that in Section 10.</em></p>
<p>The backend holds zero private keys. When it needs a signature, it hashes the payload locally and sends a 32-byte digest to the cluster, which returns <code>(r, s, v)</code>. Own the application server completely, and you get nothing to steal, because the thing you came for was never there.</p>
<h2>Section 3: MPC in 60 seconds (for engineers)</h2>
<p>Two different things get called "MPC," and mixing them up causes half the confusion in this space:</p>
<ol>
<li><strong>General MPC / SMPC</strong> — parties jointly compute a function over private inputs without revealing them. Banks scoring fraud together, hospitals running joint analytics, private ML.</li>
<li><strong>Threshold signature MPC (TSS)</strong> — the specialized case for key management and signing.</li>
</ol>
<p>Wallet infra lives in the second one: threshold ECDSA (Ethereum, Bitcoin, most EVM/L2 chains) or threshold EdDSA (Solana and friends).</p>
<p>There are only two ceremonies to understand:</p>
<p><strong>DKG (distributed key generation).</strong> The nodes jointly generate key shares. No node ever sees the whole private key; each ends up holding its own share plus the shared public key. Keygen needs every participating node online.</p>
<p><strong>Signing.</strong> Your backend hashes the message and sends the 32-byte digest. A threshold of nodes runs the multi-round protocol and returns a standard <code>(r, s, v)</code>, valid on-chain.</p>
<p><img src="https://miro.medium.com/v2/1*Cs3EqWAPuuHf9Jrud-Bjbw.png" alt="DKG runs once. After that, signing only needs a threshold of nodes, not all of them." />
<em>DKG runs once. After that, signing only needs a threshold of nodes, not all of them.</em></p>
<p>One detail trips up everyone on day one:</p>
<blockquote>
<p>In <code>tss-lib</code>-style implementations, <code>threshold</code> is the polynomial degree <code>t</code>, and the number of signers required is <code>t + 1</code>. So a 2-of-3 wallet is configured with <code>threshold: 1</code> (degree 1 → 2 signers), <em>not</em> <code>threshold: 2</code>.</p>
</blockquote>
<p>Get that wrong and you've mis-provisioned your security by one. And remember the asymmetry: keygen needs every node, signing needs only the threshold. A node can be down and you can still sign, which is exactly the availability you want.</p>
<h2>Section 4: The protocol reality check (2026)</h2>
<p>This is where naive advice goes wrong. People pick a protocol by name and stop. A protocol name is not a security guarantee. You have to evaluate implementation, version, and threat model together.</p>
<p><img src="https://miro.medium.com/v2/1*2sQonQYVg3uyo0fP7876hg.png" alt="The same protocol family can be safe or unusable depending on the exact library and version. The name alone tells you nothing." />
<em>The same protocol family can be safe or unusable depending on the exact library and version. The name alone tells you nothing.</em></p>
<h3>GG18 / GG20</h3>
<p>The most widely implemented family, and the basis of most turnkey servers. It also had serious holes:</p>
<ul>
<li><a href="https://www.fireblocks.com/blog/gg18-and-gg20-paillier-key-vulnerability-technical-report">**BitForge (CVE-2023-33241)</a>**, disclosed August 2023, CVSS 9.1–9.6. A malicious signer crafts a bad Paillier modulus and, in some implementations, extracts another party's key share in as few as 16 signing attempts (the real range runs from 16 up to ~a billion, depending on the implementation). The catch: it needs a malicious <em>participating</em> party. If you control all your own nodes, an outside attacker can't mount it.</li>
<li><a href="https://verichains.io/tsshock/">**TSSHOCK</a>** (Verichains, Black Hat USA 2023), a related class against GG18/GG20/CGGMP21 that can "leave no trace and appear innocent to the other parties."</li>
</ul>
<p>The fix, Paillier-Blum plus no-small-factor proofs, is in <code>tss-lib</code> v2.x and maintained forks. So GG18/GG20 on a <em>patched</em> implementation, with all nodes under your control, is a defensible self-hosted choice. On an unpatched one, it isn't.</p>
<h3>CGGMP21</h3>
<p>Newer, academically stronger, and carrying a live problem in Rust. <a href="https://rustsec.org/advisories/RUSTSEC-2025-0127.html">**CVE-2025-66017 / RUSTSEC-2025-0127</a>**: signature forgery through altered presignatures, CVSS 8.2. The advisory is blunt: "Patched: no patched versions" for the <code>cggmp21</code> crate. There's a second twist that lands right on our path here: presignatures plus HD/BIP-32 derivation drop effective security to 85 bits, and this whole article is about to steer you toward HD derivation (Section 7).</p>
<p>Every serious writeup says the same thing: don't start a new system on raw <code>cggmp21</code>.</p>
<h3>CGGMP24</h3>
<p>The safer forward path. The same class of issue is patched in <code>[v0.7.0-alpha.2</code>](<a href="https://github.com/LFDT-Lockness/cggmp21">https://github.com/LFDT-Lockness/cggmp21</a>), and the API hardening is real. If you're choosing a modern protocol today, this is the direction.</p>
<h3>DKLS-family (DKLS23)</h3>
<p>Strong research, good performance in specific stacks. Watch the license (Section 6). Implementation quality and licensing vary wildly by vendor.</p>
<ul>
<li><strong>GG18 / GG20</strong> — Status 2026: Mature, patched forks exist · New self-hosted system?: Yes, if patched and all nodes yours</li>
<li><strong>CGGMP21</strong> — Status 2026: Unpatched forgery CVE · New self-hosted system?: No, don't start here</li>
<li><strong>CGGMP24</strong> — Status 2026: Patched (<code>v0.7.0-alpha.2</code>) · New self-hosted system?: Yes, modern forward path</li>
<li><strong>DKLS23</strong> — Status 2026: Strong, license-dependent · New self-hosted system?: Only after reading the LICENSE</li>
</ul>
<p>Never trust a "we use protocol X" claim. Check the exact library, the exact version, whether the CVE patch is actually in the code path you call, and whether your threat model even includes a malicious party.</p>
<h2>Section 5: Build your own vs. a turnkey server</h2>
<p>If you want self-hosted MPC today, you have two real options.</p>
<h3>Option A — build on a library</h3>
<p>You take a cryptographic library and build the server, networking, orchestration, storage, backups, and observability yourself.</p>
<ul>
<li><code>[**bnb-chain/tss-lib</code>](<a href="https://github.com/bnb-chain/tss-lib">https://github.com/bnb-chain/tss-lib</a>)** — Language: Go · License: MIT · Audit: Kudelski (Oct 2019) · Notes: The de-facto GG18/GG20 lib</li>
<li><code>[**coinbase/cb-mpc</code>](<a href="https://github.com/coinbase/cb-mpc">https://github.com/coinbase/cb-mpc</a>)** — Language: C++ · License: MIT · Audit: Cure53 (Q4 2024) · Notes: High-quality, library only — no server/storage/API</li>
<li><strong>LFDT-Lockness <code>[cggmp24</code>](<a href="https://github.com/LFDT-Lockness/cggmp21">https://github.com/LFDT-Lockness/cggmp21</a>)</strong> — Language: Rust · License: MIT/Apache-2.0 · Audit: Kudelski · Notes: Modern protocol, patched</li>
</ul>
<p>You get total control and protocol-level flexibility. You also now own networking, retries, key storage, backups, disaster recovery, monitoring, and the entire incident-response surface. That's weeks to months of work before you sign your first production transaction.</p>
<h3>Option B — an open-source MPC server</h3>
<p>The most turnkey route right now is <a href="https://github.com/fystack/mpcium">**Mpcium</a>**: a Go server backed by <code>tss-lib</code>, shipping a full cluster (nodes, message bus, service discovery, encrypted storage) plus client SDKs, under Apache-2.0.</p>
<p>You get a real cluster out of the box and a dramatically lower starting cost. In exchange you accept an opinionated architecture, feature gaps you'll fork around (trusted-dealer key import, a REST façade over its NATS pub/sub), and a DevOps burden that's still yours.</p>
<p>The heuristic I keep coming back to: weeks of building networking, storage, and DR from scratch, versus adding one feature to a server that already works. Unless you have a hard protocol requirement, extend the existing server.</p>
<p>A quick note on the rest of the field, so you don't lose a week evaluating them:</p>
<ul>
<li><strong>ZenGo</strong> — the <code>[multi-party-ecdsa</code>](<a href="https://github.com/ZenGo-X/multi-party-ecdsa">https://github.com/ZenGo-X/multi-party-ecdsa</a>) library is explicitly abandoned; <code>[gotham-city</code>](<a href="https://github.com/ZenGo-X/gotham-city">https://github.com/ZenGo-X/gotham-city</a>) is 2-of-2 only, demo-grade, and effectively dormant (last commit 2024).</li>
<li><a href="https://github.com/safeheron">**Safeheron</a>** — open crypto libraries, proprietary server.</li>
<li><a href="https://www.sodot.dev/">**Sodot</a>** — self-hostable, but proprietary: a closed-source SDK you can't inspect or fork.</li>
<li><a href="https://litprotocol.com/">**Lit Protocol</a>** — a decentralized node network you consume, not a private cluster you run yourself.</li>
</ul>
<h2>Section 6: Licensing traps</h2>
<p>This one surprises people in legal review, not code review.</p>
<p>A repo can be source-available or non-commercial while looking open at a glance. The protocol paper can be fully open while the implementation's license is restrictive.</p>
<p>The cleanest example: one popular <code>dkls23</code> implementation (<a href="https://github.com/silence-laboratories/dkls23">Silence Labs</a>) ships under a non-commercial license (SLL), and commercial use needs a separate paid license. A <em>different</em>, independent DKLs23 implementation (<a href="https://github.com/0xCarbon/DKLs23">0xCarbon</a>) is dual MIT/Apache-2.0. Same protocol, completely different rights.</p>
<p>Open the actual <code>LICENSE</code> file, in the exact repo and version you plan to ship. Not the README, not the paper. The license, pinned to your version.</p>
<h2>Section 7: One master wallet for a million users (CKD)</h2>
<p>Give every user their own wallet the obvious way and each one gets its own DKG across your nodes. That's Mpcium's default, and it works fine. The catch isn't performance, it's what it does to your backups and your operations.</p>
<p>Every per-user DKG needs all three nodes online, and every wallet it creates is another independent set of shards on every node. So the MPC state you have to protect grows with your user base: every signup is one more ceremony to run and one more set of shards you can never lose. At a handful of users that's nothing. At a few million, provisioning and backup are the whole problem.</p>
<p>CKD collapses all of it into one ceremony. BIP-32 hierarchical deterministic wallets, the trick Bitcoin gave us in 2012, derive an effectively infinite tree of child keys from one master. Run the DKG once, and every user after that is a pure derivation off the master — no ceremony, no nodes online, and a single backup covers all of them. That's <strong>CKD (child key derivation)</strong>:</p>
<ol>
<li>Run one DKG at init → a single master MPC wallet (still split into 3 shards, still never assembled)</li>
<li>For each user, derive <code>child = derive(masterPubKey, chainCode, userIndex)</code>. Pure math, no ceremony, no nodes required online</li>
<li>To sign, each node applies the derivation offset to its own share and signs. The child key, like the master, is never assembled</li>
</ol>
<p><img src="https://miro.medium.com/v2/1*frKfbNDcHvpwVl1fTr3sXQ.png" alt="Per-user DKG on the left runs a full ceremony per user. CKD on the right derives every user from one master, with no per-user ceremony." />
<em>Per-user DKG on the left runs a full ceremony per user. CKD on the right derives every user from one master, with no per-user ceremony.</em></p>
<p>The operational gap isn't incremental, it's categorical:</p>
<ul>
<li><strong>Creating a user</strong> — Per-user DKG: a key ceremony, all nodes · CKD (one master): a derivation, no ceremony</li>
<li><strong>10,000 users</strong> — Per-user DKG: 10k ceremonies, 30k shards · CKD (one master): 1 ceremony, 3 shards</li>
<li><strong>Backup cadence</strong> — Per-user DKG: after every new user · CKD (one master): once, at init</li>
<li><strong>MPC node down</strong> — Per-user DKG: new users wait · CKD (one master): new users still created</li>
<li><strong>Disaster recovery</strong> — Per-user DKG: restore everything · CKD (one master): restore master; users are DB rows</li>
</ul>
<p>"Is it as secure?" is the fair question, and the answer is yes, under the same threat model:</p>
<ul>
<li><strong>1 node compromised</strong> — Per-user DKG: Safe · CKD: Safe</li>
<li><strong>2 of 3 nodes compromised</strong> — Per-user DKG: Compromised · CKD: Compromised</li>
<li><strong>Backup stolen</strong> — Per-user DKG: Compromised · CKD: Compromised</li>
<li><strong>2 nodes in a secure enclave</strong> — Per-user DKG: Protected · CKD: Protected</li>
</ul>
<p>That equivalence is about <em>node compromise</em>. Be clear-eyed about what CKD gives up, because this is the part the marketing pages skip:</p>
<ul>
<li>You can't rotate or revoke a single user's key in isolation. Everyone derives from the same master; there's no per-user share to roll.</li>
<li>You can't prove a user's key is cryptographically independent of the others, which some regulatory or audit regimes want.</li>
<li>Anyone holding the master public key and chain code can enumerate every user address. For an opt-in custodial service that's a non-issue; for a privacy-sensitive product it's a real deanonymization vector.</li>
</ul>
<p>If none of that bites your product, CKD is a clean way to scale. If one does — and for many products one will — per-user DKG is the safer default, and that's when it earns its cost.</p>
<p>The BIP-32 caveat, stated honestly: with the parent public key, the chain code, and a full child private key, you can recover the parent key. In MPC that's defused, because the child private key never exists as a whole; each node holds only a share delta. The <code>chain_code</code> isn't secret-shared either (every node has a full copy), but that doesn't change the picture: recovering the parent still needs a full child private key, and that needs a threshold of nodes, which is total compromise already.</p>
<p>I didn't take any of this on faith. An end-to-end recovery test ran the full flow, create the master, derive several users, sign with each, back up, destroy the cluster, restore, verify, and passed 12 of 12 checks: every derived address matched, every signature still verified, and a brand-new user created <em>after</em> the restore worked too. A separate cross-machine drill, destroy the cluster and rebuild on fresh hardware, is what flushed out eleven runbook bugs, one of which would have quietly lost every key in a real emergency. The next article takes that drill apart in full.</p>
<h2>Section 8: Who actually runs MPC in production</h2>
<p>MPC is no longer a crypto-startup curiosity. The clearest sign of how seriously the industry takes it: <strong>Fireblocks</strong>, one of the largest custody providers, is the team that discovered and disclosed the <a href="https://www.fireblocks.com/blog/bitforge-fireblocks-researchers-uncover-vulnerabilities-in-over-15-major-wallet-providers">BitForge</a> vulnerability class from Section 4. When your custody vendor is the one running the offensive research that finds MPC's own weak spots, the primitive has left the lab.</p>
<p>Who else is standing on it:</p>
<ul>
<li><a href="https://newsroom.ibm.com/ibm-digital-asset-haven">**IBM Digital Asset Haven</a>** (2025), developed with <strong>Dfns</strong>, combines Dfns's threshold signing with IBM's HSM and offline-signing models for institutional custody. MPC under an IBM-branded product is about as mainstream as it gets.</li>
<li><strong>Dfns</strong> runs its <a href="https://dfns.co/article/ku23/">KU23 protocol</a> on its key-management network — by its own figures 10M+ keys and ~$1B signed per month — and plans to open-source it and move governance to the Linux Foundation's LF Decentralized Trust (the Lockness project).</li>
<li><strong>Coinbase</strong> open-sourced its own MPC library, <code>[cb-mpc</code>](<a href="https://github.com/coinbase/cb-mpc">https://github.com/coinbase/cb-mpc</a>) (MIT, Cure53-audited). You don't open-source your custody crypto unless it's core to how you operate.</li>
<li><strong><a href="https://fordefi.com/">Fordefi</a>, <a href="https://www.bitgo.com/">BitGo</a>, and <a href="https://copper.co/">Copper</a></strong> ship MPC/TSS custody to institutional desks and DeFi.</li>
</ul>
<p>Adopting MPC isn't an exotic bet. It's the same primitive regulated, multi-billion-dollar custodians standardized on.</p>
<h2>Section 9: Migrating from encrypted keys to MPC</h2>
<p>You rarely start clean. Here's a rollout that doesn't need a big-bang cutover.</p>
<p><img src="https://miro.medium.com/v2/1*JPT6SE-9ytS74Qjy4Xzc1Q.png" alt="The safe path is a signer abstraction: your service code never changes, only what sits behind the interface does." />
<em>The safe path is a signer abstraction: your service code never changes, only what sits behind the interface does.</em></p>
<ol>
<li><strong>Inventory every signing call site.</strong> Every place a key is created, decrypted, or used, across services, jobs, and scripts.</li>
<li><strong>Introduce a signer abstraction.</strong> Replace direct wallet creation with a single signing interface so the rest of your code stops caring where the key lives.</li>
</ol>
<pre><code>   // Before: the key exists in your process
   const wallet = new ethers.Wallet(decrypt(row.encryptedKey));
   const sig = await wallet.signTransaction(tx);

   // After: the key never exists here — you send a digest out to sign
   const signer = mpc.getSigner(user.derivationPath);
   const sig = await signer.signTransaction(tx); // digest -&gt; MPC -&gt; (r, s, v)
</code></pre>
<ol>
<li><strong>Add MPC signer adapters.</strong> One thin adapter for <code>ethers</code>, one for <code>viem</code>, both just forwarding the digest to the cluster.</li>
<li><strong>Dual-run behind a flag.</strong> <code>MPC_ENABLED</code>, rolled out to a controlled cohort first.</li>
<li><strong>Cut over new users first.</strong> New wallets go through MPC; the legacy path becomes read-only.</li>
<li><strong>Migrate or rotate legacy keys.</strong> Import if your server supports trusted-dealer import; otherwise move funds to MPC-managed wallets and retire the old ones.</li>
<li><strong>Delete the decryption path.</strong> Rip out every <code>getPrivateKey()</code>-style helper. If the code can't decrypt a key, neither can an attacker who owns the process.</li>
<li><strong>Test disaster recovery before launch.</strong> Restore drills are mandatory, not optional. (The next article is one long argument for why.)</li>
</ol>
<h2>Section 10: The mistakes teams make</h2>
<p>After shipping this to production, the most useful thing I learned is that the cryptography almost never breaks. The orchestration does. The mistakes repeat:</p>
<ul>
<li>Misreading threshold semantics, the <code>t</code> vs <code>t + 1</code> count from Section 3.</li>
<li>Underestimating the message bus and metadata store. The signing protocol is chatty and stateful, and your NATS/Consul/BadgerDB layer is now load-bearing.</li>
<li>Assuming one datastore backup is enough. It isn't; you need the key shares <em>and</em> the routing metadata.</li>
<li>Running keygen while application consumers race for the result events, silently stealing DKG output from your init flow.</li>
<li>Treating MPC as "security complete" and skipping the policy layer, which brings us to the most important caveat here.</li>
</ul>
<h3>MPC is not magic</h3>
<p>MPC dramatically reduces key-<em>theft</em> risk. On its own it does nothing about:</p>
<ul>
<li>signing the wrong payload, if your backend is tricked into it</li>
<li>compromised approval or policy logic</li>
<li>frontend / UI supply-chain attacks that fool an operator into approving a bad action</li>
<li>broad application-layer abuse after a backend compromise</li>
</ul>
<p>The cautionary tale is <a href="https://www.elliptic.co/blog/bybit-hack-largest-in-history">**Bybit (February 2025, ~$1.5B)</a>**, the largest crypto theft to date. Bybit <em>had</em> multisig (Safe). The attackers didn't break the keys; they compromised the UI so operators approved a malicious transaction. MPC wouldn't have stopped it, because the failure was in intent, not custody.</p>
<p>So treat MPC as a very strong primitive inside a layered system. Policy engine, transaction simulation, allowlists, anomaly detection, segregated duties, hardened ops: all still matter.</p>
<h2>Section 11: One strong primitive, not the whole system</h2>
<p>If your backend can decrypt every user's private key today, you don't have a cryptography problem. You have a system-design problem: one key, one server, one blast radius.</p>
<p>MPC-TSS is the most practical way I know to redesign that system so key theft becomes structurally hard and the blast radius of a compromise shrinks. Not impossible. Materially harder. In security architecture, that difference is the whole game.</p>
<p>One warning before you run off and build it, though. The day I moved our first wallet onto MPC and deleted the old <code>getPrivateKey()</code> helper felt great, right up until I realized MPC had solved the <em>theft</em> problem and none of the others. It will faithfully sign the wrong transaction if something upstream tells it to. So ship it as one strong primitive in a layered system, not as the thing that lets you stop worrying.</p>
<p>Next up, the hands-on half: deploying a self-hosted 2-of-3 MPC signing cluster with Docker. The DKG ceremony, the backup model, and the disaster-recovery drill that caught a silent, cluster-killing bug — the one I mentioned above, in full.</p>
<p><em>If this was useful, I write about backend, crypto infrastructure, and shipping distributed systems to production.</em></p>
<p><strong>Follow me:</strong></p>
<p>LinkedIn: <a href="https://www.linkedin.com/in/anatolii-zhadan/">https://www.linkedin.com/in/anatolii-zhadan/</a>
Hackernoon: <a href="https://hackernoon.com/u/zhadan">https://hackernoon.com/u/zhadan</a>
Github: <a href="https://github.com/ImmuneFOMO">https://github.com/ImmuneFOMO</a></p>
]]></content:encoded></item></channel></rss>