Killing duplicate Telegram posts with idempotent sends and rate limits

<p>Our channel bot used to publish the same announcement twice, sometimes three times. Not because the code was clever — because it was naive: a retry after a timeout, a restart halfway through a broadcast, an admin tapping the "publish" button twice. Each of those produced a fresh <code>sendMessage</code> call, and the Bot API happily accepted it.</p> <p>Here is the setup that fixed it for us. It is roughly 45 minutes of work at an intermediate level, and it is built from three cheap layers: a content fingerprint in SQLite, a per-minute token bucket, and a permalink check after the fact.</p> <p>The whole flow looks like this:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>publisher -&gt; [1] idempotency key -&gt; [2] rate limiter -&gt; [3] sendMessage | | | SQLite outbox token bucket message_id (UNIQUE key) (per-minute cap) | | | +---------- skip on duplicate &lt;--------------+ </code></pre> </div> <h2>

What you need before you start

</h2> <p>Python 3.9+ with <code>requests</code>, a bot token from <a class="mentioned-user" href="https://dev.to/botfather">@botfather</a>, and a private test channel where that bot is an administrator with the "Post messages" permission. Note both the numeric channel id (<code>-100...</code>) and the public username, since both end up in the code. Keep the Bot API docs open in a tab — the limits below are what we worked with, but Telegram adjusts them, so verify the current numbers before you hardcode production values.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code><span class="nb">mkdir</span> <span class="nt">-p</span> ~/tg-dedup <span class="o">&amp;&amp;</span> <span class="nb">cd</span> ~/tg-dedup python3 <span class="nt">-m</span> venv .venv <span class="o">&amp;&amp;</span> <span class="nb">source</span> .venv/bin/activate pip <span class="nb">install </span>requests

<span class="nb">export </span><span class="nv">TG_BOT_TOKEN</span><span class="o">=</span><span class="s2">"<token-from-BotFather>"</span> <span class="nb">export </span><span class="nv">TG_CHANNEL</span><span class="o">=</span><span class="s2">"-1001234567890"</span> <span class="nb">export </span><span class="nv">TG_CHANNEL_USER</span><span class="o">=</span><span class="s2">"my_test_channel"</span> </code></pre>

</div> <p>The token lives in an environment variable and never in the repository. Quick sanity check: <code>echo "${TG_BOT_TOKEN:0:10}"</code> should print the first ten characters.</p> <h2>

Layer 1: fingerprint the content, not the click

</h2> <p>An idempotency key is a fingerprint of an operation. Run the same operation twice, get the same fingerprint, and the second run is thrown away instead of executed. For us the fingerprint is a SHA-256 hash of the channel id plus the normalized message text, truncated to 32 hex characters.</p> <p>Normalization matters more than the hash. Collapsing whitespace and lowercasing means <code>"Hello World"</code> and <code>"hello world"</code> produce one key — which is exactly the duplicate we kept getting from a re-rendered template.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="c1"># dedup.py </span><span class="kn">import</span> <span class="n">hashlib</span> <span class="kn">import</span> <span class="n">sqlite3</span>

<span class="n">SCHEMA</span> <span class="o">=</span> <span class="sh">"""</span><span class="s"> CREATE TABLE IF NOT EXISTS outbox ( idem_key TEXT PRIMARY KEY, channel_id TEXT NOT NULL, text TEXT NOT NULL, status TEXT NOT NULL DEFAULT </span><span class="sh">'</span><span class="s">pending</span><span class="sh">'</span><span class="s">, -- pending | sent | failed message_id INTEGER, attempts INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime(</span><span class="sh">'</span><span class="s">now</span><span class="sh">'</span><span class="s">)), sent_at TEXT ); </span><span class="sh">"""</span>

<span class="k">def</span> <span class="nf">idempotency_key</span><span class="p">(</span><span class="n">channel_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span> <span class="n">normalized</span> <span class="o">=</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="nf">split</span><span class="p">()).</span><span class="nf">strip</span><span class="p">().</span><span class="nf">lower</span><span class="p">()</span> <span class="n">raw</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">channel_id</span><span class="si">}</span><span class="s">:</span><span class="si">{</span><span class="n">normalized</span><span class="si">}</span><span class="sh">"</span><span class="p">.</span><span class="nf">encode</span><span class="p">()</span> <span class="k">return</span> <span class="n">hashlib</span><span class="p">.</span><span class="nf">sha256</span><span class="p">(</span><span class="n">raw</span><span class="p">).</span><span class="nf">hexdigest</span><span class="p">()[:</span><span class="mi">32</span><span class="p">]</span>

<span class="k">def</span> <span class="nf">init_db</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="sh">"</span><span class="s">outbox.db</span><span class="sh">"</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span> <span class="n">con</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="n">con</span><span class="p">.</span><span class="nf">executescript</span><span class="p">(</span><span class="n">SCHEMA</span><span class="p">)</span> <span class="n">con</span><span class="p">.</span><span class="nf">commit</span><span class="p">()</span> <span class="n">con</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span>

<span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">:</span> <span class="nf">init_db</span><span class="p">()</span> </code></pre>

</div> <p>Check that normalization actually works before moving on:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>python3 dedup.py python3 <span class="nt">-c</span> <span class="s2">" from dedup import idempotency_key print(idempotency_key('-1001234567890', 'Hello World')) print(idempotency_key('-1001234567890', 'hello world')) "</span> </code></pre> </div> <p>Both lines must be identical 32-character hex strings. If they differ, your normalization is broken and every downstream layer is decoration.</p> <h2>

Layer 2: let SQLite decide who wins

</h2> <p>Because <code>idem_key</code> is the <code>TEXT PRIMARY KEY</code>, the database itself becomes the arbiter. The second insert of the same key conflicts, and <code>ON CONFLICT DO NOTHING</code> turns that conflict into a silent no-op. Two processes cannot both walk away thinking they own the message.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span> <span class="sh">"""</span><span class="s">Insert first; False means someone already owns this key.</span><span class="sh">"""</span> <span class="n">con</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span> <span class="n">cur</span> <span class="o">=</span> <span class="n">con</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span> <span class="sh">"</span><span class="s">INSERT INTO outbox (idem_key, channel_id, text) VALUES (?, ?, ?) </span><span class="sh">"</span> <span class="sh">"</span><span class="s">ON CONFLICT(idem_key) DO NOTHING</span><span class="sh">"</span><span class="p">,</span> <span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">CHANNEL</span><span class="p">,</span> <span class="n">text</span><span class="p">),</span> <span class="p">)</span> <span class="n">con</span><span class="p">.</span><span class="nf">commit</span><span class="p">()</span> <span class="n">claimed</span> <span class="o">=</span> <span class="n">cur</span><span class="p">.</span><span class="n">rowcount</span> <span class="o">==</span> <span class="mi">1</span> <span class="n">con</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span> <span class="k">return</span> <span class="n">claimed</span> </code></pre> </div> <p>The ordering is the point: the row is written <em>before</em> the network call, not after. If the process dies between the insert and the send, you are left with an explicit <code>pending</code> row you can inspect and decide about — instead of an invisible duplicate you find out about from a subscriber.</p> <h2>

Layer 3: a token bucket plus retry_after

</h2> <p>One thing worth internalizing: <code>sendMessage</code> has no idempotency parameter. Deduplication is entirely your job (confirm the current parameter list in the docs). The second part of the job is not getting throttled during a broadcast loop.</p> <p>The limiter below is a token bucket — it keeps timestamps of recent sends and sleeps when the per-minute cap is reached. We sized it at 20 per minute for a single channel, based on the documented figure of roughly 20 messages per minute per group or channel and a global cap near 30 messages per second.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="c1"># publisher.py </span><span class="kn">import</span> <span class="n">os</span> <span class="kn">import</span> <span class="n">sqlite3</span> <span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">time</span>

<span class="kn">import</span> <span class="n">requests</span>

<span class="kn">from</span> <span class="n">dedup</span> <span class="kn">import</span> <span class="n">idempotency_key</span><span class="p">,</span> <span class="n">init_db</span>

<span class="n">BOT_TOKEN</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="sh">"</span><span class="s">TG_BOT_TOKEN</span><span class="sh">"</span><span class="p">]</span> <span class="n">CHANNEL</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="sh">"</span><span class="s">TG_CHANNEL</span><span class="sh">"</span><span class="p">]</span> <span class="n">USERNAME</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="sh">"</span><span class="s">TG_CHANNEL_USER</span><span class="sh">"</span><span class="p">]</span> <span class="n">API</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="s">https://api.telegram.org/bot</span><span class="si">{</span><span class="n">BOT_TOKEN</span><span class="si">}</span><span class="sh">"</span> <span class="n">DB</span> <span class="o">=</span> <span class="sh">"</span><span class="s">outbox.db</span><span class="sh">"</span>

<span class="k">class</span> <span class="nc">RateLimiter</span><span class="p">:</span> <span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">per_minute</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">20</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">per_minute</span> <span class="o">=</span> <span class="n">per_minute</span> <span class="n">self</span><span class="p">.</span><span class="n">stamps</span> <span class="o">=</span> <span class="p">[]</span>

textCopy
<span class="k">def</span> <span class="nf">wait</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="bp">None</span><span class="p">:</span>
    <span class="n">now</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="nf">monotonic</span><span class="p">()</span>
    <span class="n">self</span><span class="p">.</span><span class="n">stamps</span> <span class="o">=</span> <span class="p">[</span><span class="n">t</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">self</span><span class="p">.</span><span class="n">stamps</span> <span class="k">if</span> <span class="n">now</span> <span class="o">-</span> <span class="n">t</span> <span class="o">&lt;</span> <span class="mi">60</span><span class="p">]</span>
    <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">stamps</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="n">self</span><span class="p">.</span><span class="n">per_minute</span><span class="p">:</span>
        <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">60</span> <span class="o">-</span> <span class="p">(</span><span class="n">now</span> <span class="o">-</span> <span class="n">self</span><span class="p">.</span><span class="n">stamps</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">+</span> <span class="mf">0.5</span><span class="p">)</span>
    <span class="n">self</span><span class="p">.</span><span class="n">stamps</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="nf">monotonic</span><span class="p">())</span>

<span class="k">def</span> <span class="nf">publish</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">limiter</span><span class="p">:</span> <span class="n">RateLimiter</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span> <span class="n">key</span> <span class="o">=</span> <span class="nf">idempotency_key</span><span class="p">(</span><span class="n">CHANNEL</span><span class="p">,</span> <span class="n">text</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="nf">claim</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">skip: duplicate idem_key=</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span>

textCopy
<span class="n">r</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
    <span class="n">limiter</span><span class="p">.</span><span class="nf">wait</span><span class="p">()</span>
    <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span>
        <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">API</span><span class="si">}</span><span class="s">/sendMessage</span><span class="sh">"</span><span class="p">,</span>
        <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">chat_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">CHANNEL</span><span class="p">,</span> <span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">:</span> <span class="n">text</span><span class="p">,</span> <span class="sh">"</span><span class="s">disable_web_page_preview</span><span class="sh">"</span><span class="p">:</span> <span class="bp">True</span><span class="p">},</span>
        <span class="n">timeout</span><span class="o">=</span><span class="mi">15</span><span class="p">,</span>
    <span class="p">)</span>
    <span class="k">if</span> <span class="n">r</span><span class="p">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="mi">429</span><span class="p">:</span>
        <span class="k">break</span>
    <span class="n">retry_after</span> <span class="o">=</span> <span class="n">r</span><span class="p">.</span><span class="nf">json</span><span class="p">().</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">parameters</span><span class="sh">"</span><span class="p">,</span> <span class="p">{}).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">retry_after</span><span class="sh">"</span><span class="p">,</span> <span class="mi">30</span><span class="p">)</span>
    <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">429: sleeping </span><span class="si">{</span><span class="n">retry_after</span><span class="si">}</span><span class="s">s</span><span class="sh">"</span><span class="p">)</span>
    <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="n">retry_after</span><span class="p">)</span>

<span class="n">body</span> <span class="o">=</span> <span class="n">r</span><span class="p">.</span><span class="nf">json</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">body</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">ok</span><span class="sh">"</span><span class="p">):</span>
    <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">failed: </span><span class="si">{</span><span class="n">body</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">'</span><span class="s">description</span><span class="sh">'</span><span class="p">)</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span>
    <span class="k">return</span>

<span class="n">message_id</span> <span class="o">=</span> <span class="n">body</span><span class="p">[</span><span class="sh">"</span><span class="s">result</span><span class="sh">"</span><span class="p">][</span><span class="sh">"</span><span class="s">message_id</span><span class="sh">"</span><span class="p">]</span>
<span class="n">con</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span>
<span class="n">con</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span>
    <span class="sh">"</span><span class="s">UPDATE outbox SET status=</span><span class="sh">'</span><span class="s">sent</span><span class="sh">'</span><span class="s">, message_id=?, sent_at=datetime(</span><span class="sh">'</span><span class="s">now</span><span class="sh">'</span><span class="s">) </span><span class="sh">"</span>
    <span class="sh">"</span><span class="s">WHERE idem_key=?</span><span class="sh">"</span><span class="p">,</span>
    <span class="p">(</span><span class="n">message_id</span><span class="p">,</span> <span class="n">key</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">con</span><span class="p">.</span><span class="nf">commit</span><span class="p">()</span>
<span class="n">con</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span>
<span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">sent: message_id=</span><span class="si">{</span><span class="n">message_id</span><span class="si">}</span><span class="s"> https://t.me/</span><span class="si">{</span><span class="n">USERNAME</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">message_id</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span>

<span class="k">if</span> <span class="n">name</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">:</span> <span class="nf">init_db</span><span class="p">()</span> <span class="nf">publish</span><span class="p">(</span><span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]),</span> <span class="nc">RateLimiter</span><span class="p">(</span><span class="n">per_minute</span><span class="o">=</span><span class="mi">20</span><span class="p">))</span> </code></pre>

</div> <p>The retry loop gives up after three attempts, and it reads <code>retry_after</code> from the response instead of using a fixed sleep. That detail is what stops a second <code>429</code> from landing one second after the first.</p> <h2>

Step 4: confirm the post is really there

</h2> <p>Run it once:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>python3 publisher.py <span class="s2">"Hello from the outbox"</span> </code></pre> </div> <p>Expected output:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>sent: message_id=42 https://t.me/my_test_channel/42 </code></pre> </div> <p>Then open the permalink. This third layer is the one people skip: an HTTP 200 only proves Telegram accepted <em>something</em>. A permalink check is what catches a wrong channel id, a wrong username, or a post that was deleted right after publishing.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>sqlite3 outbox.db <span class="s2">"SELECT status, message_id FROM outbox;"</span> </code></pre> </div> <p>Run the script again with the same text and you should see <code>skip: duplicate idem_key=...</code> and no new row.</p> <h2>

Things I keep an eye on

</h2> <ul> <li> <strong>Limits drift.</strong> The 30/sec global and ~20/min per-channel figures were accurate when I built this; re-read the docs before setting production values.</li> <li> <strong>One worker only.</strong> The token bucket lives in process memory, so it does nothing for you across multiple workers. Run a single publisher and queue the work.</li> <li> <strong>Failed sends stay visible.</strong> A <code>pending</code> row after a crash is a feature, not a bug — but somebody has to look at it and decide whether to retry.</li> <li> <strong>Environment variables, always.</strong> The token in a shell export is boring and that is precisely why it works.</li> </ul> <h2>

What I took away

</h2> <p>Duplicate posts are rarely a Telegram problem. They are an ordering problem: we called the network before we recorded the intent, so the system had no memory of what it had already done. Writing the intent first, letting a unique key arbitrate, throttling to a documented rate, and verifying with a permalink cost us a table and about 90 lines of Python — and the duplicates stopped.</p> <p>If you want more write-ups like this one, this article was originally published on AI Release — <a href="https://ai-release.org" rel="noopener noreferrer">https://ai-release.org</a>. There is also a channel with new guides at <a href="https://t.me/ai_release1" rel="noopener noreferrer">https://t.me/ai_release1</a>.</p>

Original source

Read the original article on dev.to