<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Telio Blog]]></title><description><![CDATA[The latest blog posts, announcements, and guides.]]></description><link>https://blog.gettelio.com/</link><image><url>https://blog.gettelio.com/favicon.png</url><title>Telio Blog</title><link>https://blog.gettelio.com/</link></image><generator>Ghost 5.75</generator><lastBuildDate>Mon, 06 Apr 2026 13:17:23 GMT</lastBuildDate><atom:link href="https://blog.gettelio.com/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Scaling EV Repairs: How RepairWise Handles 1,000+ Daily Messages with AI]]></title><description><![CDATA[RepairWise uses 24/7 Telio AI agents to manage 1,000+ daily messages, automating 60% of customer SMS and speeding up responses by 2.85x.]]></description><link>https://blog.gettelio.com/how-repairwise-handles-1000-daily-messages/</link><guid isPermaLink="false">6953fd9432554a0001a7602e</guid><category><![CDATA[Customer Stories]]></category><dc:creator><![CDATA[Evgeny Li]]></dc:creator><pubDate>Tue, 06 Jan 2026 07:39:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/12/preview-13.png" medium="image"/><content:encoded><![CDATA[<h3 id="highlights">Highlights</h3><ul><li><strong>Company:</strong> <a href="https://www.repairwise.pro/?ref=blog.gettelio.com">RepairWise</a></li><li><strong>Industry:</strong> Electric Vehicle Diagnostics and Repair</li><li><strong>Challenge:</strong> managing 1,000+ daily messages across customer SMS and repair shop communications while scaling operations</li><li><strong>Solution:</strong> dual AI agent system powered by Telio integrated with:<ul><li><a href="https://docs.gettelio.com/integrations/dialpad?ref=blog.gettelio.com">Dialpad</a>, real-time SMS communications with customers</li><li><a href="https://docs.gettelio.com/integrations/postgres-supabase?ref=blog.gettelio.com" rel="noreferrer">PostgreSQL Database</a>, customers, orders, and repair shops information</li><li><a href="https://docs.gettelio.com/integrations/web?ref=blog.gettelio.com" rel="noreferrer">Repair Manuals</a>, proprietary documentation for EV models</li><li><a href="https://docs.gettelio.com/integrations/google-docs?ref=blog.gettelio.com" rel="noreferrer">Google Docs</a>, response templates and guidelines</li><li><a href="https://docs.gettelio.com/integrations/web?ref=blog.gettelio.com" rel="noreferrer">Help Center</a>, public-facing FAQ and support content</li></ul></li><li><strong>Results:</strong><ul><li>60% of customer SMS fully automated</li><li>40% of customer messages drafted for one-click review and send</li><li>2.85x faster median response time</li><li>100% of repair shop communications receive AI-generated drafts</li><li>Same 9-person team of service advisors and technicians handles growing volume without additional headcount</li></ul></li></ul><h3 id="about-repairwise">About RepairWise</h3><img src="https://blog.gettelio.com/content/images/2025/12/preview-13.png" alt="Scaling EV Repairs: How RepairWise Handles 1,000+ Daily Messages with AI"><p>RepairWise is revolutionizing how electric vehicle owners get their cars serviced. The platform remotely diagnoses EV issues and provides instant online quotes, making repairs more transparent and convenient. RepairWise connects thousands of EV owners with a nationwide network of qualified repair shops across the United States.</p><h3 id="the-challenge-two-communication-channels-one-scaling-problem">The Challenge: Two Communication Channels, One Scaling Problem</h3><p>As RepairWise expanded its network of customers and repair shop partners, the volume of daily communications became overwhelming for its lean team.</p><div class="kg-card kg-callout-card kg-callout-card-blue"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">&quot;We were handling over a thousand messages every single day. On one side, EV owners were texting us constantly with questions about their diagnostics, repair statuses, and appointments. On the other side, our partner mechanics needed technical guidance, parts information, and software update assistance. Both channels were growing faster than we could hire.&quot; </em></i><br><br>&#x2014; James Castillo, Co-Founder &amp; Head of Engineering</div></div><p><strong>On the customer side:</strong> EV owners communicate via SMS through Dialpad, asking about diagnostics, scheduling, pricing, insurance, repair status, and general vehicle health questions.</p><p><strong>On the shop side:</strong> auto mechanics use RepairWise&apos;s proprietary portal and messaging system for technical guidance, diagnostic assistance, parts logistics, software updates, and customer coordination.</p><p>As the business scaled, this structure wasn&apos;t sustainable without a fundamental change in how they handled communications.</p><h3 id="the-solution-two-specialized-ai-agents-working-in-parallel">The Solution: Two Specialized AI Agents Working in Parallel</h3><p>RepairWise built two distinct AI agents using Telio, each tailored to its communication channel and audience. Rather than a one-size-fits-all approach, they created specialized agents that understand the unique context and integrate with the exact knowledge sources needed for that role.</p><p><strong>Agent #1: The Service Advisor for Customer SMS</strong></p><p>The first agent operates as a virtual service advisor, handling SMS conversations with EV owners through Dialpad&apos;s real-time integration.</p><p>How it works:</p><ol><li><strong>Dialpad receives the SMS</strong> with Telio&apos;s real-time integration immediately capturing it</li><li><strong>Analyzes the inquiry</strong> to understand customer intent and categorize it</li><li><strong>Accesses comprehensive context</strong> about the customer and car history, help center articles, and response templates</li><li><strong>Generates review-ready drafts</strong> that service advisors can approve and send with one click</li><li><strong>Automatically performs actions</strong> for 60% of messages to update the order status and send a reply SMS</li></ol><div class="kg-card kg-callout-card kg-callout-card-blue"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">&quot;The Service Advisor agent knows everything about a customer&apos;s diagnostic history, their vehicle, and their past interactions. It&apos;s like having our best service advisor assistant available 24/7, but it never gets tired or overwhelmed.&quot;</em></i><br><br>&#x2014; James Castillo, Co-Founder &amp; Head of Engineering</div></div><p><strong>Agent #2: The Technical Expert for Mechanics</strong></p><p>The second AI agent serves as a master technician, supporting the auto repair shop partners through RepairWise&apos;s messaging system and portal.</p><p>How it works:</p><ol><li><strong>Monitors the proprietary messaging system</strong> for inquiries from repair shop mechanics</li><li><strong>The AI agent examines</strong> the technical question or issue</li><li><strong>It searches connected systems</strong>,<strong> </strong>including repair manuals for specific EV models and the order &amp; customer database</li><li><strong>It generates a comprehensive draft</strong> with repair guidance, diagnostic steps, parts information, or customer communication suggestions</li><li><strong>A human technician reviews</strong> the draft and sends it with a single click</li></ol><div class="kg-card kg-callout-card kg-callout-card-blue"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">&quot;The Technical Expert agent handles incredibly complex questions. It can reference specific repair procedures from our manuals, cross-reference diagnostic codes with known issues.</em></i>&quot;<br><br>&#x2014; James Castillo, Co-Founder &amp; Head of Engineering</div></div><h3 id="the-results-scaling-support-without-scaling-staff">The Results: Scaling Support Without Scaling Staff</h3><p>The impact of RepairWise&apos;s dual-agent system has been transformative for its operations.</p><ul><li><strong>Volume handled:</strong> over 1,000 daily messages now flow through AI-assisted systems, with every single response, whether fully automated or drafted, generated by AI agents working alongside the human team.</li><li><strong>Automation rate:</strong> 60% of customer SMS messages are fully automated, while the remaining 40% benefit from AI-drafted responses that reduce human effort to &quot;review and click send&quot;.</li><li><strong>Response speed:</strong> the 2.85x improvement in median response time means customers get answers faster, leading to higher satisfaction and reduced inquiry volume from follow-up questions.</li><li><strong>Technical accuracy:</strong> by connecting to proprietary repair manuals and order databases, the technical agent provides mechanics with accurate, model-specific guidance that previously required extensive research.</li><li><strong>Team efficiency:</strong> the same 9-person team continues handling increasing volume without burnout. They&apos;ve shifted from being message responders to quality reviewers and complex problem solvers.</li></ul><h3 id="why-this-approach-works">Why This Approach Works</h3><p>RepairWise&apos;s success with AI-powered messaging automation demonstrates several key principles.</p><p><strong>Specialized agents for specialized audiences.</strong> Rather than building one general-purpose agent, RepairWise created two agents optimized for their specific domains. The Service Advisor agent speaks the language of concerned EV owners, while the Technician agent communicates with the precision and depth mechanics require.</p><p><strong>Human-in-the-loop for quality control.</strong> The 60/40 automation split for customer messages and 100% draft generation for technical communications ensures accuracy while maximizing efficiency. This hybrid approach delivers speed without sacrificing the personal touch.</p><p><strong>Production-safe architecture.</strong> By aggregating all data into our scalable open-source <a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com" rel="noreferrer">BemiDB</a> database with built-in replication, RepairWise eliminated any risk of AI queries impacting their core platform. The AI agents can securely retrieve all synchronized data across various knowledge sources with granular permissions and aggregated queries running up to 2000x faster than regular PostgreSQL, while significantly reducing LLM token consumption.</p><h3 id="looking-ahead-revolutionizing-auto-repair">Looking Ahead: Revolutionizing Auto Repair</h3><p>For RepairWise, AI-powered communication automation represents more than just efficiency gains. It&apos;s the foundation for ambitious growth plans in the rapidly expanding EV repair market and investment in making their existing team more effective.</p><p>As RepairWise continues expanding its network of repair shops, adding support for new EV models, and onboarding more customers, their support infrastructure scales automatically.</p><hr><h3 id="ready-to-scale-your-customer-communication-with-ai-agents">Ready to Scale Your Customer Communication with AI Agents?</h3><p>If you&apos;re managing high-volume 24/7 communications and need to scale without proportional headcount growth, AI agents can transform your operations.</p><p><a href="https://gettelio.com/?ref=blog.gettelio.com">Telio</a> integrates with various tools and systems, allowing you to build intelligent automation that understands your business context and scales with your growth.</p>]]></content:encoded></item><item><title><![CDATA[How Rollups Scales Fintech Support: Automating 1,000s of Emails in Front]]></title><description><![CDATA[Rollups scales support with Telio AI agents that handle 100+ daily emails and integrate with Front, Notion, Attio, and Mintlify to draft context-aware replies.]]></description><link>https://blog.gettelio.com/how-rollups-scales-fintech-support/</link><guid isPermaLink="false">695303b832554a0001a75f52</guid><category><![CDATA[Customer Stories]]></category><dc:creator><![CDATA[Evgeny Li]]></dc:creator><pubDate>Tue, 30 Dec 2025 21:51:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2026/01/preview--1-.png" medium="image"/><content:encoded><![CDATA[<h3 id="highlights">Highlights</h3><ul><li><strong>Company:</strong> <a href="https://rollups.com/?ref=blog.gettelio.com" rel="noreferrer">Rollups</a></li><li><strong>Industry:</strong> Financial Technology / Cap Table Management</li><li><strong>Challenge:</strong> Managing thousands of emails while maintaining personalized and accurate responses</li><li><strong>Solution:</strong> Email automation with Telio AI agents integrated with:<ul><li><a href="https://docs.gettelio.com/integrations/front?ref=blog.gettelio.com" rel="noreferrer">Front</a>, shared email inbox</li><li><a href="https://docs.gettelio.com/integrations/attio?ref=blog.gettelio.com" rel="noreferrer">Attio</a>, CRM</li><li><a href="https://docs.gettelio.com/integrations/notion?ref=blog.gettelio.com" rel="noreferrer">Notion</a>, internal knowledge center and style guide</li><li><a href="https://docs.gettelio.com/integrations/web?ref=blog.gettelio.com" rel="noreferrer">Help Center</a>, public-facing website</li></ul></li><li><strong>Results:</strong> Fully automated email draft generation with context-aware responses</li></ul><h3 id="about-rollups">About Rollups</h3><img src="https://blog.gettelio.com/content/images/2026/01/preview--1-.png" alt="How Rollups Scales Fintech Support: Automating 1,000s of Emails in Front"><p>Rollups helps startups and growth-stage companies take control of their equity by consolidating stakeholders into a single vehicle. It is part of AngelList&#x2019;s ecosystem of brands, and it is trusted by over 50,000 investors and companies. Rollups offers two primary solutions: Roll Up Vehicles (RUVs) for raising capital and Consolidation Vehicles (CVs) for cleaning up complex cap tables.</p><h3 id="the-challenge-scaling-support-in-a-complex-domain">The Challenge: Scaling Support in a Complex Domain</h3><p>As Rollups grew, their team received a flood of repetitive questions that prevented them from focusing on complex customer issues.</p><div class="kg-card kg-callout-card kg-callout-card-blue"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">&quot;We were getting the same questions over and over. Questions about how Roll Up Vehicles work, what documents are needed, and other basic support questions. Our team was spending hours each day answering these.&quot;</em></i><br><br>&#x2014; Sumukh Sridhara, Rollups</div></div><p>The numbers told the story:</p><ul><li>Thousands of emails flowing through their system</li><li>A significant portion were repetitive, common questions</li><li>Response times stretched during peak periods</li><li>Team stretched thin between routine and complex inquiries</li></ul><h3 id="the-solution-ai-agents-that-think-like-support-experts">The Solution: AI Agents That Think Like Support Experts</h3><p>Rollups built AI agents using Telio with a strategic approach. Rather than replacing their human support, they built an AI layer that handles routine inquiries automatically with a human-in-the-loop approach.</p><h3 id="how-it-works-in-practice">How It Works in Practice</h3><p>When a customer support email arrives in Front, Telio&apos;s AI agent:</p><ol><li><strong>Reads and analyzes</strong> the customer inquiry</li><li><strong>Searches connected systems</strong> for relevant information: help center content, internal knowledge center and style guide articles, CRM records.</li><li><strong>Drafts a complete response</strong> that matches Rollups&apos; tone and includes accurate, contextual information</li><li><strong>Presents the draft</strong> to the human team for review and one-click sending directly in the Front interface without switching tools or disrupting the workflow.</li></ol><div class="kg-card kg-callout-card kg-callout-card-blue"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">&quot;By automating the email reply drafting process, we&apos;ve eliminated the &apos;blank page&apos; problem for our team. The AI does the research and writing, using our existing help collateral, and our team provides the final human touch.&quot;</em></i><br><br>&#x2014; Sumukh Sridhara, Rollups</div></div><h3 id="the-results-more-time-for-what-matters">The Results: More Time for What Matters</h3><p>The impact on Rollups&apos; support operation has been transformative:</p><ul><li><strong>Volume handled:</strong> Thousands of emails now flow through the AI-assisted system, with all replies drafted automatically.</li><li><strong>Response quality:</strong> Answers tailored to each customer&#x2019;s specific situation, using the same trusted knowledge sources as the human team.</li><li><strong>Team focus:</strong> Support specialists now spend their time on high-value activities like advising on complex consolidations, rather than answering questions that the team has seen before.</li><li><strong>Scalability:</strong> As Rollups continues growing their customer base, their support capacity scales automatically without proportional headcount increases.</li></ul><h3 id="why-this-approach-works">Why This Approach Works</h3><p>Rollups&apos; success with AI-powered support comes down to two key factors.</p><p><strong>Connected knowledge systems</strong>. By integrating Front with their internal knowledge center, style guides, help center, and CRM, the AI agent has access to holistic information, ensuring responses are accurate and aligned with the company&apos;s style.</p><p><strong>Human-in-the-loop design</strong>. The AI generates email drafts that can be reviewed by a human and sent with a single click. This hybrid approach combines the speed of automation with the judgment and nuance of human oversight.</p><h3 id="looking-ahead">Looking Ahead</h3><p>For Rollups, AI-powered email automation represents more than just efficiency gains. It&apos;s about scaling their ability to deliver an exceptional customer experience as they continue to onboard companies at every stage, from pre-seed to Series D+ startups.</p><hr><h3 id="ready-to-transform-your-support-operations">Ready to Transform Your Support Operations?</h3><p>If you&apos;re handling high volumes of repetitive customer inquiries and your team is stretched thin, our AI agents can help.</p><p><a href="https://gettelio.com/?ref=blog.gettelio.com" rel="noreferrer">Telio</a> integrates with various tools and services, allowing you to create intelligent email automation that actually understands your business and customers.</p>]]></content:encoded></item><item><title><![CDATA[Telio Achieves SOC 2 Type II Compliance: Secure 24/7 AI Agents]]></title><description><![CDATA[We’re excited to announce a major milestone in our commitment to data security and trust: Telio is now officially SOC 2 Type II certified.]]></description><link>https://blog.gettelio.com/telio-achieves-soc-2-type-ii-compliance/</link><guid isPermaLink="false">6952b2db32554a0001a75eb4</guid><category><![CDATA[News]]></category><dc:creator><![CDATA[Evgeny Li]]></dc:creator><pubDate>Tue, 14 Oct 2025 16:58:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/12/preview-14.png" medium="image"/><content:encoded><![CDATA[<img src="https://blog.gettelio.com/content/images/2025/12/preview-14.png" alt="Telio Achieves SOC 2 Type II Compliance: Secure 24/7 AI Agents"><p>We&#x2019;re excited to announce a major milestone in our commitment to data security and trust: <strong>Telio is now officially SOC 2 Type II certified</strong>, the gold standard for security compliance in the tech industry.</p><h3 id="what-is-soc-2-type-ii">What is SOC 2 Type II?</h3><p><strong>SOC 2</strong> is an auditing framework developed by the American Institute of Certified Public Accountants that evaluates how well a service provider protects customer data.</p><p>A Type I report only checks whether security controls were correctly designed at a single point in time. A <strong>Type II</strong> report, on the other hand, is much more rigorous, requiring an independent auditor to monitor our systems for several months to demonstrate that our security controls are actually working in practice.</p><p>Telio has independently verified that it meets strict requirements across five Trust Service Criteria:</p><ul><li><strong>Security</strong>: Our systems and infrastructure are protected against unauthorized access, both physical and logical.</li><li><strong>Availability</strong>: Our services operate reliably and are available 24/7 as promised to our customers.</li><li><strong>Processing Integrity</strong>: System processing is complete, valid, accurate, timely, and authorized.</li><li><strong>Confidentiality</strong>: Sensitive information is protected as committed or agreed upon.</li><li><strong>Privacy</strong>: Personal information is collected, used, retained, disclosed, and disposed of in accordance with our privacy policies and applicable regulations.</li></ul><h3 id="why-this-matters-for-our-customers"><strong>Why This Matters </strong>for Our Customers</h3><p>As an <strong>AI-powered platform that handles thousands of customer interactions</strong> through voice calls, text messages, and emails, we understand the critical importance of data security and privacy. Our customers trust us with sensitive business information and customer data every single day, and this certification validates that trust.</p><p>For businesses considering Telio for their customer support needs, SOC 2 Type II provides additional confidence that we take data protection seriously. Customers can request our SOC 2 report directly as part of due diligence or procurement reviews.</p><h3 id="what-this-means-moving-forward">What This Means Moving Forward</h3><p>Security isn&apos;t a &quot;one-and-done&quot; checkbox for us, it&#x2019;s part of our DNA. Being HIPAA-compliant and now also achieving SOC 2 Type II is just one milestone in our long-term roadmap to provide the most secure and human-like AI agents.</p><p><strong>We will continue to maintain and enhance our security practices</strong>, undergo regular audits, and stay ahead of emerging threats and industry standards. Learn more about our security at <a href="https://gettelio.com/security?ref=blog.gettelio.com" rel="noreferrer">gettelio.com/security</a> and compliance at <a href="https://trust.gettelio.com/?ref=blog.gettelio.com" rel="noreferrer">trust.gettelio.com</a>.</p><hr><p><strong>Ready to experience secure, 24/7 AI-powered customer support?</strong> </p><p>Visit <a href="https://gettelio.com/?ref=blog.gettelio.com" rel="noreferrer">gettelio.com</a> to learn more about how Telio can transform your customer service while keeping your data safe and secure.</p>]]></content:encoded></item><item><title><![CDATA[Product Analytics Queries Without Database Meltdown]]></title><description><![CDATA[Stop product dashboard meltdowns: the story of scaling Postgres, going to a warehouse, and evaluating more modern solutions.]]></description><link>https://blog.gettelio.com/product-analytics-db/</link><guid isPermaLink="false">68ae7578e3a2820001805d93</guid><category><![CDATA[Engineering]]></category><dc:creator><![CDATA[Arjun Lall]]></dc:creator><pubDate>Fri, 02 May 2025 15:54:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/08/image-762-min.png" medium="image"/><content:encoded><![CDATA[<img src="https://blog.gettelio.com/content/images/2025/08/image-762-min.png" alt="Product Analytics Queries Without Database Meltdown"><p>I&apos;ve personally watched this journey unfold &#x2014; what starts as shipping a simple dashboard can eventually lead to temporary patches, complexity creep, and full rearchitectures just to avoid killing the transactional database. This&apos;ll walk you through the journey I&apos;ve seen a team take many years ago at a fast-scaling startup and help you short-circuit this cycle, saving months of discovery pain along the way.</p><h2 id="meltdown">Meltdown</h2><p>One of our biggest international customers launched a flash sale that drove 50x their normal transaction volume. At 3 AM, our on-call was paged and Slack lit up with alerts. &quot;DATABASE CPU: CRITICAL.&quot; &quot;API LATENCY: CRITICAL.&quot; &quot;CHECKOUT FAILURE RATE: CRITICAL.&quot;. This customer eventually churned due to the lost revenue from the incident. The culprit? Constant refreshing of their dashboard to track sale success.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/05/image-760--1-.jpg" class="kg-image" alt="Product Analytics Queries Without Database Meltdown" loading="lazy" width="2446" height="1216"></figure><p>Let&#x2019;s take a closer look at the query:</p><pre><code class="language-sql">SELECT 
  date_trunc(&apos;month&apos;, purchase_date) as month,
  customer_region,
  AVG(customer_age) as avg_age,
  SUM(purchase_amount) as total_revenue,
  COUNT(DISTINCT customer_id) as unique_customers
FROM purchases
JOIN customers ON purchases.customer_id = customers.id
WHERE purchase_date &gt; NOW() - INTERVAL &apos;12 months&apos;
GROUP BY date_trunc(&apos;month&apos;, purchase_date), customer_region
ORDER BY month, total_revenue DESC;
</code></pre><p>When this query executes in a transactional database, several resource-intensive operations occur:</p><ol><li><strong>Full table scan</strong>: Without proper indexes, the database reads every row in both tables to find matching records.</li><li><strong>Join operation</strong>: The database builds an in-memory hash table for the join, which can consume significant RAM when joining large tables.</li><li><strong>Grouping operation</strong>: Another in-memory hash table gets built to track each unique group (month + region).</li><li><strong>Distinct counting</strong>: The <code>COUNT(DISTINCT customer_id)</code> requires tracking all unique customer IDs seen so far, adding more memory pressure.</li><li><strong>Sorting</strong>: Finally, the results get sorted by month and revenue, potentially needing disk if the result set is large enough.</li></ol><p>In transactional databases like PostgreSQL, Multiversion concurrency control (MVCC) avoids locking by letting each query see a consistent snapshot of the data. This works by checking the visibility of each row version at read time. But for large analytical queries&#x2014;especially those that scan millions of rows and compute aggregates&#x2014;these per-row checks can spike CPU usage. Add operations like sorting, grouping, and COUNT(DISTINCT), and memory and disk I/O can quickly become bottlenecks. Even though MVCC avoids blocking other queries directly, a single heavyweight query can still monopolize enough resources to severely impact overall performance.</p><h2 id="the-%E2%80%98just-scale-it%E2%80%99-phase">The &#x2018;Just Scale It&#x2019; Phase</h2><p>After turning off feature flags or killing queries, the natural first step was to optimize the existing system.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/Just-use-Postgres-3.png" class="kg-image" alt="Product Analytics Queries Without Database Meltdown" loading="lazy"></figure><h2 id="indexes">Indexes</h2><p>The quickest win was to add indexes to speed up the query execution:</p><pre><code class="language-sql">CREATE INDEX idx_purchases_date 
ON purchases(purchase_date);

CREATE INDEX idx_purchases_customer_region_date 
ON purchases(customer_region, purchase_date);

CREATE INDEX idx_customers_region 
ON customers(region);
</code></pre><p>Indexes create specialized data structures (typically B-trees) that organize record locations in a predefined order. They contain key values and pointers to the actual data rows, allowing the database to locate records matching specific criteria without sequentially scanning the entire table. For our query, these indexes help the database quickly identify relevant purchase records by date range, find matching customer regions, and efficiently join the tables together.</p><p>Performance improved drastically at first, dropping the dashboard&#x2019;s 99th percentile load time to about 10 seconds. But the indexes had some downsides:</p><ol><li><strong>Write performance degradation</strong>: Every write operation has to now also update each index. As more indexes get added for new dashboard charts, this overhead grows linearly, and slows down critical writes.</li><li><strong>Bloat</strong>: Indexes can consume lots of disk space, increasing infra and maintenance costs. I&#x2019;ve seen production databases with indexes more than double the size of the actual data.</li><li><strong>Limited gains</strong>: Indexes help with filtering but not with aggregations like COUNT(DISTINCT).</li></ol><h3 id="read-replica">Read Replica</h3><p>Indexes gave us some breathing room, but analytical and transactional workloads were still competing for the same resources. I pointed out to the team that we had a read replica that was heavily underutilized - sitting at just 12% CPU utilization and low disk IOPS.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/05/Group-15991--1-.jpg" class="kg-image" alt="Product Analytics Queries Without Database Meltdown" loading="lazy" width="1671" height="1385"></figure><p>Routing all dashboard queries to the read replica worked surprisingly well, with the primary database seeing a 15% CPU reduction and dashboard p99 latency improving by 30%.</p><p>Since read replicas maintain exact copies of the primary, any index the team needed to add for dashboard charts still had to exist on the primary. This meant write performance on the primary was still degraded. There were still improvements to make since the dashboard could still take a few seconds to load.</p><h3 id="materialized-views">Materialized Views</h3><p>The team next needed a way to pre-compute dashboard query results:</p><pre><code class="language-sql">CREATE MATERIALIZED VIEW monthly_regional_metrics AS
SELECT
  date_trunc(&apos;month&apos;, purchase_date) as month,
  customer_region,
  AVG(customer_age) as avg_age,
  SUM(purchase_amount) as total_revenue,
  COUNT(DISTINCT customer_id) as unique_customers
FROM purchases
JOIN customers ON purchases.customer_id = customers.id
GROUP BY date_trunc(&apos;month&apos;, purchase_date), customer_region;

</code></pre><p>Materialized views pre-compute and store query results, turning slow queries into simple table lookups. This successfully made the dashboard p99 latency drop 50%, but introduced the new challenge of the operational complexity of maintaining fresh views as data scale grew. </p><h3 id="table-partitioning">Table Partitioning</h3><p>The team didn&#x2019;t actually partition data, but I&#x2019;ll list it here to be thorough:</p><pre><code class="language-sql">CREATE TABLE purchases (
  id SERIAL,
  customer_id INTEGER,
  purchase_date TIMESTAMP,
  purchase_amount DECIMAL(10,2)
) PARTITION BY RANGE (purchase_date);

CREATE TABLE purchases_2019_q1 PARTITION OF purchases
  FOR VALUES FROM (&apos;2019-01-01&apos;) TO (&apos;2019-04-01&apos;);

CREATE TABLE purchases_2019_q2 PARTITION OF purchases
  FOR VALUES FROM (&apos;2019-04-01&apos;) TO (&apos;2019-07-01&apos;);

</code></pre><p>Partitioning divides large tables into smaller physical chunks. When a query specifies a date range, the database can scan only relevant partitions instead of the entire table.</p><h3 id="server-scaling">Server Scaling</h3><p>As our data volume continued to grow, the team considered a few other scaling approaches:</p><p><strong>Vertical Scaling</strong>: The most straightforward of which is upgrading the hardware and giving it more resources. This continues to buy some time.</p><p><strong>Horizontal Scaling</strong>: The most complex approach is distributing data across multiple servers called shards. This means implementing application-level logic to determine which shard to access when reading or writing data.</p><p>The approaches that were tried helped temporarily but didn&apos;t address the fundamental architectural issue, and the mismatch was becoming more apparent as the company scaled.</p><h2 id="the-architectural-mismatch">The Architectural Mismatch</h2><p>Transactional databases such as PostgreSQL excel at:</p><ul><li>Quick point lookups and small range scans</li><li>High concurrency for many small operations</li><li>Strong data consistency guarantees</li></ul><p>Analytical workloads like a dashboard have fundamentally different characteristics:</p><ul><li>They scan across table columns rather than accessing specific records</li><li>They aggregate data across millions of rows</li><li>They often need to access only a few columns from wide tables</li></ul><p>When our dashboard query needed just <code>purchase_date</code> and <code>purchase_amount</code> from a 20-column table with say a million rows, PostgreSQL still reads all 20 million values. This massive I/O inefficiency is why our CPU and disk metrics kept hitting the ceiling.</p><p>That&#x2019;s why databases made for analytical workloads store data in columnar format.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/05/image-1.png" class="kg-image" alt="Product Analytics Queries Without Database Meltdown" loading="lazy" width="2604" height="1326"></figure><p>This columnar storage provides massive benefits for analytical queries:</p><ul><li>Only needed columns are pulled from disk</li><li>Similar data in columns compresses better</li><li>Operations can process batches of the same data type simultaneously</li><li>Filtering happens before data is loaded into memory</li></ul><p>Eventually, the temporary dashboard scaling workarounds weren&#x2019;t enough and the team needed to move to an analytics optimized architecture.</p><h2 id="the-data-warehouse">The Data Warehouse</h2><p>Before all scaling options were fully exhausted, the company had invested in setting up Snowflake with proper data pipelines, and all product teams were encouraged to leverage this new analytics infrastructure.</p><p>With the help from the data engineering team, they implemented a traditional extract-transform-load (ETL) architecture:</p><ol><li><strong>Extraction</strong>: Airflow DAGs would run scheduled jobs that pulled data from our production PostgreSQL database</li><li><strong>Transformation</strong>: The extracted data underwent transformation steps before loading - cleaning values, applying business rules, and restructuring data.</li><li><strong>Loading</strong>: Finally, the transformed data was loaded into Snowflake tables optimized for analytical workloads.</li></ol><p>For the dashboard to display this processed data, the team implemented a reverse ETL process that completed a data round-trip: production DB &#x2192; Snowflake &#x2192; back to production DB, but now with pre-computed aggregations ready for fast querying. This dropped dashboard load times to milliseconds!</p><h3 id="the-batch-reality-vs-real-time-promise">The Batch Reality vs. Real-Time Promise</h3><p>The team initially accepted a 24-hour refresh cycle for the dashboard metrics, comfortable with this temporary limitation based on the data team&apos;s roadmap promising real-time capabilities &quot;soon.&quot; Our dashboard would clearly display: &quot;Data updated daily at 4:00 AM UTC.&quot;</p><p>The company&apos;s leadership was understanding about this refresh cycle during the transition, but customers accustomed to seeing their latest transactions reflected immediately were confused by the sudden shift to day-old data. This created an unexpected support burden.</p><h3 id="unexpected-complexities">Unexpected Complexities</h3><p>What started as a simple architectural improvement quickly revealed hidden operational costs:</p><p><strong>Pipeline fragility</strong>: Weekly pipeline failures would result in cryptic error messages that the product engineering team struggled to troubleshoot, often requiring escalation to the busy data engineering team.</p><p><strong>Dependency challenges</strong>: The promised real-time capabilities kept getting delayed as the data team discovered the true complexity of implementing Change Data Capture and Kafka streaming infrastructure.</p><p>While this architecture was theoretically correct, the company underestimated the operational overhead. Our mid-sized startup didn&apos;t actually have the petabyte-scale data that would justify the complexity, and our engineering team lacked a lot of data engineering talent. The organizational impact was equally problematic - our product team became dependent on an overextended data team, creating unclear ownership and delayed issue resolution.</p><p>For a company with a large data engineering org and massive data volumes, this architecture makes perfect sense. But for our relatively straightforward analytics needs, we had overcomplicated our infrastructure without considering the long-term operational and organizational costs.</p><h2 id="modern-analytical-solutions">Modern Analytical Solutions</h2><p>Today, teams have simpler options available available. Several analytical databases can bridge the gap between transactional and analytical workloads without as much overhead as a Snowflake.</p><h3 id="duckdb-the-embedded-analytical-engine">DuckDB: The Embedded Analytical Engine</h3><p>DuckDB has emerged as a powerful columnar analytical database that can be embedded directly into applications - essentially SQLite for analytics:</p><ul><li><strong>Embedded architecture</strong>: Runs inside a host process with bindings for languages like Python, with the ability to directly place data into structures like NumPy arrays</li><li><strong>Columnar-vectorized execution</strong>: Uses a columnar-vectorized query execution engine, where queries are still interpreted, but a large batch of values (a &quot;vector&quot;) are processed in one operation</li><li><strong>Interoperability</strong>: Seamlessly works with the broader data science ecosystem</li></ul><p>DuckDB is ideal for data scientists and analysts who need fast analytical capabilities integrated directly into their data workflows without the overhead of setting up a separate database server. It excels at scenarios like exploratory data analysis, ad-hoc queries, and processing moderately large datasets locally.</p><h3 id="clickhouse-high-performance-analytical-database">ClickHouse: High-Performance Analytical Database</h3><p>Originally developed at Yandex for their analytics, Clickhouse now powers analytics at companies like Cloudflare and Uber. It&apos;s is a column-oriented DBMS built for analytical workloads at extreme scale:</p><ul><li><strong>Distributed architecture</strong>: Designed to scale horizontally across clusters of commodity hardware</li><li><strong>Vectorized execution</strong>: Processes data in parallel using SIMD instructions for maximum performance</li><li><strong>Real-time ingestion</strong>: Can ingest large amounts of data with the Clickpipes integration</li></ul><p>ClickHouse shines when dealing with massive datasets where query performance is critical. </p><h3 id="bemidb-the-analytical-read-replica">BemiDB: The Analytical Read Replica</h3><p><em>Disclaimer: I&#x2019;m a BemiDB open source contributor.</em></p><p>BemiDB is a data warehouse that can be used as a PostgreSQL read replica optimized for analytics:</p><ol><li><strong>Automatic replication</strong>: Data syncs from Postgres with no pipeline code</li><li><strong>Open columnar format</strong>: Stores data either on a local file system or on S3-compatible object storage</li><li><strong>Full Postgres compatibility</strong>: Uses the same SQL syntax as PostgreSQL </li></ol><p>BemiDB is ideal for teams trying to scale for analytics while staying close to PostgreSQL. The use cases it shines would be in-app analytics, BI queries, and centralizing PostgreSQL data.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/BemiDB-2.png" class="kg-image" alt="Product Analytics Queries Without Database Meltdown" loading="lazy"></figure><h3 id="materialize-streaming-sql-for-real-time-analytics">Materialize: Streaming SQL for Real-Time Analytics</h3><p>Materialize takes a different approach by focusing on real-time analytics using a streaming architecture:</p><ol><li><strong>Incremental view maintenance</strong>: Updates query results as data changes</li><li><strong>Streaming architecture</strong>: Processes data changes in real-time as they occur</li><li><strong>Differential dataflow</strong>: Uses sophisticated algorithms to minimize computation</li></ol><p>Materialize shines for use cases requiring real-time analytics on constantly changing data, including fraud detection, real-time notifications, and operational dashboards. </p><h3 id="choosing-the-right-solution-for-your-needs">Choosing the Right Solution for Your Needs</h3><p>The best choice would depend on your specific needs:</p>
<!--kg-card-begin: html-->
<table>
<thead>
<tr>
<th>Solution</th>
<th>Best For</th>
<th>Key Advantage</th>
</tr>
</thead>
<tbody>
<tr>
<td>DuckDB</td>
<td>Embedded analytics, local analysis</td>
<td>Lightweight, no server needed</td>
</tr>
<tr>
<td>ClickHouse</td>
<td>Massive scale analytics</td>
<td>Extreme query performance</td>
</tr>
<tr>
<td>BemiDB</td>
<td>Simplicity, Postgres compatibility</td>
<td>Single Docker image, direct Postgres replication</td>
</tr>
<tr>
<td>Materialize</td>
<td>Real-time streaming analytics</td>
<td>Incremental view maintenance</td>
</tr>
</tbody>
</table>
<!--kg-card-end: html-->
<h2 id="a-simpler-path-forward">A Simpler Path Forward</h2><p>Instead of the painful progression most companies follow:</p><ol><li>Query production directly until it breaks</li><li>Add indexes until writes become slow</li><li>Build complex ETL pipelines to data warehouses</li><li>Create reverse ETL to bring data back to the application</li></ol><p>Consider this simpler approach:</p><ol><li>Keep transactional workloads on your production database</li><li>Use a columnar analytical engine for reports and dashboards</li><li>Only move to a full data warehouse when you truly need to integrate many different data sources or process petabytes of data</li></ol><p>This approach would&#x2019;ve saved that team months of trouble. The mid-sized startup needed operational simplicity and the team first and foremost wanted to eliminate late-night pages, pipeline debugging, and the performance dance of trying to make the transactional database not meltdown. The right solution for you will depend on your specific requirements around data volume, real-time needs, and data org maturity.</p><hr><p><em>There are several open-source solutions mentioned worth exploring, including </em><a href="https://github.com/duckdb/duckdb?ref=blog.gettelio.com"><em>DuckDB</em></a><em>, </em><a href="https://github.com/ClickHouse/ClickHouse?ref=blog.gettelio.com"><em>ClickHouse</em></a><em>, </em><a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com"><em>BemiDB</em></a><em>, and </em><a href="https://github.com/MaterializeInc/materialize?ref=blog.gettelio.com"><em>Materialize</em></a><em>. Each has different trade-offs that might make it the right fit for your specific needs.</em></p>]]></content:encoded></item><item><title><![CDATA[Proprietary Data Analytics Platforms are a Trap]]></title><description><![CDATA[Proprietary data analytics platforms are a trap—because of high costs, vendor lock-in, and unnecessary complexity. Modern tech makes them obsolete.]]></description><link>https://blog.gettelio.com/cloud-data-scam/</link><guid isPermaLink="false">68ae74fde3a2820001805d7f</guid><category><![CDATA[Engineering]]></category><dc:creator><![CDATA[Arjun Lall]]></dc:creator><pubDate>Fri, 14 Mar 2025 12:50:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/08/Group-15987.png" medium="image"/><content:encoded><![CDATA[<img src="https://blog.gettelio.com/content/images/2025/08/Group-15987.png" alt="Proprietary Data Analytics Platforms are a Trap"><p>When analytical queries and workloads scale out of transactional databases like Postgres, most companies default to a cloud vendor like Snowflake, connecting and transforming their data in the process. Proprietary data platforms are the industry standard here. We have one of the fastest growing open source database projects, and every advisor tells us to &#x201C;build a cloud platform&#x201D;. It&#x2019;s what the industry expects and also the easiest revenue model.</p><p>But we&#x2019;re taking a contrarian path. I&#x2019;ll explain why we think it&#x2019;s time to rethink data analytics and why current platforms are a trap&#x2014;namely because of high costs, vendor lock-in, and unnecessary complexity.</p><h3 id="complexity"><strong>Complexity</strong></h3><p>Cloud data analytics is a scam, <em>today</em>. Snowflake and other old proprietary cloud data warehouses were built on the assumption that distributed clusters, data movement orchestration, and constant infrastructure maintenance are necessary for analytics at scale. Let&#x2019;s revisit if this Hadoop-era way of thinking still holds true with today&#x2019;s technical shifts:</p><ul><li>Hardware advances: the raw power of a single server today is orders of magnitude more powerful than what it used to be. A single machine with 64 CPU cores can easily scan 1TB of data. And since <a href="https://motherduck.com/blog/redshift-files-hunt-for-big-data/?ref=blog.gettelio.com">98% of Redshift queries</a> on &gt;10TB datasets scan less than 1TB, distributed systems and CAP theorem tradeoffs aren&#x2019;t necessary for analytical workloads anymore.</li><li>Embedded analytics engines: projects like <a href="https://github.com/duckdb/duckdb?ref=blog.gettelio.com">DuckDB</a> show you can quickly scan and analyze billions of rows of data on even your laptop. It runs fully in-process, sharing memory with your application, which means there&#x2019;s no separate database server or network overhead.</li><li>Separation of compute and storage: with accessible durable object storage like S3, you can store massive datasets cheaply without needing always-on infrastructure. You only spin up compute for the data you actually query, which reduces overhead&#x2014;especially since most queries only scan a fraction of what&#x2019;s stored.</li></ul><p>Together, these shifts enable a simpler architecture &#x2014; you can self-host without the bloat and cost of managing complex infrastructure. For example, our <a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com">BemiDB</a> project is just a single Docker image. It embeds DuckDB, automatically syncs data from Postgres databases to an analytics optimized S3 bucket, and leverages Postgres&#x2019; wire and query protocol to speak Postgres. In practice, this means spinning up a fast and scalable data warehouse can be as simple as:</p><pre><code class="language-bash">&gt; curl -sSL https://raw.githubusercontent.com/BemiHQ/BemiDB/refs/heads/main/scripts/install.sh | bash

&gt; ./bemidb sync --pg-sync-interval 10m --pg-database-url postgres://&lt;user&gt;:&lt;pass&gt;@&lt;host&gt;:5432/&lt;dbname&gt;

&gt; ./bemidb start

&gt; psql postgres://localhost:54321/bemidb
bemidb=&gt; SELECT country, COUNT(*) FROM users GROUP BY country;</code></pre><h3 id="the-lock-in-tax"><strong>The lock in tax</strong></h3><p>Snowflake and similar vendors store data in proprietary formats, meaning you&#x2019;re effectively stuck and unable to leave. Especially as you scale, this means exorbitant fees and no flexibility of using any other tools and services with your data. This is in stark contrast to modern open table formats like <a href="https://github.com/apache/iceberg?ref=blog.gettelio.com">Apache Iceberg</a> which are now becoming the standard.</p><p>Iceberg provides a consistent structure for your data on object storage that keeps it readable by all data tools and services. Paired with open-standard columnar files like <a href="https://github.com/apache/parquet-format?ref=blog.gettelio.com">Apache Parquet</a>, your data remains fully portable.</p><p>Additionally, proprietary cloud query engines limit where you can run and optimize workloads. Open source query engines let you deploy on VMs, bare metal, or containers&#x2014;whatever fits your performance and security needs.</p><p>Embracing open formats and open source helps avoid the lock-in tax and keeps your data truly yours.</p><h3 id="monetization"><strong>Monetization</strong></h3><p>Data teams end up paying well over $1,000 per month on cloud data warehouses&#x2014;often climbing to tens of thousands when you factor in ETL pipelines, data egress, and infrastructure overhead. By storing data in cheap, durable object storage within the same region and running queries on a modest VM, you can cut those monthly costs down to hundreds.</p><figure class="kg-card kg-image-card"><img src="https://media0.giphy.com/media/v1.Y2lkPTc5MGI3NjExZW5tbDFsc3g3cjIzaHM4N2g4czAyb3VsdTkzdWRpbGhlMXg2bnc4aiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/987p6CfE8uzSg/giphy.gif" class="kg-image" alt="Proprietary Data Analytics Platforms are a Trap" loading="lazy"></figure><p>We&#x2019;re a venture backed startup that&#x2019;s after profit, but our mission is to simplify data, and we encourage everyone to use our open source and self-host. Not everything can be packaged into a single Docker container, so we charge for support and extra features that require integrating additional components.</p><h3 id="give-self-hosted-a-try"><strong>Give self hosted a try</strong></h3><p>The exorbitant cloud costs and lock-in are worth it when the alternative is complex infrastructure to build or maintain. Unless you&#x2019;re in the <a href="https://motherduck.com/blog/big-data-is-dead/?ref=blog.gettelio.com">big data one percent</a>, this isn&#x2019;t the case anymore for data analytics.</p><p><a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com"><em>Check out our GitHub repo</em></a><em> and give BemiDB a star! We&#x2019;re always pushing to make data simpler and more open.</em></p>]]></content:encoded></item><item><title><![CDATA[Data Analytics with PostgreSQL: The Ultimate Guide]]></title><description><![CDATA[In this blog post, we will compare the main techniques and approaches for running data analytics with PostgreSQL.]]></description><link>https://blog.gettelio.com/analytics-with-postgresql/</link><guid isPermaLink="false">68ae7421e3a2820001805d6d</guid><category><![CDATA[Engineering]]></category><category><![CDATA[Guide]]></category><dc:creator><![CDATA[Evgeny Li]]></dc:creator><pubDate>Mon, 10 Feb 2025 14:51:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/08/Analytics-with-Postgres-4.png" medium="image"/><content:encoded><![CDATA[<img src="https://blog.gettelio.com/content/images/2025/08/Analytics-with-Postgres-4.png" alt="Data Analytics with PostgreSQL: The Ultimate Guide"><p><strong>TL;DR</strong></p><p>In this blog post, we will compare the main techniques and approaches for running data analytics with PostgreSQL:</p><ol><li><strong>&quot;Just use Postgres&quot; and Scale It</strong><ol><li>Indexes: choosing index types, multicolumn/partial/expression indexes</li><li>Materialized Views: denormalization</li><li>Table Partitioning: declarative partitioning</li><li>Server Scaling: read replica, vertical scaling, horizontal sharding</li></ol></li><li><strong>Install PostgreSQL Extensions</strong><ol><li>Foreign Data Wrappers: columnar storage formats, Parquet, ETL</li><li>Analytics Query Engines: vectorized execution, parallel execution on GPU, DuckDB</li><li>Super Extensions: TimescaleDB and Citus</li></ol></li><li><strong>Integrate with Analytics Databases</strong><ol><li>BemiDB: read replica, vectorized engine and columnar storage, open table formats, Iceberg</li><li>ClickHouse: PostgreSQL wire protocol, vectorized engine and columnar storage, logical replication</li></ol></li><li><strong>Use Proprietary Solutions</strong><ol><li>Google Cloud AlloyDB: hybrid transactional and analytical workloads</li><li>EDB Analytics Accelerator: proprietary extension on PostgreSQL with replication</li><li>Crunchy Data Warehouse: proprietary extensions on PostgreSQL</li><li>Firebolt: forked ClickHouse with PostgreSQL dialect</li></ol></li></ol><p>Each of them has their pros and cons, so we will take a look at them and see which one may be the best for your use case.</p><div class="kg-card kg-callout-card kg-callout-card-grey"><div class="kg-callout-text"><i><em class="italic" style="white-space: pre-wrap;">The PostgreSQL ecosystem has matured and become capable of dealing with analytical workloads in recent years. There is no need to bring heavy big-data tools like Apache Spark or build data pipelines with Kafka anymore.</em></i></div></div><hr><h2 id="just-use-postgres-and-scale-it"><strong>&quot;Just use Postgres&quot; and Scale It</strong></h2><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/Just-use-Postgres-3.png" class="kg-image" alt="Data Analytics with PostgreSQL: The Ultimate Guide" loading="lazy" width="2848" height="1640"></figure><p>PostgreSQL is a highly flexible and powerful database. It simplifies your data stack by serving as a single transactional (OLTP) engine for a range of purposes&#x2014;including full-text search, queueing, caching, and event streaming&#x2014;while also being tunable for analytics (OLAP) queries. Below are the main built-in features.</p><p><a href="https://www.postgresql.org/docs/current/indexes.html?ref=blog.gettelio.com" rel="noreferrer"><strong>Indexes</strong></a></p><p>To identify slow queries and their bottlenecks, we can use PostgreSQL&apos;s <code>EXPLAIN ANALYZE</code> which will show the sequential and index scans. Then, depending on the data structure and queries, you can choose an appropriate index type such as:</p><ul><li>B-tree for equality <code>=</code> and range conditions like <code>&gt;</code> or <code>&lt;</code></li><li>GIN for composite values like <code>JSONB</code> or <code>ARRAY</code></li></ul><p>See the full list of <a href="https://www.postgresql.org/docs/current/indexes-types.html?ref=blog.gettelio.com" rel="noreferrer">supported PostgreSQL index types</a> and the optimized operators that work with them.</p><p>You can also create specialized indexes:</p><ul><li>Multicolumn index (for columns frequently used together in queries): </li></ul><pre><code class="language-sql">CREATE INDEX index_name ON table (column1, column2);</code></pre><ul><li>Partial index<strong> </strong>(for frequent filtering a subset of rows):</li></ul><pre><code class="language-sql">CREATE INDEX index_name ON table (column)
  WHERE column &gt; 1 AND column &lt; 1000;</code></pre><ul><li>Expression index (for computed expressions or functions):</li></ul><pre><code class="language-sql">CREATE INDEX index_name ON table (LOWER(column));</code></pre><p><a href="https://www.postgresql.org/docs/current/rules-materializedviews.html?ref=blog.gettelio.com" rel="noreferrer"><strong>Materialized Views</strong></a></p><p>Denormalization is a strategy that allows improving read performance at the expense of adding redundant copies of data, similarly to a cache. One example is the use of PostgreSQL materialized views that can pre-compute and store query results in a table-like form for frequently accessed data.</p><figure class="kg-card kg-code-card"><pre><code class="language-sql">CREATE MATERIALIZED VIEW summary_sales AS
  SELECT seller_id, invoice_date, sum(invoice_amount) AS sales_amount
  FROM invoices
  WHERE invoice_date &lt; CURRENT_DATE
  GROUP BY seller_id, invoice_date;</code></pre><figcaption><p><span style="white-space: pre-wrap;">Creating a materialized view</span></p></figcaption></figure><p>PostgreSQL also allows adding indexes on materialized views and querying them like it is a regular table:</p><figure class="kg-card kg-code-card"><pre><code class="language-sql">CREATE INDEX summary_sales_index
  ON summary_sales (seller_id, invoice_date);

SELECT * FROM summary_sales
  WHERE seller_id = 1 AND invoice_date = &apos;2025-01-01&apos;</code></pre><figcaption><p><span style="white-space: pre-wrap;">Creating an index and querying a materialized view</span></p></figcaption></figure><p><a href="https://www.postgresql.org/docs/current/ddl-partitioning.html?ref=blog.gettelio.com" rel="noreferrer"><strong>Table Partitioning</strong></a></p><p>Partitioning is a technique that allows splitting a large table into smaller physical ones called partitions. This helps improve query performance by scanning only relevant partitions instead of the entire large table.</p><figure class="kg-card kg-code-card"><pre><code class="language-sql">CREATE TABLE invoices (issued_on DATE NOT NULL)
  PARTITION BY RANGE (issued_on);

CREATE TABLE invoices_2025_01 PARTITION OF invoices
    FOR VALUES FROM (&apos;2025-01-01&apos;) TO (&apos;2025-02-01&apos;);

CREATE TABLE invoices_2025_02 PARTITION OF invoices
    FOR VALUES FROM (&apos;2025-02-01&apos;) TO (&apos;2025-03-01&apos;);
</code></pre><figcaption><p><span style="white-space: pre-wrap;">Declarative partitioning of a table by range</span></p></figcaption></figure><p>Native PostgreSQL declarative partitioning treats the partitioned table like a &#x201C;virtual&#x201D; one that delegates reads/writes to the underlying partitions:</p><figure class="kg-card kg-code-card"><pre><code class="language-sql">-- Stores the row into invoices_2025_01
INSERT INTO invoices (issued_on) VALUES (&apos;2025-01-31&apos;);
-- Stores the row into invoices_2025_02
INSERT INTO invoices (issued_on) VALUES (&apos;2025-02-01&apos;);

-- Selects the rows from invoices_2025_01
SELECT * FROM invoices WHERE issued_on = &apos;2025-01-31&apos;</code></pre><figcaption><p><span style="white-space: pre-wrap;">Writing and reading a partitioned table</span></p></figcaption></figure><p><strong>Server Scaling</strong></p><p>There are a few server scaling options available for PostgreSQL when it comes to running analytical workloads.</p><ul><li>Read replica. To offload read queries from the primary server, you can set up one or more PostgreSQL read replicas.</li><li>Vertical scaling. This is the most straightforward approach. When a server can&apos;t handle the load, consider upgrading the hardware and giving it more resources. </li><li>Horizontal scaling. Another option is to use sharding by distributing data across multiple servers called shards. For example, you can implement application-level logic to determine which shard to access when reading or writing data.</li></ul><h3 id="postgresql-pros">PostgreSQL Pros</h3><ul><li>Powerful general-purpose database transactional database (OLTP) that can handle basic analytical workloads (OLAP) out of the box.</li><li>&quot;Just use Postgres&quot; allows keeping the data stack as simple as possible without adding additional tools and services.</li></ul><h3 id="postgresql-cons">PostgreSQL Cons</h3><ul><li>Creating indexes tailored for specific queries negatively impacts the &quot;write&quot; performance for transactional queries and resource usage.</li><li>Materialized views as a &quot;cache&quot; require manual maintenance and can become increasingly slow to refresh as the data volume grows.</li><li>Table partitioning is a leaky abstraction that doesn&apos;t perfectly encapsulate the details and doesn&apos;t play nice with triggers, constraints, etc.</li><li>Scaling up servers often is a short-term solution that buys some time but doesn&apos;t solve the underlying performance issues.</li><li>Horizontal scaling using sharding significantly increases the complexity of the database architecture and introduces engineering and operational overhead.</li><li>Continuously spending a lot of engineering resource to gain meaningful long-term performance improvements.</li><li>Further tuning and optimization may not be possible if executing various ad-hoc analytical queries.</li></ul><hr><h2 id="install-postgresql-extensions"><strong>Install PostgreSQL Extensions</strong></h2><p>PostgreSQL has a rich ecosystem of extensions that enhance its functionality in different areas, including data analytics. There are a few different approaches that these extensions take, which can be grouped into the categories described below.</p><p><strong>Foreign Data Wrappers</strong></p><p>One of the most popular data storage file formats used in analytics is Parquet. Think of it as &quot;CSV on steroids for analytics&quot;. Here are its key features:</p><ul><li>Columnar storage format with data organized by columns rather than rows</li><li>Excellent compression due to storing similar data together in columns</li><li>Strongly typed schema with explicitly defined types for each column</li></ul><p>The common pattern is extracting data from PostgreSQL (or other data sources), transforming it if necessary, and loading it, for example, in Parquet format to S3. This extract, transform, and load process is called ETL.</p><p>To query Parquet files using PostgreSQL, you can use foreign data wrappers (FDW). This is a mechanism that allows to access data stored outside a database as it is stored in a local table. The most popular PostgreSQL extensions for Parquet are:</p><ul><li><a href="https://github.com/adjust/parquet_fdw?ref=blog.gettelio.com" rel="noreferrer">parquet_fdw</a> for reading Parquet files from a local file system</li><li><a href="https://github.com/pgspider/parquet_s3_fdw?ref=blog.gettelio.com" rel="noreferrer">parquet_s3_fdw</a> for reading Parquet files from S3-compatible object storage</li></ul><p>Here is an example how these foreign data wrappers can be used:</p><figure class="kg-card kg-code-card"><pre><code class="language-sql">-- Set up the extension
CREATE EXTENSION parquet_s3_fdw;
CREATE SERVER parquet_s3_server FOREIGN DATA WRAPPER parquet_s3_fdw;

-- Create a foreign table
CREATE FOREIGN TABLE invoices (issued_on DATE NOT NULL)
  SERVER parquet_s3_server
  OPTIONS (filename &apos;s3://bucket/dir/invoices.parquet&apos;);

-- Querying data from a Parquet file
SELECT issued_on FROM invoices;</code></pre><figcaption><p><span style="white-space: pre-wrap;">Querying Parquet data using a foreign data wrapper</span></p></figcaption></figure><p><strong>Analytics Query Engines</strong></p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/Postgres-extensions-1.png" class="kg-image" alt="Data Analytics with PostgreSQL: The Ultimate Guide" loading="lazy" width="2703" height="1071"></figure><p>To get the best performance when running analytical workloads, using columnar storage format like Parquet is often not enough because the PostgreSQL query engine still processes each row sequentially.</p><p>Enter <a href="https://github.com/duckdb/duckdb?ref=blog.gettelio.com" rel="noreferrer">DuckDB</a>, a columnar-vectorized query execution engine. Think of DuckDB as &quot;SQLite for analytics&quot; that is lightweight and can store all its data on a disk or in memory. Here are its main key features:</p><ul><li>It uses a columnar-verctorized engine, which supports parallel execution and can efficiently process large batches of values, a.k.a. vectors.</li><li>It is a single-binary program without any external dependencies that can run on all operating systems or can be embedded into another program.</li><li>It provides a universal SQL access to various data types such as Parquet, CSV, JSON, and sources such as remote S3 buckets, API endpoints, Excel.</li><li>And similarly to PostgreSQL, it also supports extensions to improve its core functionality.</li></ul><p>DuckDB version 1.0 was released in 2024, and since then many new PostgreSQL extensions that embed DuckDB or build their own query engines started being developed. Here are some of them:</p><ul><li><a href="https://github.com/duckdb/pg_duckdb?ref=blog.gettelio.com" rel="noreferrer">pg_duckdb</a> embeds DuckDB and can query Parquet files from object storage</li><li><a href="https://github.com/Mooncake-Labs/pg_mooncake?ref=blog.gettelio.com" rel="noreferrer">pg_mooncake</a> embeds DuckDB and uses columnar storage within PostgreSQL</li><li><a href="https://github.com/paradedb/pg_analytics?ref=blog.gettelio.com" rel="noreferrer">pg_analytics</a> embeds DuckDB and uses foreign data wrappers to read from S3</li><li><a href="https://github.com/heterodb/pg-strom?ref=blog.gettelio.com" rel="noreferrer">pg-strom</a> uses a parallel execution engine that can leverage GPU cores</li></ul><figure class="kg-card kg-code-card"><pre><code class="language-sql">-- Set up the extension
CREATE EXTENSION pg_duckdb;

-- Querying data from a Parquet file
SELECT issued_on FROM read_parquet(&apos;s3://bucket/dir/invoices.parquet&apos;);</code></pre><figcaption><p><span style="white-space: pre-wrap;">Querying Parquet data using pg_duckdb extension</span></p></figcaption></figure><p><strong>Super Extensions</strong></p><p>There are some extensions that significantly alter how PostgreSQL works. Such extensions are sometimes called &quot;super extensions&quot; and very often installed on a dedicated PostgreSQL server.</p><p>These extensions are not designed for data analytics, but some of their features can still improve PostgreSQL query execution and performance. Here are some of the extensions:</p><ul><li><a href="https://github.com/timescale/timescaledb?ref=blog.gettelio.com" rel="noreferrer">timescaledb</a> is designed to turn PostgreSQL into a time-series database. It can automatically partition tables by time-columns, use hybrid row-columnar store, and refresh materialized views incrementally.</li><li><a href="https://github.com/citusdata/citus?ref=blog.gettelio.com" rel="noreferrer">citus</a> is designed to turn PostgreSQL into a distributed database. It can automatically shard tables across servers and use columnar storage for compression and query performance. </li></ul><h3 id="extensions-pros">Extensions Pros</h3><ul><li>Adding new features to PostgreSQL while keeping everything under one roof.</li><li>Variety of open-source extensions that can provide great flexibility and customization to your PostgreSQL.</li><li>Querying data stored in compressed columnar format and/or bringing an analytical query engine to improve performance.</li></ul><h3 id="extensions-cons">Extensions Cons</h3><ul><li>Performance overhead when running analytical queries within the same PostgreSQL that can negatively affect transactional queries.</li><li>Very limited support for installable extensions in managed PostgreSQL services. For example, here is the AWS Aurora&#xA0;<a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/AuroraPostgreSQL.Extensions.html?ref=blog.gettelio.com#AuroraPostgreSQL.Extensions.16" rel="nofollow">allowlist</a>.</li><li>Increased dependency management and maintenance complexity when using extensions or upgrading PostgreSQL/extension versions.</li><li>Manual data syncing and data mapping using ETL pipelines or within PostgreSQL from native row-based storage to columnar storage for best performance.</li><li>Some TimescaleDB features are not available under an open-sourced license: incremental materialized views, compression, and query optimizations.</li></ul><hr><h2 id="integrate-with-analytics-databases"><strong>Integrate with Analytics Databases</strong></h2><p>There are a few OLAP databases that can integrate with PostgreSQL databases and are also compatible, so you can use PostgreSQL tools like database drivers and adapters as usual.</p><p><a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com" rel="noreferrer"><strong>BemiDB</strong></a></p><p><em>Disclaimer: I&#x2019;m a BemiDB contributor. And even though I&#x2019;m biased and want more people to use BemiDB as a simple solution to the PostgreSQL data analytics problem, I&#x2019;ll try to be as objective as possible.</em></p><p>BemiDB is a read replica optimized for analytics. It connects to a PostgreSQL database, automatically syncs data into a compressed columnar storage, and uses a Postgres-compatible analytics query engine to read the data.</p><p>Here are its main key features:</p><ul><li>Single binary that can be run on any machine. The compute is stateless and separated from storage, making it easier to run and manage.</li><li>Embeds DuckDB to improve performance, a columnar-vectorized query execution engine optimized for analytical workloads.</li><li>Uses an open columnar format for tables with compression. The data can be stored either on a local file system or on S3-compatible object storage.</li><li>Postgres-integrated, both on the SQL dialect and table data level. I.e., all <code>SELECT</code> queries executed on a primary PostgreSQL server can be ported to BemiDB as is.</li></ul><figure class="kg-card kg-code-card"><pre><code class="language-sh"># Sync data from PostgreSQL
bemidb --pg-database-url postgres://localhost:5432/dbname sync

# Start BemiDB
bemidb start

# Query BemiDB as a PostgreSQL read replica
psql postgres://localhost:54321/bemidb -c &quot;SELECT COUNT(*) FROM table_from_postgres&quot;</code></pre><figcaption><p><span style="white-space: pre-wrap;">Running BemiDB as a read replica optimized for analytics</span></p></figcaption></figure><p>We&apos;ve already described Parquet data format and its benefits. The next evolutional approach is using open table formats, such as Iceberg that is used by BemiDB under the hood. These formats use Parquet files to store data in compressed columnar format and stitch them together using metadata files according to format specifications. This helps adding smaller Parquet data files incrementally instead of fully rewriting files on every data change.</p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/BemiDB-2.png" class="kg-image" alt="Data Analytics with PostgreSQL: The Ultimate Guide" loading="lazy" width="2876" height="1094"></figure><p>With open table formats like Iceberg, in addition to query performance benefits, it&apos;s possible to achieve things that are not possible with PostgreSQL. For example, data interoperability across different databases/tools/services and schema evolution/time travel enabling access to historical versions.</p><p><a href="https://github.com/ClickHouse/ClickHouse?ref=blog.gettelio.com" rel="noreferrer"><strong>ClickHouse</strong></a></p><p>ClickHouse is a column-oriented database designed for real-time analytics. The company recently acquired <a href="https://github.com/PeerDB-io/peerdb?ref=blog.gettelio.com" rel="noreferrer">PeerDB</a> that allows syncing data from PostgreSQL into ClickHouse in real-time. It also has some basic PostgreSQL compatibility allowing you to connect and run ClickHouse SQL queries via the PostgreSQL wire protocol.</p><p>Here are its main key features:</p><ul><li>Distributed processing across multiple servers in a cluster enabling horizontal scalability.</li><li>Vectorized query execution engine optimized for analytical workloads.</li><li>Columnar storage that enables data compression and improved query performance.</li><li>ClickHouse is optimized for inserting large batches of rows, usually between 10K and 100K rows.</li></ul><p>PeerDB that behaves like an ETL tool that connects to PostgreSQL databases using <a href="https://www.postgresql.org/docs/current/logical-replication.html?ref=blog.gettelio.com" rel="noreferrer">logical replication</a> and <a href="https://www.postgresql.org/docs/current/logicaldecoding.html?ref=blog.gettelio.com" rel="noreferrer">decoding</a>. The transformations can be performed using custom Lua scripts.</p><h3 id="analytics-databases-pros">Analytics Databases Pros</h3><ul><li>The best performance tailored specifically for analytical workloads.</li><li>Integrate with PostgreSQL databases and replicate data into a scalable columnar storage format. Minimal impact on PostgreSQL performance, resource usage, and internal configuration.</li><li>BemiDB consists of a single binary allowing to easily run and manage an optimized for analytics PostgreSQL read replica.</li><li>ClickHouse allows batch data inserts directly, bypassing PostgreSQL.</li></ul><h3 id="analytics-databases-cons">Analytics Databases Cons</h3><ul><li>They are not PostgreSQL and don&apos;t support many PostgreSQL-specific features or extensions.</li><li>Increased system complexity with extra server processes running in addition to PostgreSQL.</li><li>BemiDB&#xA0;doesn&apos;t support direct Postgres-compatible write operations (yet), so it can only work as a read replica.</li><li>ClickHouse is quite different from PostgreSQL and OLTP databases in many ways: different SQL dialect, limitations on data mutability, no support for ACID (atomicity, consistency, isolation, durability), and many others.</li></ul><hr><h2 id="use-proprietary-solutions"><strong>Use Proprietary Solutions</strong></h2><p>Due to PostgreSQL&apos;s popularity, many companies started building their custom proprietary solutions for analytics either on top of PostgreSQL or making them Postgres-compatible.</p><p><a href="https://cloud.google.com/products/alloydb?ref=blog.gettelio.com" rel="noreferrer"><strong>Google Cloud AlloyDB</strong></a></p><figure class="kg-card kg-image-card"><img src="https://blog.bemi.io/content/images/2025/02/AlloyDB.png" class="kg-image" alt="Data Analytics with PostgreSQL: The Ultimate Guide" loading="lazy" width="2765" height="998"></figure><p>AlloyDB is a managed PostgreSQL-compatible database for hybrid transactional and analytical workloads (HTAP). It can replace PostgreSQL for transactional queries and also deliver good performance for analytical queries.</p><p>Here are its main features:</p><ul><li>Enhanced query processing layers in PostgreSQL kernel for performance and shared storage in a region.</li><li>Embeds a proprietary vectorized engine and storage with an additional columnar format.</li><li>Query planner that automatically chooses an execution&#xA0;fully on columnar data, fully on row-oriented data, or a hybrid of the two.</li><li>Has a downloadable version called AlloyDB Omni that can also run on AWS and Azure in a Docker container.</li></ul><p><a href="https://www.enterprisedb.com/products/analytics?ref=blog.gettelio.com" rel="noreferrer"><strong>EDB Analytics Accelerator</strong></a></p><p>EDB (a.k.a. EnterpriseDB) Postgres AI is a data platform for both transactional and analytical workloads. The analytics product is powered by PostgreSQL and the proprietary extension called PGAA.</p><p>Here are the main analytics features:</p><ul><li>Vectorized query engine optimized for columnar data formats.</li><li>Tiered storage, with hot data on a disk and cold data in object storage in open table formats.</li><li>Storage and compute separation with dedicated PostgreSQL replicas for analytical queries.</li></ul><p><a href="https://www.crunchydata.com/products/warehouse?ref=blog.gettelio.com" rel="noreferrer"><strong>Crunchy Data Warehouse</strong></a></p><p>Crunchy Data is a company that specializes in providing services, support, and solutions for PostgreSQL. The company released Crunchy Data Warehouse in 2024, an analytics database built on PostgreSQL.</p><p>The main features include:</p><ul><li>The latest versions of PostgreSQL with proprietary extensions.</li><li>Integrated DuckDB query engine by delegating parts of the query to it for vectorized execution.</li><li>S3 for storage with an Iceberg table format that can be queried with tools like Apache Spark.</li></ul><p><a href="https://www.firebolt.io/?ref=blog.gettelio.com" rel="noreferrer"><strong>Firebolt</strong></a></p><p>Firebolt is a cloud data warehouse. It started by forking ClickHouse to implement better storage and compute decoupling, along with other improvements. In 2024, they added Postgres SQL dialect compatibility.</p><ul><li>Vectorized query execution engine, ACID compliant.</li><li>Proprietary columnar data format and tiered storage in memory, local SSD, and S3.</li></ul><h3 id="proprietary-solutions-pros">Proprietary Solutions Pros</h3><ul><li>Fully managed cloud data warehouses optimized for analytical workloads.</li></ul><h3 id="proprietary-solutions-cons">Proprietary Solutions Cons</h3><ul><li>Vendor lock-in and limited control over the source code and data.</li><li>Very limited or no support at all for installable extensions. For example, here is the GCP AlloyDB&#xA0;<a href="https://cloud.google.com/alloydb/docs/reference/extensions?ref=blog.gettelio.com" rel="noreferrer">allowlist</a>.</li><li>Crunchy Data and Firebolt require manual data syncing using ETL pipelines or within PostgreSQL from native row-based storage to columnar storage.</li><li>Can be more expensive compared to other alternatives.</li></ul><hr><h2 id="conclusion"><strong>Conclusion</strong></h2><p>There is a wide variety of options for handling data analytics with PostgreSQL.</p><ul><li>Just using PostgreSQL and scaling it can be a great starting point for simpler analytical needs initially, allowing to keep the data stack simple.</li><li>With enough PostgreSQL expertise and access to custom extensions, installing them can help improve analytical performance within PostgreSQL.</li><li>If you don&apos;t want to spend time tuning PostgreSQL and all you need is a simple read replica optimized for analytics, then BemiDB is the best choice.</li><li>If you deal with many terabytes of mostly append-only data and don&apos;t mind switching to another SQL dialect, then ClickHouse is a great choice.</li><li>And if you already host PostgreSQL on platforms like GCP or EDB, then choosing their analytics solutions can reduce the number of data providers.</li></ul><hr><p><em>Check out the </em><a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com"><em>BemiDB GitHub repo</em></a><em> if you want to give it a shot. And subscribe to our blog if you want to learn more about PostgreSQL and data analytics.</em></p>]]></content:encoded></item><item><title><![CDATA[When Postgres Indexing Went Wrong]]></title><description><![CDATA[It’s important to understand basics of indexing and best practices around them for preventing system downtime. ]]></description><link>https://blog.gettelio.com/indexing/</link><guid isPermaLink="false">68ae707be3a2820001805d4e</guid><category><![CDATA[Engineering]]></category><dc:creator><![CDATA[Arjun Lall]]></dc:creator><pubDate>Mon, 23 Sep 2024 05:08:00 GMT</pubDate><media:content url="https://blog.gettelio.com/content/images/2025/08/image-671--2-.png" medium="image"/><content:encoded><![CDATA[<img src="https://blog.gettelio.com/content/images/2025/08/image-671--2-.png" alt="When Postgres Indexing Went Wrong"><p>Indexing in Postgres seems simple, but it&#x2019;s important to understand the basics of how it really works and the best practices for preventing system downtime.</p><p>TLDR: Be careful when creating indexes &#x2014; a lesson I learned the hard way when concurrent indexing failed silently.</p><h2 id="critical-incident">Critical incident</h2><p>At a previous company, we managed a high-volume Postgres instance with billions of rows of transactional data. As we scaled, query performance became a key priority, and one of the first optimizations was adding indexes. To avoid downtime, we used <code>CREATE INDEX CONCURRENTLY</code>, which allows indexing large tables without locking out writes for hours. Initially, p99 query performance improved dramatically.</p><p>A few weeks later, another team launched a new feature that was built to rely heavily on the new index. Everything seemed routine&#x2014;until the traffic spiked.</p><p>At first, the problem was subtle. A few queries took longer than expected. But within hours, the load began to spike. Query response times slowed to a crawl, and some requests were timing out. </p><p>We couldn&#x2019;t immediately see why. The index was in place, a quick <code>EXPLAIN ANALYZE</code> confirmed it was being used. But users were still experiencing massive slowdowns, and we were on the brink of a full-scale production outage.</p><p>It wasn&#x2019;t until we checked the server logs did we piece together what happened:</p><pre><code class="language-sql">CREATE INDEX CONCURRENTLY idx_email_2019 ON users_2019 (email);
ERROR: deadlock detected
DETAIL: Process 12345 waits for ShareLock on transaction 54321; blocked by process 54322.
</code></pre><h2 id="concurrent-indexing-can-fail-silently"><strong>Concurrent indexing can fail (silently)</strong></h2><p>Concurrent indexing needs more total work than a standard index build and takes much longer to complete. It uses a 2 phase approach that helps avoid locking the table:</p><ul><li><strong>Phase 1:</strong> A snapshot of the current data gets taken, and the index is built on that.</li><li><strong>Phase 2:</strong> Postgres then catches up with any changes (inserts, updates, or deletes) that happened during phase 1.</li></ul><p>Since this process is asynchronous, the <code>CREATE INDEX</code> command might fail, leaving an incomplete index behind. An &#x201C;invalid&#x201D; index is ignored during querying, but this oversight can have serious consequences if not monitored.</p><pre><code>postgres=# \d users_emails_2019
       Table &quot;public.users_emails_2019&quot;
 Column |  Type   | Collation | Nullable | Default
--------+---------+-----------+----------+---------
  ...   |            |           |          |
Indexes:
    &quot;idx&quot; btree (email) INVALID
</code></pre><p>In our case, the issue was amplified by the fact that our data was partitioned. The index had failed on some partitions but not others, leading to a situation where some queries were using the index while others were hitting unindexed partitions. This imbalance resulted in uneven query performance and significantly increased load on the system.</p><p>If we hadn&#x2019;t caught it when we did, we would have faced a full-blown production outage, impacting every user on the platform.</p><h2 id="best-practices-for-postgres-indexing"><strong>Best practices for Postgres indexing</strong></h2><p>To help others navigate this terrain, here are some best practices for Postgres indexing that can prevent these issues:</p><h3 id="avoid-dangerous-operations"><strong>Avoid dangerous operations</strong></h3><p>Always use the <code>CONCURRENTLY</code> flag when creating indexes in production. Without it, even smaller tables can block writes for unacceptably long, leading to system downtime. While <code>CONCURRENTLY</code> takes more CPU and I/O, the trade-off is worth it to maintain availability. Keep in mind that concurrent index builds can only happen one at a time on the same table, so plan accordingly for large datasets.</p><h3 id="monitor-concurrent-index-creation-closely"><strong>Monitor concurrent index creation closely</strong> </h3><p>Don&#x2019;t take successful index creation for granted. The system table <code>pg_stat_progress_create_index</code> can be queried for progress reporting while indexing is taking place.</p><pre><code class="language-sql">postgres=# SELECT * FROM pg_stat_progress_create_index;
-[ RECORD 1 ]------+---------------------------------------
pid                | 896799
datid              | 16402
datname            | postgres
relid              | 17261
index_relid        | 136565
command            | CREATE INDEX CONCURRENTLY
phase              | building index: loading tuples in tree
lockers_total      | 0
lockers_done       | 0
current_locker_pid | 0
blocks_total       | 0
blocks_done        | 0
tuples_total       | 10091384
tuples_done        | 1775295
partitions_total   | 0
partitions_done    | 0
</code></pre><h3 id="manually-validate-indexes"><strong>Manually validate indexes</strong></h3><p>If you don&#x2019;t check your indexes, you might think you&#x2019;re able to rely on them when you can&#x2019;t. And although an invalid index gets ignored during querying, it still consumes update overhead. Common causes for index failures include:</p><ol><ul><li>Deadlocks: Index creation might conflict with ongoing transactions, leading to deadlocks.</li><li>Disk Space: Large indexes may fail due to insufficient disk space.</li><li>Constraint Violations: Creating unique indexes on columns with non-unique data will result in failures.</li></ul></ol><p>You can find all invalid indexes by running the following:</p><pre><code>SELECT * FROM pg_class, pg_index WHERE pg_index.indisvalid = false AND pg_index.indexrelid = pg_class.oid;
</code></pre><p>You can also query the <code>pg_stat_all_indexes</code> and <code>pg_statio_all_indexes</code> system views to verify that the index is being accessed.</p><h3 id="fix-invalid-indexes">Fix invalid indexes</h3><p>Invalid indexes can be recovered using the <code>REINDEX</code> command. It&#x2019;s the same as dropping and recreating the index, except it would also lock out reads that attempt to use that index (if not specifying <code>CONCURRENTLY</code>). Note that <code>CONCURRENTLY</code> reindexing isn&#x2019;t supported in versions below Postgres 12.</p><pre><code class="language-sql">REINDEX INDEX CONCURRENTLY idx_users_email_2019;
</code></pre><p>If a problem occurs while rebuilding the indexes, it&#x2019;d leave behind a new invalid index suffixed with&#xA0;<code>_ccnew</code>. Drop it and retry&#xA0;<code>REINDEX CONCURRENTLY</code>.</p><pre><code class="language-sql">postgres=# \d users_2019
       Table &quot;public.tab&quot;
 Column |  Type   | Modifiers
--------+---------+-----------
 col    | integer |
Indexes:
    &quot;users_emails_2019&quot; btree (col) INVALID
    &quot;users_emails_2019_ccnew&quot; btree (col) INVALID
</code></pre><p>If the invalid index is suffixed with <code>_ccold</code>, it&#x2019;s the original index that wasn&#x2019;t fully replaced. You can safely drop it, as the rebuild has succeeded.</p><h3 id="create-partition-indexes-consistently"><strong>Create partition indexes consistently</strong></h3><p>Newly created partitioned tables or small tables (&lt;100k) can easily just create indexes synchronously on the parent table, and it&apos;d automatically propagate indexes to all partitions, including any newly created ones in the future.</p><pre><code>CREATE INDEX idx_users_email ON users (email);
</code></pre><p>But it&#x2019;s currently not possible to use the <code>CONCURRENTLY</code> flag when creating an index on the root partitioned table. What you should use instead is the <code>ONLY</code> flag. This tells the parent table to not apply the index recursively to children, so the table isn&#x2019;t locked.</p><pre><code class="language-sql">-- Create an index on the parent table (metadata only operation);
CREATE INDEX idx_users_email ON ONLY users (email);
</code></pre><p>This creates an invalid index first. Then we can create indexes for each partition and attach them to the parent index:</p><pre><code class="language-sql">CREATE INDEX CONCURRENTLY idx_users_email_2019
    ON users_2019 (email);
ALTER INDEX idx_users_email
    ATTACH PARTITION idx_users_email_2019;

CREATE INDEX CONCURRENTLY idx_users_email_2020
    ON users_2020 (email);
ALTER INDEX idx_users_email
    ATTACH PARTITION idx_users_email_2020;

// repeat for all partitions
</code></pre><p>Only once all partitions are attached, the index for the root table will be marked as valid automatically. The parent itself is just a &#x201C;virtual&#x201D; table without any storage, but can serve to ensure all partitions maintain a consistent indexing strategy.</p><h3 id="check-the-query-execution-plan"><strong>Check the query execution plan</strong></h3><p>Using the <code>EXPLAIN ANALYZE</code> command provides a comprehensive view of the query execution plan, detailing how Postgres processes your query. This breakdown is essential for verifying that the expected indexes are being utilized effectively.</p><pre><code class="language-sql">EXPLAIN ANALYZE SELECT * FROM users_2019 WHERE email = &apos;arjun@bemi.io&apos;;

Index Scan using idx_users_email_2019 on users_2019  (cost=0.15..0.25 rows=1 width=48) (actual time=0.015..0.018 rows=1 loops=1)
  Index Cond: (email = &apos;arjun@bemi.io&apos;::text)
Planning Time: 0.123 ms
Execution Time: 0.028 ms
</code></pre><h3 id="remove-unused-indexes"><strong>Remove unused indexes</strong></h3><p>Sometimes the indexes we add aren&#x2019;t as valuable as expected. To prune our indexes to optimize write performance, we can check which indexes haven&#x2019;t been used:</p><pre><code class="language-sql">select 
    indexrelid::regclass as index, relid::regclass as table 
from 
    pg_stat_user_indexes 
    JOIN pg_index USING (indexrelid) 
where 
    idx_scan = 0 and indisunique is false;
</code></pre><p>By implementing these best practices, you can avoid scary mistakes. Remember to monitor, validate, and understand the implications of your indexing strategy. The cost of overlooking these details can be significant, and a proactive approach will help you maintain a stable and efficient database.</p><p><em>When Postgres indexing isn&apos;t enough to scale, check out the </em><a href="https://github.com/BemiHQ/BemiDB?ref=blog.gettelio.com" rel="noreferrer"><em>BemiDB</em></a><em> for handling analytical workloads on Postgres.</em></p>]]></content:encoded></item></channel></rss>