<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[KnowEaseAI Engineering]]></title><description><![CDATA[Honest takes on software engineering, AI systems, and the gap between how tech is taught and how it actually works. What job descriptions don't tell you, what actually works in production, and what the industry gets wrong.]]></description><link>https://knoweaseai.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6af784cb754f226e6dde5dfd/f68b0b16-b62a-427c-bfd1-6f81e3694f5d.png</url><title>KnowEaseAI Engineering</title><link>https://knoweaseai.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sat, 21 Nov 2026 20:12:49 GMT</lastBuildDate><atom:link href="https://knoweaseai.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Kafka Didn't Fail. Your Configuration Did.]]></title><description><![CDATA[Consumer lag, broker disk failures, schema changes breaking every consumer, 3 real Kafka production failures, what went wrong, and the exact fix need.]]></description><link>https://knoweaseai.hashnode.dev/kafka-failure</link><guid isPermaLink="true">https://knoweaseai.hashnode.dev/kafka-failure</guid><category><![CDATA[kafka]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[System Design]]></category><category><![CDATA[distributed systems]]></category><dc:creator><![CDATA[Tanmoy Dasgupta]]></dc:creator><pubDate>Sat, 14 Nov 2026 16:46:38 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af784cb754f226e6dde5dfd/c9b03a22-a756-4f54-bb5d-6d927d0a7a2b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><ul><li><p>Consumer lag + rebalance storms kill pipelines during traffic spikes</p></li><li><p>Disk fills silently until a broker dies and partitions go offline</p></li><li><p>One schema change without a default field breaks every consumer overnight</p></li><li><p>Kafka didn't fail in any of these. The configuration did.</p></li></ul></div>
</div>

<p>Kafka runs LinkedIn's activity pipeline, Uber's trip events, and Robinhood's order stream. It processes trillions of messages a day without blinking. And yet, it brings down production systems regularly — not because Kafka is unreliable, but because engineers ship it like a managed service when it is closer to a loaded weapon. Here is what actually goes wrong.</p>
<hr />
<h2>Where Does Kafka Actually Run in Production?</h2>
<p>Kafka sits in the middle of systems that cannot afford gaps: real-time fraud detection at payments companies, event sourcing for microservices, CDC pipelines feeding data warehouses, activity feeds at social platforms, and log aggregation at scale. The common thread is that every downstream system trusts Kafka as the source of truth. When it misbehaves, everything downstream breaks simultaneously.</p>
<h2>Why Do Kafka Pipelines Fail in Production?</h2>
<p>Three failure patterns appear repeatedly across teams. None of them are Kafka bugs. All of them are configuration and operational gaps that looked fine until they weren't.</p>
<h2>Real Failure #1: Consumer Lag That Never Recovers</h2>
<p>A fintech company processing card transactions had a Kafka consumer group fall behind during a Black Friday traffic spike. Lag hit 4 million messages. The team scaled up consumers to catch up. The result: a rebalance storm. Every time a new consumer joined, Kafka triggered a full group rebalance, pausing all consumption. More consumers made it worse. Lag kept climbing.</p>
<p>What went wrong: <a href="http://max.poll.interval.ms">max.poll.interval.ms</a> was set too low. Consumers were being marked dead before finishing processing, triggering constant rebalances.</p>
<p>Fix: Tune <a href="http://max.poll.interval.ms">max.poll.interval.ms</a> to match your actual processing time. Separate slow processing into async workers. Add consumer lag monitoring before you need it, not after.</p>
<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>Never scale Kafka consumers during a rebalance storm. Pause, let the group stabilize, then scale.</p></div>
</div>

<h2>Real Failure #2: Disk Fills Up, Broker Dies</h2>
<p>A logistics company ran Kafka on three brokers with default 7-day retention. A downstream consumer went down for 10 days during a migration. Messages accumulated. One broker's disk hit 95%, writes slowed, then the broker went offline. With replication factor 2 and one broker down, one partition lost its leader. Producers started throwing NotLeaderForPartitionException. The pipeline stopped.</p>
<p>What went wrong: No disk alerting. Replication factor too low. Retention policy had no relationship to actual consumer SLA.</p>
<p>Fix: Set retention as min(disk capacity ÷ daily throughput, consumer recovery SLA). Use replication factor 3 for anything production. Alert at 70% disk, not 95%.</p>
<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>Replication factor 2 is not redundancy. It is one broker failure away from data unavailability.</p></div>
</div>

<h2>Real Failure #3: Schema Change Breaks Every Consumer</h2>
<p>A data platform team added a required Avro field without a default value. Deployed the producer. Every consumer running the old schema started throwing deserialization exceptions. The pipeline went silent. No messages were lost — nothing was being processed. The error was not obvious because consumers were technically running.</p>
<p>What went wrong: No schema registry enforcement. No backward compatibility check before deploy. A required field without a default is a breaking change.</p>
<p>Fix: Use Confluent Schema Registry with BACKWARD compatibility mode enforced. Every new field ships with a default. Run a schema diff in CI before any producer deploy.</p>
<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>Treat schema changes like database migrations. Reviewed, versioned, and rollback-tested before they touch production.</p></div>
</div>

<h2>What Actually Prevents Kafka Failures?</h2>
<p>The engineers who avoid these failures are not smarter. They have runbooks, they set alerts before incidents, and they treat Kafka as infrastructure that needs as much care as a database. Three non-negotiables:</p>
<ul>
<li>Lag alerting at the consumer group level, not the broker level</li>
<li>Replication factor 3 for every production topic</li>
<li>Schema registry with compatibility enforcement on every deploy</li>
</ul>
<p>Kafka itself held in all three cases above. The data was intact. What failed was observability, configuration discipline, and schema governance.</p>
<h2>Frequently Asked Questions</h2>
<h3>What causes Kafka consumer lag to spike suddenly?</h3>
<p>Consumer lag spikes when processing time per message exceeds <a href="http://max.poll.interval.ms">max.poll.interval.ms</a>, causing Kafka to mark consumers as dead and trigger group rebalances. Scaling consumers during a rebalance worsens it.</p>
<h3>How do you prevent Kafka broker disk failures in production?</h3>
<p>Set log retention based on your consumer recovery SLA, not a fixed default. Alert at 70% disk usage. Use replication factor 3 so a single broker failure does not take partitions offline.</p>
<h3>What is a Kafka rebalance storm?</h3>
<p>A rebalance storm happens when consumers continuously join and leave a group faster than Kafka can stabilize assignments. Each rebalance pauses all consumption, creating a feedback loop where lag worsens as you scale.</p>
<h3>How do you safely evolve Kafka schemas without breaking consumer?</h3>
<p>Use a schema registry with BACKWARD compatibility enforced. Every new field must have a default value. Run schema compatibility checks in CI before deploying any producer change.</p>
<h3>What is the right replication factor for Kafka in production?</h3>
<p>Minimum replication factor 3 for production topics. Replication factor 2 means one broker failure can make a partition unavailable if the remaining replica is not caught up.</p>
<h2>Practice These Concepts</h2>
<p>Kafka failure modes are a standard question in data engineering and ML infrastructure system design interviews — consumer lag, partition leadership, schema evolution, exactly-once semantics.</p>
<p>If you are preparing for roles at product companies, <code>KnowEaseAI</code> covers Kafka and distributed systems in the Data Engineering track (<a href="https://knoweaseai.com">https://knoweaseai.com</a>) — real interview questions, structured answers, and graded practice so you know where you actually stand.</p>
<p>No fluff. No FAANG templates. Just what interviewers actually ask.</p>
<p>Start practicing free → (<a href="https://knoweaseai.com/audit">https://knoweaseai.com/practice</a>)</p>
]]></content:encoded></item><item><title><![CDATA[The Tech Job Market Is Not Broken]]></title><description><![CDATA[After scraping and scoring 1,000+ LinkedIn job listings, the data reveals why engineers aren't getting responses — and it's not the market at all.]]></description><link>https://knoweaseai.hashnode.dev/the-tech-job-market-is-not-broken</link><guid isPermaLink="true">https://knoweaseai.hashnode.dev/the-tech-job-market-is-not-broken</guid><category><![CDATA[techjobs]]></category><category><![CDATA[job search]]></category><category><![CDATA[Machine Learning]]></category><category><![CDATA[Software Engineering]]></category><category><![CDATA[Career]]></category><dc:creator><![CDATA[Tanmoy Dasgupta]]></dc:creator><pubDate>Sat, 14 Nov 2026 12:10:06 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af784cb754f226e6dde5dfd/51c38840-6d86-4918-a722-d36d39072817.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most engineers who are struggling to get responses believe the market is the problem. Too competitive. Too many layoffs. Too few openings. The data says something different.</p>
<p>The market is not broken. It is polluted. There is a meaningful difference between the two, and it changes everything about how you should be spending your time.</p>
<p>I built a scraper that pulls LinkedIn job listings daily, scores them against quality signals, and filters for roles actually worth applying to. After processing over a thousand listings across major tech hubs globally, what emerged was not a frozen market. It was a market where the noise-to-signal ratio has gotten so bad that most engineers are applying in the dark. Here is what the data shows.</p>
<hr />
<h2>How many LinkedIn tech job listings are real?</h2>
<p>The first thing the scraper forced me to confront was how many listings come from companies with no intention of building a product. Staffing agencies, body-shop consultancies, and IT outsourcing arms post aggressime is their business model. They are not hiring a machinelearning engineer to build something. They are filling a bench.</p>
<p>When I ran the data, roughly 38 to 42 percent of raw listings either came from known staffing players or matched body-shop patterns: generic JDs, no tech stack named, roles duplicated across multiple cities, company <code>solutions</code>, <code>placement</code>, or <code>technologies</code>. The pattern holds regardless of geography.</p>
<p>If you apply to raw LinkedIn results without filtering, you are spending more than a third of your effort on roles that will not advance your career.</p>
<hr />
<h2>What do most AI Engineer job descriptions actually require?</h2>
<p><code>AI Engineer</code> is now the most overloaded job title in tech hirno AI product, no ML infrastructure, and no defined datastrategy post for AI Engineers because the title attracts candidates and signals modernity to investors.</p>
<p>Here is what those JDs contain. Generic requirements like <code>familiarity with AI tools</code> and <code>experience with ChatGPT</code>. No named framework. No model. No compute environment. No problem described.</p>
<p>Roughly 60 percent of JDs with <code>AI Engineer</code> in the title had beyond <code>Python and AI</code>. Compare this to roles titled <code>ML Engineer</code> or <code>Applied Scientist</code> at the same companies, which named PyTorch or TensorFlow, specified the domain, and described the team. The signal-to-noise gap between the two title conventions is not c</p>
<p>If a company cannot describe what model you will work on and wey do not have an AI engineering role. They have a marketingrequirement dressed as a technical one.</p>
<hr />
<h2>Should you apply to jobs with 500+ applicants on LinkedIn?</h2>
<p>This one is math, not opinion. When a role crosses 500 applicacomes unmanageable. The filtering is brutal: keyword matching,referrals jumping the queue, inbound connections from people the recruiter already knows. Cold applications at 500+ are, statistically, not being read.</p>
<p>The average applicant count on a listing that was two days old was 340. For Easy Apply roles, it was closer to 580. The role did not change. What changed was that two days of LinkedIn feed exposure turned a fresh opp.</p>
<p>Under 50 applicants, you have a real shot. you need an internal referral or an extraordinary portfolio.</p>
<hr />
<h2>Most Job Descriptions Are Copy-Pasted Boilerplate</h2>
<p>JD quality is genuinely poor. Not occasionally. Structurally. ech stack, describes the team, and explains the problem. What most contain instead is a generic requirements section from a previous hire, technologies listed broadly enough to filter no one, and a responsibilities section that could apply to any role at any company.</p>
<p>Around 65 percent of listings scored below the threshold for "ny were funded startups with real products. The JD was justnever treated as a signal worth crafting.</p>
<p>The irony: a vague JD harms the company as much as the candidate. When you describe nothing specifically, you attract everyone generally.</p>
<hr />
<h2>Does LinkedIn Easy Apply hurt your chances?</h2>
<p>Easy Apply roles received on average 2.8 times more applicatioiring an external application. Same job titles, same companytiers. Only the application mechanic differed. For the company, three times the volume with the same hiring capacity. For the engineer, your application lands in a pile with hundreds who clicked through in thirty se</p>
<p>A role requiring an ATS submission, a targeted note, or a briets. The friction filters out the casual click. For coldapplications to competitive roles, the external path is a real advantage simply because fewer people bother.</p>
<hr />
<h2>Ghost Companies Post More Aggressively Than Real Ones</h2>
<p>The follower count on a company's LinkedIn page is a surprisinwith fewer than 200 followers post aggressively andconsistently produce low-quality JDs — not because small companies are bad, but because companies genuinely hiring at that size typically source through their network and do not need to spam job boards.</p>
<p>A company with 150 followers, a <code>Senior AI Engineer</code> JD, no pre-day-old posting is almost certainly a consulting firm or acompany that has been reposting the same role for months because they cannot close it. A company with 5,000 followers, recent activity, and a JD that names specific tools is a different signal entirely.</p>
<hr />
<h2>What Actually Signals a Quality Listing</h2>
<p>Named tech stack with specific tools. A described team structure. A realistic experience range. Under 100 applicants. An active company page with meaningful follower count. A role title that matches what the</p>
<p>None of these are revolutionary criteria. The problem is applyreds of listings per day is not realistic. That is exactly whywe built the jobs feed at <a href="http://knoweaseai.com/jobs">KnowEaseAI</a> — staffing companies removed, applicant thresholds applied, JD quality scored, company signal checked. Not a comprehensive job board. A short, filtered list of roles actua</p>
<hr />
<p>The global tech job market is not broken. It is noisy. The underlying demand for strong engineers at product companies is real. The problem is a filtering problem, not a jobs problem. Once you understand that the raw for your benefit, the approach shifts from applying to moreroles to filtering harder, applying faster, and focusing on signal.</p>
<p>The data makes that possible. You just have to know where to look.</p>
<p>Browse the filtered feed at <a href="http://knoweaseai.com/jobs">KnowEaseAI jobs</a>.</p>
<h2>Frequently Asked Questions</h2>
<p><strong>Is the tech job market actually bad in 2026?</strong> The market has open roles but extreme noise. 38–42% of listings come from staffing agencies, not product companies.</p>
<p><strong>How many applicants is too many on a LinkedIn job post?</strong> Over 500 applicants means your cold application statistically won't be read. Under 50 gives you a real chance.</p>
<p><strong>Why are AI Engineer jobs so hard to get?</strong> 60% of AI Engineer JDs require nothing beyond "Python and AI." They're marketing posts, not real engineering roles.</p>
<p><strong>What makes a LinkedIn job listing worth applying to?</strong> Named tech stack, under 100 applicants, company page with 1,000+ followers, role title matching the JD content.</p>
]]></content:encoded></item></channel></rss>