The DevSecOps Talks Podcast

Gitops a new concept on devops. Whats is it and how can you use it when deploy and setup your k8s cluster. 

---

Summary

GitOps sounds simple — put Kubernetes manifests in Git and let the cluster pull changes — but the episode quickly reveals the real debate is not about Git at all. Andrey argues the only genuinely novel thing about GitOps is the pull-based model where an in-cluster agent reconciles state, while Julien questions whether GitOps is good for day-2 operations or just for bootstrapping clusters. The spiciest moment: Andrey declares "life is too short to do pull requests" and advocates pushing straight to master with strong CI/CD guardrails instead.

Key Topics

What GitOps actually is — and what it is not

Andrey frames the discussion by separating what is genuinely new about GitOps from what teams have already been doing for years. Storing deployment specifications in Git, he argues, is just version control — teams have done that for a decade. The meaningful difference is the deployment model: instead of an external CI/CD server pushing changes into Kubernetes by calling the cluster API, GitOps places an agent inside the cluster that either receives a webhook or polls a Git repository, pulls in the desired state, and applies it from within.

That pull-based model is what Andrey identifies as the core innovation. It eliminates the need to expose the Kubernetes API externally — a real concern when using hosted CI services like CircleCI, which would otherwise need network access to the cluster. As Andrey puts it, exposing the API externally is risky "unless you want someone mining bitcoin on your cluster."

He references the tooling landscape at the time: Weaveworks (the company that coined the term "GitOps" and created WeaveNet, a Kubernetes CNI driver), Flux, Argo, and Jenkins X. He notes that Flux and Argo were joining forces at the time of recording. He also mentions Jenkins X as a potential GitOps tool, since it runs CI/CD jobs natively in Kubernetes, but expresses skepticism about using Kubernetes for build workloads — Kubernetes is declarative about desired state, but "you cannot declare my build is successful because you have no idea how your build gonna go."

Editor's note: Weaveworks, the company that originated the term "GitOps," shut down in February 2024. Flux continues as a CNCF graduated project. The GitOps principles have since been formalized by the OpenGitOps project under the CNCF.

The Weaveworks definition, read straight from the source

Andrey reads Weaveworks' concise GitOps definition from their blog and walks through its key points:

1. The desired state of the whole system is described declaratively — Git is the single source of truth for every environment.
2. All changes to desired state are Git commits — operations are driven through version control.
3. The cluster state is observable — so teams can detect when desired and observed states have converged or diverged.
4. A convergence mechanism brings the system back — when states diverge, the cluster automatically reconciles, either triggered immediately by a webhook or on a configurable polling interval. Rollback is simply convergence to an earlier desired state.

Andrey also raises a nuance about Helm: since Helm templates can produce different output depending on input variables, true GitOps implies committing not only the Helm charts but also the rendered manifests — because the generated output is what actually represents the declarative desired state.

He draws a comparison to GitHub's earlier promotion of ChatOps, noting that many of the same ideas — observable, verifiable changes driven through a central workflow — were already part of GitHub's operational philosophy, just with a different interface.

Two layers: infrastructure-as-code and in-cluster GitOps

Julien offers a more practical framing, splitting the problem into two distinct layers:

1. Infrastructure as code — setting up the underlying infrastructure (VPCs, clusters, networking)
2. GitOps — managing what runs inside the Kubernetes cluster: applications, operational tooling like monitoring (he mentions FluentD as an example), and supporting services

In Julien's model, a Git repository becomes the authoritative inventory of everything that should exist in the cluster. He describes the ideal: "if anything else is running here, alert me or kill it." That gives teams confidence that the observed cluster state matches the intended one, and helps prevent configuration drift — a problem the hosts discussed in their earlier infrastructure-as-code episode.

Day-2 operations: where the model gets tested

While Julien appreciates GitOps for defining and bootstrapping cluster state, he is openly skeptical about its effectiveness for long-running operations. He distinguishes between two very different challenges: "setting up things" versus "running things for a long time — they're not the same."

Real environments drift. People intervene manually during incidents. Urgent fixes happen outside the normal workflow. The clean desired-state model becomes harder to maintain once the messiness of day-2 operations enters the picture. Julien frames this as an open question rather than a settled answer: GitOps may be excellent for establishing a clean baseline, but whether it holds up as a complete long-term operating model remains to be proven.

Who controls changes: developers, operators, or both?

Andrey raises a governance concern: GitOps can look like a direct developer-to-cluster pathway. If a developer changes a YAML file, commits it, and the cluster automatically applies the change, operations staff are effectively bypassed — "there is nowhere an operation person can interfere with this."

Julien pushes back, arguing that the workflow — not the tooling — determines who has control. If changes go through pull requests with review and approval, it does not matter whether the author is a developer or an operator. Both participate in the same process. The mechanism is the same one used for application code: propose a change, review it, merge it.

Pull requests, compliance, and "push to master"

The conversation takes its most opinionated turn when the topic shifts to pull requests.

Andrey is blunt: "Life is too short to do pull requests. You never get anything done. You do a pull request, you ask for review and then you hunt the person for two days." His preference is to push directly to master and build CI/CD pipelines strong enough to catch mistakes — "you build your system to defend yourself from the fools."

He does acknowledge an important exception: regulated industries where every production deployment must be peer-reviewed or approved. In those environments, formal review is not just a process preference but a compliance mechanism that can significantly reduce legal exposure when something goes wrong.

Andrey also shares a personal practice: because he frequently switches between projects and loses context, the first thing he does is document every verification step as part of the CI/CD pipeline. That way, when he returns to a project months later, the pipeline already encodes everything he would need to remember. "There is no guarantee that someone else has a better understanding of what I did."

Observability gaps in GitOps pipelines

Andrey identifies a practical developer-experience problem with GitOps: the visibility gap.

In a traditional pipeline, a developer can trace a change end-to-end — build, test, deploy — in one place. With GitOps, the CI pipeline ends when it commits changes to a repository. The actual deployment happens later, inside the cluster, through a separate reconciliation process. "My pipeline stops at the place where I do commit, push, done. Since then, pipeline doesn't have much to absorb."

To understand whether a deployment succeeded, the developer needs to inspect cluster state rather than the original pipeline. Bridging that gap requires additional tooling and represents a real paradigm shift in how teams observe deployments.

He also flags a repository-structure problem: if source code and deployment manifests live in the same repository, updating manifests can trigger the source-code pipeline again — requiring conditional logic to prevent unnecessary rebuilds.

Deployment ordering and full-system validation

Julien closes the discussion with a practical concern: deployment order matters in real systems. A proxy may need a backend to exist first. Some components cannot be rolled out in arbitrary order without causing failures.

He also questions the validation model. In a software build pipeline, teams rebuild and test the entire application from the main branch to verify the whole system works. But with GitOps, a change to one part of the cluster may be applied incrementally without validating the full cluster state end-to-end. "I will never test the full master branch and rebuild the full cluster from it, except everything goes."

That leaves an open question the hosts do not fully resolve: how can teams preserve the elegance of declarative Git-driven deployment while managing sequencing, dependencies, and whole-system confidence?

Highlights

"Unless you want someone mining bitcoin on your cluster"

Andrey explains the security motivation behind the pull-based GitOps model — if you use an external CI system, you need to expose your Kubernetes API, which is not exactly ideal. His colorful warning about cryptocurrency miners makes the point memorable.

Listen to the episode for Andrey's full breakdown of why the pull-vs-push distinction is the real heart of GitOps.

"Life is too short to do pull requests."

The spiciest take of the episode. Andrey argues that pull requests slow teams to a crawl — you open one, ask for review, then spend two days hunting the reviewer. His alternative: push to master and build pipelines strong enough to protect against mistakes. He does carve out an exception for regulated industries where peer review is legally required.

Listen to the episode and decide whether you agree or strongly disagree.

"GitOps is a nice way to set up your Kubernetes cluster — but is it a good tool to keep it running? I'm not sure."

Julien draws a sharp line between bootstrapping a cluster and operating it long-term. Setting up things and running things for a long time are "not the same." It is a refreshingly honest admission that a clean architecture pattern does not automatically solve the messy reality of day-2 operations.

Listen to the episode for a take that many GitOps advocates skip over.

"You build your system to defend yourself from the fools."

Andrey's philosophy in one sentence. Rather than relying on human review processes, invest in CI/CD pipelines and automated guardrails that prevent mistakes regardless of who pushes the change. He backs this up with a personal habit: encoding every verification step into the pipeline so future-him does not have to remember anything.

Listen to the episode for a practical argument in favor of automation over process.

"If anything else is running here — alert me or kill it."

Julien describes the appeal of GitOps as an authoritative inventory of what should exist in a cluster. If the Git repository defines the desired state and the cluster enforces it, anything unauthorized can be flagged or removed. It is one of the clearest expressions of why teams are drawn to the GitOps model.

Listen to the episode for a practical view of GitOps as cluster hygiene.

The daughter interruption

Mid-argument about observability gaps, Andrey's daughter walks in wanting to share something exciting. It is a charming reminder that even deep infrastructure debates happen in real life with real interruptions.

Listen to the episode for the unscripted moment — and Andrey's smooth recovery.

Resources

• Flux — the GitOps family of projects — The CNCF-graduated GitOps toolkit originally created by Weaveworks. Continuously reconciles Kubernetes cluster state with Git repositories.
• Argo CD — Declarative GitOps CD for Kubernetes — A declarative, GitOps continuous delivery tool for Kubernetes with a rich web UI for visualizing application state and deployments.
• Jenkins X — Cloud Native CI/CD Built On Kubernetes — An opinionated CI/CD platform for Kubernetes that automates pipelines, preview environments, and promotion using GitOps principles.
• OpenGitOps — CNCF Sandbox Project — The vendor-neutral, CNCF-backed project that formalizes GitOps principles: declarative, versioned and immutable, pulled automatically, and continuously reconciled.
• What is GitOps Really? — Weaveworks Blog — The original Weaveworks blog post defining GitOps that Andrey reads from during the episode. Weaveworks coined the term before shutting down in February 2024.
• gitops.tech — A community resource explaining GitOps concepts, principles, and the ecosystem of tools that implement the pattern.
• Awesome ChatOps — A curated list of ChatOps resources and tools, relevant to Andrey's comparison between GitOps and GitHub's earlier ChatOps movement driven by Hubot.

Are infra as code always the best way to go and if not when and where should you use it. Here we are trying to better understand when its god to use and when its not. 

---

Summary

In this inaugural episode, Mattias, Andrey, and Julian discuss what infrastructure as code really means, why teams adopt it, and where it can go wrong. They explore the evolution from manual server management to declarative infrastructure, the differences between configuration management and infrastructure provisioning, the growing complexity of tools like Terraform and CloudFormation, and why culture, process, and operational discipline matter as much as the tooling itself.

Key Topics

What Infrastructure as Code Actually Solves

The discussion starts with Mattias describing the shift from manually editing Apache configs over SSH to defining cloud environments in code. He recalls the progression: first managing individual servers by hand, then adopting configuration management tools like Puppet, Chef, and Ansible, and finally arriving at cloud-native tools like AWS CloudFormation that can provision entire environments declaratively.

Andrey pushes the conversation toward first principles, arguing that it is important to separate the "what" from the "how." He explains that infrastructure as code depends on having APIs — software-defined interfaces that allow infrastructure to be created and managed programmatically. Without that kind of interface, teams are limited to SSH and the manual tools they had before. The rise of public cloud providers and platforms like OpenStack finally gave teams the APIs they needed to describe infrastructure declaratively in definition files.

Configuration Management vs Infrastructure as Code

A key distinction in the episode is the difference between server configuration tools and true infrastructure as code. Andrey notes that tools like Puppet, Chef, and Ansible were originally conceived as server configuration management tools — designed to automate the provisioning and configuration of servers, not to define infrastructure itself.

He acknowledges this is a gray area, since tools like Ansible can now call AWS APIs and manage infrastructure directly. But historically, the configuration management era was about fighting configuration drift on existing servers, while the cloud era introduced the ability to declare entire environments as code. If you asked the vendors selling Chef, they would tell you Chef is "all about infrastructure as code" — but the original intent was different.

When to Automate — and When Not To

The hosts caution against automating too early. Andrey says he tends not to automate things until they genuinely need automation. If creating one cluster with a few nodes and one database is all you need, full automation may be premature. But if you know you will eventually manage hundreds or thousands, starting early makes sense.

Julian reinforces this point with a memorable gym analogy: "You go to the gym, you see Arnold Schwarzenegger lifting 200 kilos from the ground and you say, he does it, I can do it. And then you pick up the little weight and find out that if you start with 200 kilos, you're gonna break your back." His point is that infrastructure as code tools get you up and running fast — that is what they are designed for — but day two operations always come knocking. The automation itself can become a burden if you are not careful about what you automate and when.

Infrastructure as Documentation and Source of Truth

Mattias describes one of his main reasons for using infrastructure as code: knowing what is actually running. He sees the codebase as documentation and as proof of the intended state of the environment — a way to verify that what he thinks is deployed matches what is actually in the cloud.

The hosts agree with that idea, but they also point out the tension between declared state and reality. If people still make manual changes in the cloud console, the code drifts away from what is actually running. Andrey notes the problem: if undocumented manual changes are not reflected back into code, the next infrastructure deployment could recreate the original broken state — "you're back to the fire state, basically."

The Terraform Complexity Problem

Julian brings up Terraform as "the elephant in the room" and argues that it has become significantly more complex over time. He says the language started out as purely descriptive, but newer features in HCL2 — such as for loops, conditionals, and sequencing logic — have pushed it closer to a general-purpose programming language.

His concern is that this makes infrastructure definitions harder to read and reason about. Instead of simply describing desired state, users now have to mentally execute the code to understand what it will produce. Andrey agrees there is a legitimate need for this evolution — once a declarative setup grows large enough, you genuinely want loops and conditionals — but acknowledges it creates a tension between readability and expressiveness.

Declarative vs Imperative Approaches

The episode explores the difference between declarative and imperative models. Andrey explains that shell scripts are imperative — you tell the system exactly what to do, step by step — while a declarative tool lets a team state the desired outcome and rely on the platform to converge on that state.

Kubernetes is presented as a strong example of the declarative model. You submit manifests that declare what you want, and operators work to make reality match that intent — not necessarily immediately, but as soon as all requirements are fulfilled. Andrey suggests infrastructure tooling may evolve in this direction, with systems that continuously enforce declared state rather than only applying changes on demand. He gives a security example: an intruder stops AWS CloudTrail, but a reactive system — like a Kubernetes operator — detects the deviation and turns it back on automatically.

Julian adds that this is already happening. He mentions that a Kubernetes operator exists to bridge the gap to cloud APIs, allowing teams to define infrastructure resources inside Kubernetes YAML manifests and have the operator create them in the cloud. Google Cloud's Config Connector is a concrete example of this pattern, letting teams manage GCP resources as native Kubernetes objects.

Immutable Infrastructure and Emergency Changes

Andrey strongly advocates for immutable infrastructure: baking golden images using tools like Packer, deploying them as-is, and replacing systems rather than patching them in place. In that model, people should not be logging into systems or making changes manually. If you need a change, you burn a new image and roll it out. SSH should not even be enabled in a proper cloud setup.

Mattias raises a practical challenge: in real incidents, people with admin access to the cloud console often need to click a button to resolve the problem quickly. He describes his own experience — the team started with read-only production access but had to grant write access once on-call responsibilities kicked in. Andrey agrees that teams should not be dogmatic when production is on fire: "You go and do whatever it takes to put fire down." But those emergency fixes must be reflected back into code, and the team must know exactly what was changed. Otherwise, the next deployment may recreate the original problem.

Culture and Process Matter More Than Tools

One of the clearest themes in the conversation is that infrastructure as code is not just a tooling choice. Julian argues that it does not matter what technology you use if your process and culture are not aligned with security and best practices: "You can fix the technology only so much, but it's mainly about people."

Mattias describes a setup where Jenkins applies all CloudFormation changes, and every modification to the cloud goes through pull requests, code review, and change management — the same workflow used for application code. This means infrastructure changes become auditable, reviewable, and easier to track. Andrey sees this as applying development principles to infrastructure: version history, visibility into who changed what, the ability to ask someone why they made a change, and code review before changes are applied.

Guardrails for Manual Changes

Andrey shares a practical example from a previous engagement where developers had near-admin access to the AWS console and would create EC2 instances, S3 buckets, and other resources outside of Terraform or CloudFormation. To control cost and reduce unmanaged resources, the team built a system using specific tags generated by a Terraform module.

A Lambda function ran every night, scanned for resources without the required tags, posted a Slack notification saying "I found these, gonna delete them next day," and tagged them for deletion. The following night, anything still tagged for deletion was removed. This gave developers flexibility for experimentation — they could spin up resources manually and try things out — while preventing forgotten resources from becoming permanent, invisible infrastructure. It also helped keep costs under control.

Tooling Is Only the Start

Julian stresses that adopting infrastructure as code does not automatically make systems reliable, immutable, or resilient. In his view, it is "just the beginning of the journey." He warns against the myth that infrastructure as code equals immutable infrastructure — you can absolutely build stateful, mutable systems with code if you choose to.

He also pushes back on the assumption that automation always saves time, admitting with self-awareness: "I automated a task, it took me two days to automate it, and I saved barely 10 seconds of my life." His advice is to measure the actual benefit rather than being seduced by the marketing brochure. Data will tell you more about a tool's real value than excitement will.

Abstraction, Code Generation, and Developer Experience

The hosts discuss the challenge of making infrastructure easy for developers who just want a database and a connection string, not a deep understanding of DBA work and security configuration. Andrey argues that abstracting best practices away from developers saves enormous organizational time, since developer time is expensive and holds back feature delivery.

He describes a third approach beyond declarative and imperative: code generators. Large companies with resources sometimes build internal generators that take simplified YAML inputs and output fully declarative specs. This creates another level of abstraction on top of existing tools, allowing developers to be productive without needing to understand infrastructure details. It is controversial — in some ways it takes power away from people — but it can dramatically simplify the developer experience.

Pulumi vs Terraform and Community Support

Andrey introduces Pulumi as an interesting new branch of infrastructure tooling that lets teams describe infrastructure in general-purpose languages like TypeScript, Python, or Go instead of domain-specific languages like HCL. He notes that while it feels familiar to developers — you stay in your comfort zone — you still need to learn a new DSL embedded in that language. It is "not entirely like you just described infrastructure in the language you know."

Julian says he tried Pulumi and found it appealing for developers who want consistency across their codebase. But he remains cautious, arguing that "code is a liability" — referencing Kelsey Hightower's satirical GitHub project nocode ("write nothing, deploy nowhere, run securely") to make the point that less code means fewer problems. For beginners, Julian recommends starting with Terraform or the native tooling from cloud providers, mainly because the community is larger, tutorials are more abundant, and there are meetup groups where people can learn from each other. His advice is pragmatic: "Just make sure that you ditch Terraform the minute it gets in your way."

Start With the Problem, Not the Tool

Andrey repeatedly returns to the same question: what problem is being solved? He argues that teams should choose tools based on business needs and existing team capabilities, not because a tool is fashionable. "A lot of people and developers, they like shiny tools — and there's nothing wrong about that — but you always have to ask, what is the problem we are solving?"

Andrey's framing connects tool selection to team dynamics: if your team already has knowledge of a particular tool, relearning a new one just because it is trendy does not make sense. What you need is not a fancy tool but to deliver business value with the capabilities you have.

Migration, Legacy, and Incremental Adoption

The hosts acknowledge that many teams are not starting from a clean slate. Andrey points out that legacy infrastructure exists for a reason: it helped the business survive and grow. As Mattias puts it bluntly: "Legacy pays the bills."

For organizations with years of manually built systems and hybrid environments, Andrey suggests doing value stream mapping to identify the biggest pain point and tackling that first. A greenfield project can serve as a success story to demonstrate the approach before trying to transform everything. He emphasizes that coming into an organization with a shiny idea and telling people "whatever you did before was crap" is a sure way to lose allies. Technology boils down to working with people — the tools are fine, but they do not replace the people running them.

The Templating Dilemma

Mattias raises a specific frustration with infrastructure as code: templating. He likes looking at his Git repository and seeing exactly what is running, but heavy use of variables and templates means he sees placeholder names instead of actual values. This tension — between reusable, DRY templates and readable, concrete definitions — is a real challenge that the hosts acknowledge without a clean resolution.

Resilience and Recovery

Near the end of the episode, Andrey gives a concrete example of losing a Kubernetes cluster in production. Because the environment had been defined as code, the team was able to recreate it and recover in about one to two hours. Some things that were not properly documented slowed them down; with complete documentation the recovery could have been as fast as 15 to 20 minutes — mostly just waiting for AWS to provision the resources after the API calls.

Julian adds context to this: he argues that even with infrastructure as code, recreating a Kubernetes cluster and failing over traffic while maintaining service is genuinely hard. The concept is sound, but having the safety net to actually do it takes time, practice, and a lot of work. His advice for building confidence is to adopt the mentality of immutable infrastructure and get into the habit of regularly recreating things and practicing failovers.

Final Advice

Andrey recommends education first. He specifically mentions the book Infrastructure as Code by Kief Morris (published by O'Reilly, now in its third edition) as a strong foundation. His broader advice: understand the domain, define the problem clearly, ask yourself what outcome you want to deliver for the business, and let the answers to those questions guide your tool decisions.

Julian's closing thought is that in a large organization, a dedicated infrastructure team using infrastructure as code can manage everything — on-prem or cloud — with a single workflow. That team can abstract complexity so developers do not need to learn Terraform, CloudFormation, or any other tool. The specialization pays off by reducing onboarding friction and letting each team focus on what they do best.

Highlights

• Julian's gym analogy: "You go to the gym, you see Arnold Schwarzenegger lifting 200 kilos from the ground and you say, he does it, I can do it. And then you pick up the little weight and find out that if you start with 200 kilos, you're gonna break your back." His point: infrastructure as code tools make the start easy, but day two operations will humble you.
• Julian on Terraform: He calls it "the elephant in the room" and argues that it has drifted from a descriptive language toward something more like a programming language, making it harder to reason about.
• Julian on coding overhead: "Code is a liability" — referencing Kelsey Hightower's nocode project to argue that every line of infrastructure code carries long-term maintenance costs.
• Julian on no silver bullets: "There is no silver bullets. Stop dreaming. Just see how much work it takes, how complicated it is."
• Julian on automation ROI: "I automated a task, it took me two days to automate it, and I saved barely 10 seconds of my life."
• Andrey on incidents: Teams should not be dogmatic — "If your production is on fire, you go and do whatever it takes to put fire down" — then put the fix back into code afterward.
• Andrey on manual cloud resources: He describes a Lambda-based cleanup system that scanned for untagged AWS resources nightly, posted to Slack, and deleted them the next day if no one claimed them.
• Andrey on shiny tools: "A lot of people and developers, they like shiny tools — and there's nothing wrong about that — but you always have to ask, what is the problem we are solving?"
• Mattias on legacy: "Legacy pays the bills" — a reminder that existing infrastructure made the business successful, and it deserves respect during any modernization effort.
• Andrey's recovery story: After losing a production Kubernetes cluster, the team recreated everything in one to two hours because it was defined as code — and could have done it in 15-20 minutes with better documentation.

Resources

• Infrastructure as Code by Kief Morris (O'Reilly) — The book Andrey recommends as essential reading for practitioners. Now in its third edition, it covers patterns and practices for building and evolving infrastructure as code.
• Pulumi — Infrastructure as Code in Any Programming Language — The tool discussed in the episode that lets teams define infrastructure using TypeScript, Python, Go, and other general-purpose languages instead of domain-specific languages like HCL.
• AWS CloudFormation — AWS's native infrastructure as code service using declarative YAML/JSON templates, which Mattias and the team use with Jenkins for their deployment pipeline.
• GCP Config Connector — The Google Cloud Kubernetes operator mentioned in the episode that lets teams manage GCP resources as native Kubernetes objects, bridging the gap between Kubernetes and cloud APIs.
• Forseti Security (archived) — The GCP security scanning tool Julian describes that could detect policy violations (like open ports) and automatically revert changes. Originally developed by Spotify and Google, it was archived in January 2025.
• Kelsey Hightower's nocode — The satirical GitHub project Julian references: "The best way to write secure and reliable applications. Write nothing; deploy nowhere." A humorous reminder that code is a liability.
• HashiCorp Terraform — The infrastructure as code tool the hosts discuss extensively, including its evolution from a simple declarative language to the more complex HCL2 with loops and conditionals.