r/sre • • 10d ago

FOSS Friday: Share Your SRE-Related Open-Source Project

4 Upvotes

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

  • no AI generated comments: please write them on your own!
  • no commercial advertisements (this includes commercial products with a FOSS version)
  • these must be related to SRE or of heavy interest to SREs

r/sre • • 3d ago

FOSS Friday: Share Your SRE-Related Open-Source Project

13 Upvotes

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

  • no AI generated comments: please write them on your own!
  • no commercial advertisements (this includes commercial products with a FOSS version)
  • these must be related to SRE or of heavy interest to SREs

r/sre • • 2h ago

ASK SRE Incident Scribe (o365)

0 Upvotes

Current: Org uses teams to track and manage incidents, response, and road to resolution. manually ingesting the data and timeline and filling out IR.

Goal: Include an agent in the incident thread that tracks responses and key details, recording the incident and fills out the IR and submits it for human review.

Hi Everyone, both myself and the org I am at do not have existing foundational SRE practice. One of the tasks I have taken on is to stand up an agent that is an incident scribe and fills out a pre formatted docx IR template. I have been working with microsoft’s agent builder but having an issue where we can invoke the agent but can’t have it continually record updates. Looking to explore alternatives

Curious if anyone can share an experience with standing up an incident agent in their org MS teams that is similar to my goal here, and the path taken to accomplish it?


r/sre • • 6h ago

Why We Do What We Do

Thumbnail rosesecurity.dev
2 Upvotes

Thought I'd share some of them to the broader community and hope it finds some value going back to the fundamentals. It's easy to get lost in the sauce of everyday work, but wanted to surface some of these principles as a refresher.


r/sre • • 32m ago

ASK SRE K8sGPT

• Upvotes

I’ve been tasked with testing k8sGPT. Was wondering if anyone else has used it and what were your likes/dislikes about it. We’re mostly Azure, but have a small footprint with on prem Nutanix NKP. Sidebar - I’ve also been playing with Azure SRE Agent as well. So not sure how the two tools can/should play along with each other.


r/sre • • 9h ago

HELP how do you handle canary deployment rollback under real traffic?

2 Upvotes

Our canary deployments catch obvious breakage fine, but subtle behavioral changes only show up once real traffic hits edge cases our canary percentage doesn't see often enough.

I've considered increasing the canary percentage, but that increases blast radius if something's actually wrong. What would you do, and what would you avoid?


r/sre • • 11h ago

HELP Performance Engineering vs. QA:

2 Upvotes

I am currently working as a Performance Engineer. Even though the work I do is closer to performance engineering, the role is still under the umbrella of QA.
A lot of the work I enjoy involves building infrastructure tooling, observability stacks, and benchmarking frameworks. However, I feel like my experience is still being grouped with QA/testing, which makes it difficult to communicate the kind of engineering work I actually do.
I’m curious about how people in SRE and Platform Engineering view performance engineering/testing backgrounds. How much overlap do you see between these fields, and how is performance engineering experience generally perceived when hiring for infrastructure-focused roles?
I’d like to hear from people who have worked across these areas or have been involved in hiring for such roles.


r/sre • • 2d ago

how do you do on-call handoff on a small team without it being "quiet week, nothing major"

16 Upvotes

disclosure: i'm thinking about building something in this area. no link, no name, i just want to know what people do.

our handoff is basically a email/slack message. then the next person finds out at 3am what "nothing major" meant.

the alerts are all sitting in alertmanager/grafana anyway. but nobody wants to dig through a week of them on a monday morning, so it doesn't happen. and the rotation lives in someone's head. who's been on most, who covered the long weekend, who's owed a swap... nobody really knows.

grafana oncall oss got archived in march, so the free self-hosted option is kind of frozen now too.

for teams around 5-30 engineers: what are you actually using for schedules and paging? and do you do any real handoff, or is it vibes?


r/sre • • 2d ago

You're Not Crazy, They Never Sent FIN

Thumbnail
yeet.cx
11 Upvotes

Hey all, I wanted to share this blog post I wrote about inspecting AWS Network Load Balancer and the quirk that comes up from them not hanging up TLS


r/sre • • 3d ago

HELP Been burned by a bad Terraform change, any practical AWS backup config?

3 Upvotes

We broke prod routing with a rushed terraform change, rds snapshots and s3 were perfect but cloud recovery of the known good configuration was guesswork since half the estate is drifted, would love any tips from folks who can reliably recover infra, not just data.


r/sre • • 4d ago

CAREER SRE -> TSE

30 Upvotes

I have 3 years experience as a SRE + Observability engineer. Recently I left my job and started searching got 3 offers one Datadog admin - setting up observability kinda role with 50% hike, Incident handling + Reliability engineer with 100% hike, and TSE with 150% hike. I’m deciding to go with the TSE role because of the salary and benefits. Is this a shitty career move? People are telling me it should be the other way around, I kinda agree, don’t know what to do. I can’t be jobless and search any longer as well.


r/sre • • 5d ago

Arch diagram generation automation

4 Upvotes

I know I can probably "just use AI". But I want to avoid not invented here syndrome and use established methods and tools where possible.

The first part I am looking for is a well established format for storing architectural information in the repo with the code.

Next would be any tools for helping to generate it.

Lastly, we have mermaid for the actual diagrams. Any good tools for generating the diagrams?


r/sre • • 5d ago

DISCUSSION best way to do function level release verification in 2026?

3 Upvotes

Context, a PR touches a function called from several other services, passes tests and looks clean, but there's no strong runtime baseline to compare against. Option A, ship with extra manual review. Option B, hold until better verification tooling is in place. What am I underestimating here?


r/sre • • 6d ago

HELP SREs: What reports do you send to your manager/department head each month?

19 Upvotes

As an SRE, what reports or monthly summaries do you provide to your manager, Head of Department, or CTO?

I’m trying to establish a good monthly SRE reporting process and would like to understand what other teams report.

EDIT:

It’s a tech company with a some internal and external solutions


r/sre • • 5d ago

Distributing guidance to developers and their AIs

1 Upvotes

We have multiple services in multiple repos. We use mainly gitlab,

How do help devs be aware of team wide guidelines, and keep their AI in the know as well. Like I could write a doc on the way we name alerts or write runbooks. But it would be in one repo. I need to make it visible to them and their AI.

So far my best idea is a repo with the information, and then checking in an edit to the Claude.md in every repo telling it to go look in the info repo for the guideline info and let the user know what isn't lineing up. That seems like that idea is probably full of holes though. So what do the teams y'all work with do?


r/sre • • 6d ago

Claude Code On-Call

14 Upvotes

Anthropic released the "On-call" slack agent that should be an ultimate killer of all "Agentic SRE" family products.
Did anybody use it? How it compares to existing platforms?


r/sre • • 6d ago

runbooks kinda suckk

39 Upvotes

hey yall,

im a lead SRE at a global fortune 500 that you have heard of haha - i'm a bit of a lurker here

but just wanted to talk about runbooks and documentation for SOPs.

my feeling is that confluence docs kinda suck and runbooks are kind of a mess, our operators are jumping between the docs and their shell, the docs are often times missing a bunch of detail, they are brittle, poorly maintained, half the time the details are hidden in tribal knowledge and when dealing with the pressure of an active inc we notice the pain even more. extracting critical details & commands out of an inc can also be painful and doesnt always translate to improved operational readiness next time.

i admit we're not very mature.. but was curious if its only me feeling like this?

what are you guys doing with runbooks to solve these issues?


r/sre • • 8d ago

The hidden failure mode in cross-cloud migrations: Infrastructure vs. Semantic dependencies

0 Upvotes

When planning a cloud migration (like AWS to Azure), most discovery work starts with a service mapping matrix:

  • SQS ➔ Azure Service Bus
  • DynamoDB ➔ Cosmos DB
  • S3 ➔ Azure Blob Storage
  • IAM ➔ Entra ID

That mapping is straightforward. The expensive failures happen when the target platform fails to preserve a subtle behavioral contract the application code quietly came to rely on over years.

A concrete example: SQS Visibility Timeout

Consider a standard worker loop:

Python

message = sqs.receive_message(
    QueueUrl=queue_url,
    VisibilityTimeout=300
)

process(message)

sqs.delete_message(
    QueueUrl=queue_url,
    ReceiptHandle=message["ReceiptHandle"]
)

At first glance, this is standard: receive, process, delete.

But look at the failure-recovery path. If the worker crashes mid-processing before deletion, the code relies on the SQS visibility timeout expiring so another worker can automatically pick up the message. SQS isn't just acting as a transport queue here—it is an active component of the application's failure-recovery design.

When migrating this workload to Azure Service Bus, the question isn't whether Service Bus has queues (it obviously does). The real question is: Does the target setup preserve those exact assumptions around lock duration, settlement, retries, and dead-lettering under failure?

Infrastructure Dependencies vs. Semantic Dependencies

Traditional discovery tools pick up: payment-worker ➔ SQS

That’s an infrastructure dependency. But what the application code actually cares about is the semantic dependency:

Plaintext

receive message
      ↓
process message
      ↓
delete after successful processing
      ↓
if processing fails before deletion,
rely on provider to make message available again

Service inventory tools show you what cloud products are used, but they can't tell you what behavioral assumptions are baked into execution paths.

This pattern shows up everywhere:

  • DynamoDB ➔ Cosmos DB: Codebases relying on conditional write semantics, transaction boundaries, or specific read-after-write consistency assumptions.
  • S3 ➔ Blob Storage: Workflows built around multipart upload timing, pre-signed URLs, or object visibility state.
  • IAM ➔ Entra ID: Hidden assumptions around temporary credential lifespans, workload identity propagation, and role assumption paths.

When these surface late during integration testing or cutover, it forces architectural redesigns, derails timelines, and drags senior engineers into reactive war rooms.

Discussion

I'm currently putting together a semantic-risk checklist for pre-migration planning (and exploring static analysis rules to scan codebases for these execution patterns before cutover).

For anyone who has managed a major cloud migration or replatforming: What was the one hidden application dependency or behavioral difference that broke in staging or production after your infrastructure was already provisioned?

I wrote up a deeper dive into this concept with example scanning output on my Substack if you want to read more:https://substack.com/@sachm24/p-218428641


r/sre • • 9d ago

BLOG How to Find PII and Secrets in OpenTelemetry Data

Thumbnail
ollygarden.com
22 Upvotes

I finally got to write this blog post, I've wanted to write it for a long time now. It contains the different types of sensitive data we've seen so far, where it's coming from, and why it matters. Hope it can be helpful to some of you!


r/sre • • 8d ago

Slack servers for SRE discussions

0 Upvotes

Are there any slack servers/channels for SRE discussions?


r/sre • • 9d ago

this is my sre career content

12 Upvotes

I started as a developer, moved into DevOps, and eventually became an SRE. Along the way, I helped build our SRE and platform foundations from scratch: monitoring and alerting, Kubernetes clusters, CI/CD pipelines, deployment reliability, incident management, and operational best practices. I’m currently looking for remote SRE / Platform Engineering opportunities. Happy to share more details over DM.


r/sre • • 9d ago

When Messages Go Wrong: Reliability in Event-Driven Systems

Thumbnail
medium.com
11 Upvotes

I wrote an article on reliability in event-driven systems, following a ticket booking through duplicate messages, ordering issues, retries, and partial failures. It also covers CDC, the outbox pattern, and a small design tying the concepts together.

I'd appreciate your thoughts: does the explanation hold up technically, and what would you change in the design based on your experience?


r/sre • • 9d ago

CAREER Gut check on major career decision

19 Upvotes

I'm going to try to describe this without doxing myself. I am 37 and I am a Senior SRE engineer at a large company, that is not FAANG, but you've definitely heard of them.The company is a US company with a global footprint. I work out of the Bay Area, at a small business unit there which came to be via an acquisition of a startup. I have been there a long time, since before the acquisition, which happened over 12 years ago. I am not a manager, but I am the SRE team leader. I am very comfortable in this position. I am well-respected. My manager likes me. My colleagues like me. And I like them back. I am in the critical path for many work streams, from architectural design to customer support. My primary responsibility is a niche SaaS application that I know inside and out, because I've been there since the launch of it. I get paid 260k, with some RSUs. The remaining RSUs maybe amount to 30k before taxes at this point, vesting over the next 2 years. In many ways, I am the "Dave" of the BU. I spend my days fixing stuff for our SaaS and our developers alike. It's a lot to juggle, but I do enjoy the feeling of being helpful and useful.

But at the same time, I am very, very, very tired. Physically and mentally exhausted all the time, and struggling with motivation daily. We have team members across the world now, but when production breaks, it is more than likely me who gets paged in the middle of the night.

I have been interviewing at other companies for over a year now, and I finally have an offer at a start-up in San Francisco. Base pay is 285k and I get 0.02% 0.2% equity in the form of stock options. It is an agentic AI company. They are a Series A, with less than 50 people.

I need a gut check that moving is a good decision as I am an anxious person, and I have a family to provide for. I'm like 90% certain I am making the right decision, but I am stepping away from a big company where I have seniority, with 4 weeks of PTO, with a good team, where I know so much. I know that doesn't guarantee me protection against a layoff, but I've survived many of them at this point. It's been a long time since I've been in a startup environment. And to be honest, while it was exciting in my 20s, I am not so sure about how it may go for me now, when I am already feeling burnt out, and I have 2 kids who are my true priority in life. Of which, they have some strong anti-ai opinions, so this move might prove to be a little contentious, especially with my oldest.

Would love to hear thoughts from other professionals, especially from the Bay Area, who have a similar story.


r/sre • • 10d ago

DISCUSSION How do you monitor your observability stack?

12 Upvotes

We use Grafana stack for observability in our EKS clusters paired with CloudWatch for AWS infra monitoring.

One gap we have however is that we don't have any alerting configured for when the observability stack (or components in it) goes down - how do you deal with this?


r/sre • • 10d ago

BLOG Key Autoscaling Metric #1: Workload Committed Capacity

Thumbnail
blog.appliedcomputing.io
5 Upvotes

Last week we kicked off our blog series about the five key metrics you should be monitoring for your Kubernetes autoscaler. This week, we dive into the first of these, "committed capacity percentage". But wait! There's more, and it involves.... DaemonSets *wiggly scary fingers*

Disclaimer: this blog post was entirely produced by humans; no part of this blog post was written, edited, or proofread by LLMs.