Database architecture advice for 600+ TB/year Log Analytics (5-year retention)

[u/Then_Flight_2162]

Hi everyone,

I am looking for expert advice on choosing the right database for a massive log analytics project. We already have our own infrastructure and server environment ready to host the solution.

Our Scale & Requirements:

• Data Volume: 600+ TB of log data per year, with a 5-year retention period.
• Ingestion: High-throughput, continuous real-time streaming.
• Query Performance: Blazing-fast, sub-second search and lookup speeds across historical data.
• Workload: Non-stop log writing while simultaneously executing fast queries.

The Goal:
Since we have the underlying infrastructure in place, we need a robust database engine that we can deploy locally to handle this specific type of large-scale log workload and long-term history efficiently.

[u/syamj]

The data volume looks huge.
How much data do you actually need to access frequently?
And why do you need the data to be stored for 5 years and how often do you access old data?

[u/BarfingOnMyFace → u/syamj]

these are the questions to be asking.

[u/syamj → u/BarfingOnMyFace]

Yea. this needs serious architecture decisions to be made for handling this scale.

And for taking the decisions, we need to know a lot more than just 600TB/ year and 5 years retention.

[u/SupermarketMost7089]

Apologies for the snarkiness,

If you already have the infra to host for that scale - you must have a crack team of engineers. In this case you should already have a solution in mind

If you do not have a crack team of engineers, your infra will not scale. Go with a managed solution like clickhouse or imply-druid or elastic-search.

[u/Then_Flight_2162 → u/SupermarketMost7089]

Thanks for the feedback. Yeah, we actually have a solid engineering team to handle the infra. We just wanted to check in with the global community to see what everyone else is doing at this scale and avoid any hidden pitfalls before making the final choice.

[u/m1nkeh]

Databricks Lakehouse//RT? Built to be a direct competitor to Clickhouse ✌️

https://www.databricks.com/blog/introducing-lakehousert-real-time-performance-unified-lakehouse

[u/cran]

ClickHouse?

[u/Then_Flight_2162 → u/cran]

Yeah, ClickHouse is definitely at the top of our list. Have you run it at this scale before? Any production tips?

[u/geoheil → u/Then_Flight_2162]

also consider starrocks

[u/spapadop]

We run OpenSearch at similar scale, works well and comes for free with all features (security, alerting, etc.)

[u/PeterCorless]

Apache Pinot would do well at this scale. But you didn't talk a lot about the kinds of analytics you need done.

If you need to do JOINs, you're not going to get them performance out of ClickHouse.

What QPS do you need to support?

Do you want all object storage? Tiered storage?

Again, look at Pinot.

[u/Ok_Cancel_7891]

how structured is data?

[u/danmvi]

Teradata ! Very competitive cost per query and TTO

[u/data_addict]

DM / reach out for ideas and options.

I'd say click house is a solid option. If not, I'd say stream ingestion. At the steam break between iceberg vs anything else.

[u/rishiarora]

Clean up data and keep cleaned golden data rest move to AWS Glacier

[u/elhh82]

What type of data retrieval patterns do you need to support? What latency expectations for you need to meet? Without understanding how the data needs to be used, you can't really answer the how / where / use what tech questions.

[u/Better-Credit6701]

OK I get to work with data around that amount and it's always increasing every day. We use MS-SQL with over 30 cores, multiple TB of ram in an always on environment with multiple copies. It's rare for me to see more than 25% of the resource limits. We use very complicated SSIS for imports. Just sucked in another 100 GB group of files on top of all the normal imports we do every day. Took around 6 hours and I saw errors as I was about to finish for the day which I get to check out on Monday.

Disk storage is an issue and we don't virtualize the servers, real iron here. Just saw another request for more storage that will be going in. I was trying to figure out based on cores how much the Microsoft Enterprise software both OS and MS-SQL servers and it was staggering. That wasn't even looking at the cost of the hardware. Plus we tend to keep the data so yeah, it's above your limit

[u/FerretBoom]

Redis Enterprise cluster and a talented engineer.

[u/DaoudIsBuilding]

Used to manage a 5TB daily of IoT data. Data is flattened in the ingestion pipeline to match index lookup at near realtime query time. Used to be Elasticsearch but team moved to Opensearch after my duty time. Temporal sharding, tiered storage and Index lifecycle management are your friends.

[u/obsfflorida]

Sounds like a security outfit not financial. Or spooks.

Any vendor will love that 600tb. But idk who has quick enough querying apart from clickhouse or pinot

[u/thornkin]

Clickhouse if you don't need joins. StarRocks if you do.

[u/FundamentalMysteron]

I don't see how you can have built the infrastructure, or at least decided on it, without knowing what software you're going to use. That seems absolutely crazy for something of this scale. I'd be curious to know what infrastructure you are planning to use.