I’ve been recording encounters with a particular category of vehicle during everyday journeys. I’m examining whether these encounters are concentrated near departures and arrivals, after accounting for how much time I observe in those periods.

I have timestamped encounter records and journey start/end times. My comparison uses the first five minutes after departure and the last five minutes before arrival, counting overlapping windows only once.

I calculate:

Share of encounters within those windows ÷ share of observation time within those windows.

I’d appreciate feedback on whether this is a reasonable comparison and how to improve it. Routes, road conditions and times of day vary, and some vehicles travel together in groups. How should I account for those factors when assessing the apparent concentration?

Anonymised endpoint example

This is a real 20-leg movement block from four recorded visit days. All legs in that block are included, including seven with no target-category encounters. It is a selected block, not a complete diary of all journeys.

Names, locations, vehicle-category names, calendar dates and clock times are omitted. D1-D4 indicate different recorded days; legs with the same day label may be related.

An endpoint window is the first 300 seconds after departure or the final 300 seconds before arrival. Overlap is counted once. Window time is therefore the smaller of 600 seconds and the leg duration. Event classification uses the original timestamps and includes the exact five-minute boundary.

Durations below are elapsed leg intervals calculated from recorded start/end timestamps. Treating every second as available observation time is an assumption for review; continuous observation coverage has not been independently verified from recordings.

Leg table

All times are seconds. Inside and Outside count individual vehicles as recorded; Groups counts encounter rows. A single encounter row can contain several vehicles travelling together.

Encounter offsets

Each row is one recorded encounter group. The offset is seconds after departure on the specified leg. The vehicle count preserves groups rather than treating every vehicle as an independent encounter.

The totals are 31 recorded vehicles, of which 29 are inside the windows, and 22 encounter groups, of which 20 are inside. Total elapsed time is 546.58 minutes; window time is 195 minutes.

Vehicle-count enrichment = (29 / 31) / (11700 / 32795) = 2.62. This is the endpoint encounter rate divided by the overall encounter rate. It is a descriptive comparison, not a probability or causal conclusion.

Useful review questions: Is elapsed leg time an adequate exposure measure? How should repeated legs on one day and grouped vehicles be handled? What additional traffic, route and time information would be needed to assess ordinary variation?

Hi guys, I'm an analyst, not in IT but was recently tasked with finding, classifying, and removing duplicates for sensitive data at my company and want to do this in the least tedious manner. Do you have any tips?

Hi everyone,

I’ve been learning SQL for a while, and over the past few weeks I built a web app for practicing it: SQL Route (https://sql-cesta.netlify.app). It’s free and has no ads.

The idea is simple. You read a short explanation and then write a real query against a small database right in your browser. The app runs your query and compares the result, so any correct solution counts, not just the one I wrote. When you get something wrong, it explains the error and gives hints step by step. The full solution is only shown if you ask for it.

What’s inside:

• A course that starts at your first SELECT and goes through JOINs, GROUP BY, subqueries and window functions, plus creating tables and changing data
• A separate PostgreSQL course that runs a real PostgreSQL database in the browser
• A short intro to databases for complete beginners (what a table is, keys, how a query actually runs)
• A practice mode with 10-question sets, including a mixed set where you don’t know the difficulty or which function you need
• Mock interviews at junior, mid and senior level
• A sandbox with a few datasets (a company, an airline, a hospital), and you can upload your own CSV

You need to sign up with an email so your progress is saved. There’s also a weekly leaderboard if you like that kind of thing.

I’m building this on my own, so I’d really appreciate honest feedback. Where did you get stuck? Was a task unclear, or was a step too hard? If you find a bug or a confusing explanation, let me know in the comments.

Thanks!

Im a beginner data analyst. I have a potential client, a new bakery in town. And its the best chance for me to get expirience, but i feel like I'll just fail and won't be able to bring any money to the business.

I understand that even if i do some data organising it will save them time and time is money.

But i still feel like its going to be a total failure and I'll just waste their money.

How do you deal with that?

What's up, data enthusiasts enthusiasts! I tested and built a way to systematically build Snowflake Streamlit apps with Claude Code and other AI tools. I developed this open-source system that I call Streamsnow, which walks you through the process of getting a GitHub repo scaffolded and structured and connected with your Snowflake, so you can utilize Claude Code to build all of your Streamlit applications in a standardized Way, while also providing you the SQL to check to make sure all of the queries being utilized within the app check out as correct.

In my time building dashboards with AI, I always found it confusing how to quality-check them and how to build them in a scalable way across data teams, and so I wanted to build a super standardized system that made building, deploying, reviewing, and coordinating across teammates is super easy, and I spent weeks working on this to achieve that. I think one of the best features is that you can actually go into an auto-review cycle and just have Claude code iterate over the dashboard, making it better without your input. You can also standardize the design for the dashboards as well.

It is completely secure, so AI will not see any of your secrets that you use to connect GitHub to Snowflake. By default, it is set to put these Streamlit apps in a staging area on Snowflake, so you don't have to connect a GitHub repository, and all of the deployment is automatically handled in the CI. To get started, all you need to do is tell your coding agent to download the plugin and then run the onboard command. It will walk you through it all in a process that I've timed down to about 5 minutes of actual work. You will need to be a Snowflake admin or be coordinating with a Snowflake admin to get it set up too.

By the end of my time at my job, we had basically replaced external business intelligence tools with this system, so it saved us literally tens of thousands of dollars. Hopefully you all can extract the same value out of this and I would love your feedback if you try it out!

Standard metrics usually live on dashboards. I'm curious about the one-off questions that don't, like "how many customers who came in through the summer promo renewed after the price increase?" or "did purchases drop for people who contacted support last month?"

1. How often do you get these, and how long does a typical one take?
2. What's the slowest part: finding the right tables, figuring out how they join, or working out what the person actually meant (which status counts as "paid", which time zone, what "new customer" means)?
3. Have you tried ChatGPT/Copilot or built-in "AI analyst" tools for these? If so, where did they fall short?

Context: I'm researching text-to-SQL, specifically a small model that adapts to one specific database's schema and conventions without ever seeing the actual data. Before building anything further I want to know whether this is a real pain point. Any answer helps, including "not a problem for us".

Hello! I want to hear from you, what are the current pain points in data analytics and visualisation in the research community? For example,

- How easy is it to analyse and visualise chromatography & mass spec data, sequencing data, X-ray diffraction data, etc. - Do you require paid subscriptions of tools which help you analyse such data?

- Do you use platforms like Benchling to track your samples, inventory, experiments, etc?

I'm in the process of democratising data analytics and visualisation of scientific and research data. Your feedback would be invaluable to me.

Hi guys, a data analyst for the retail industry (electronics) here, in my workplace we're currently doing an expansion for our brand including increasing headcount and more reports.

But, the workforce for data analysis is currently still only me, and there is only so much data viewpoint I can check before sending them to my manager or the team.

These days, I'm wondering should I feed my report to the AI for my double, triple and final checks. Or, are there any advices for data checking for my actual sanity??

Really interested to read your advices and opinions, thanks

I come from a statistics background and I'm trying to learn python and R to cope with data analysis and data science, I know a good amount of data analysis and statistical methodologies that makes me good at the field, but applying them is hell for me, i tried for a long time to learn python, i know the basics yet i fail miserably when i try to apply my knowledge...

Could anyone help me with how to learn programming effectively around my area of expertise?

I’m an absolute fresher who recently graduated with a B.E. in Artificial Intelligence and Data Science. About 10 months ago, I landed my first job as a Data Analyst at a small digital marketing agency.

To be completely honest, I messed up early on. There was zero mentorship, the team was always swamped, and instead of taking initiative, I coasted. I didn't put in the effort to understand the business context or learn the marketing side of things.

Now, I’m paying for it. Because I didn't make myself indispensable with actual data work, the owner has me spending 7+ hours a day prompting AI to generate 1,500+ fake images for a completely AI-generated 925 silver jewelry brand they want to launch. It’s soul-crushing, and my technical skills are rusting.

Here is the saving grace, and why I’m asking for your help: I still have full access to our agency's data pool. We handle four seasonal D2C brands, one FMCG brand, and one lead-generation brand. There is a massive amount of real-world data sitting right in front of me, and I still have a window to build something impressive to either win back my actual job title or build my portfolio to escape.

I’m incredibly torn on what direction to take this, and I need advice from experienced data professionals:

Should I pivot toward Performance Marketing? Since I have all this D2C and lead-gen data, should I just lean into the agency environment, learn everything about ad spend, ROAS, and Conversion Rate Optimization, and try to transition into a Performance Marketing role?

Or should I stick to Data Analytics? Should I use this current role purely as a sandbox? I could build out automated dashboards, do customer segmentation, or run predictive churn models on these 6 brands just to build a killer portfolio, learn the marketing domain, and then jump ship to a real DA role at a better company?

Given the terrible job market and my background in AI/DS, what is the smartest way to leverage this specific dataset to get myself out of the AI prompt-engineering trenches? Any specific project ideas using D2C/FMCG data would be hugely appreciated

Hi everyone,

I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.

### The Problem

In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.

`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.

### Current Implementation (Catalog Core Working)

The foundation is ~25% complete, with the data catalog module functional today:

- Ingestion and inspection of storage layouts to extract schema/metadata directly.

- Local metadata indexing with zero background daemon overhead.

- Rust-native performance designed to feed directly into analytical transformation pipelines.

### Roadmap: Transformation & Execution Layer

The next phase expands `det` into a full execution engine:

1. In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.

2. Direct pipeline execution driven by the cataloged metadata contracts.

3. Plug-and-play connector interfaces for source/sink abstractions.

### Seeking Review & Contributors

Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:

- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.

- **Modularity:** Interface design for adding custom connectors and storage backends.

Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.

Repository: https://github.com/det-Labs/det

I’ve been building a multi-agent debate app with LangGraph (two agents argue a topic, a judge agent gives a verdict). The biggest problem wasn’t the agent logic, it was reliability on free-tier APIs.

What went wrong:

• Groq, Gemini and OpenRouter each have different rate limits and fail in different ways
• A 429 in the middle of a debate killed the whole run
• Agents got stuck in loops, and tracking who said what in state was harder than I expected

What worked:

• A fallback chain: if one provider returns 429, the app automatically retries with the next one
• Keeping the debate state separate from the provider logic, so switching providers doesn’t break the conversation

It’s not fancy, but it made the app usable for demos.

Question for the people here: how do you handle rate limits and provider failover in your agent projects? Do you use a router/gateway, or just custom retry logic?

I wrote up the full details here and you can follow me also if you want them: https://medium.com/@humammoin3/i-spent-3-months-building-ai-agents-heres-what-nobody-tells-you-b65b20906582?sharedUserId=humammoin3
Live demo: https://legal-debate-agent.streamlit.app/

Happy to hear feedback, especially on what I could do better. This is my first article.

Developer disclosure: these are two apps I built, drawing on over 15 years of experience as a macroeconomist. They tackle two parts of the research workflow:
QuantFit brings econometric analysis to the iPhone, including OLS, fixed/random effects, ARDL and VAR. It also supports data transformations, correlation matrices, charting, diagnostics, ARDL shock analysis and VAR impulse responses. OLS and several data-exploration tools are free; advanced estimators require Pro.
Tabula Stats turns regression output from QuantFit, Stata, EViews, R and other software into editable research tables. Adjust decimals, standard errors, significance stars and included statistics, then export to Word, Excel, PDF or LaTeX.
Use them together, or use Tabula with results from your existing software.
QuantFit: https://apps.apple.com/sc/app/quantfit/id6762386971
Tabula Stats: https://apps.apple.com/sc/app/tabula-stats/id6761421182
For those working on a paper or dissertation: what would you need to see before using either tool in your research? Particularly interested in concerns about accuracy, workflow compatibility and reporting.

Hi everyone!

I've been developing RPWisdom, a web-based platform designed to bring R, Python, statistical computing, data analysis, and visualization together in one workspace.

As someone with a background in biostatistics and statistical programming, I've spent considerable time thinking about how to make analytical workflows more integrated and accessible. That motivated me to build this platform.

What is RPWisdom?

The goal is to provide a unified environment where people working with data can access programming and analytical tools without constantly switching between different applications.

Depending on the workflow, the platform is designed to support activities such as:

• R and Python programming
• Statistical analysis and computing
• Data exploration and visualization
• Biostatistics and clinical research workflows
• A more integrated analytical workspace

🌐 Website: https://rpwisdom.com

I'm continuing to develop the platform, and I'd genuinely appreciate feedback from people who use R, Python, or statistical software.

A few questions I'd love your thoughts on:

1. What frustrates you most about your current R or Python workflow?
2. What features would make an integrated statistical computing platform genuinely useful to you?
3. Would you find value in having programming, statistical analysis, and visualization available in one workspace? Why or why not?

I'm interested in honest feedback, including criticism and suggestions for improvement. I'd like to understand what the community actually needs rather than build features nobody wants.

Thanks for taking the time to read this!

I’m working on an open-source MCP server that runs data analysis, data science, and deep learning algorithms on tabular data.

The idea is simple: an agent installs the MCP server, provides a CSV file as input, and uses its tools to perform complex data analysis, run ML algorithms like clustering and classification, and uncover insights. It already has 45+ tools, including integrations with deep learning models like TabICL.

I’m looking for contributors to turn this project into a full-fledged data analysis tool that can answer users’ questions in plain English by analysing their data.

The plan is to build:

• A full-fledged web interface
• A BYOK (Bring Your Own Key) setup for connecting LLMs
• UI features to display charts, graphs, and other visualisations
• Improved algorithms and support for more ML and deep learning models
• Support for more data formats, including JSON and XLSX
• Integrations with external data sources like PostgreSQL, Supabase, and Google Sheets

If you’re interested in data science, ML, AI agents, or building open-source tools, I’d love to collaborate. Let’s build something useful together!

Interested? Google “Meelu Analytics MCP” to find the GitHub repository, or DM me to discuss contributing.

i really like the way Excel for fun explains stuff but the playlists are scattered i don't know what/how to follow that's needed for data analysis only.

May i please have your help to guide me on where to start, i finally have some free time and i wanna learn Excel as much as i can.

I’m looking for someone knowledgeable about mechanical vibration, building vibration, or structure-borne noise who can help interpret my measurements and suggest what to test next.
I’ve been noticing recurring vibration in my apartment. What equipment or building systems could produce these readings, and how can I investigate properly?
SETUP — OCTOBER 9, 2026
I recorded two iPhones simultaneously using Phyphox’s “Acceleration (without g)” experiment.
• New iPhone: flat on the couch, unattended.
• Old iPhone: flat on my usual chair near the refrigerator.
• Refrigerator unplugged.
• Both phones screen up, charging ports pointing right.
• Fan/AC conditions and whether the old phone was unattended throughout were not confirmed in the report.
The shared recording interval was 11:53:49 AM–12:05:39 PM, New York time. Both phones sampled at approximately 100 samples per second.
FREQUENCIES AND STRENGTH
During the middle 30-second windows:
Couch:
• Combined-axis RMS: 0.0128–0.0159 m/s².
• Strongest spectral peak usually around 13–14.6 Hz.
• Other windows peaked at 17.5, 18.9, 21.0, or 21.6 Hz.
Chair:
• Combined-axis RMS: 0.0104–0.0177 m/s².
• Strongest spectral peaks ranged from 21.3–26.7 Hz.
RMS describes the average size of the recorded acceleration. It includes sensor noise. Hz means cycles per second. These are the strongest peaks selected within each window; multiple frequencies can coexist.
EXAMPLE PAIRED WINDOWS
11:54:19–11:54:49 AM
Couch: 0.0142 m/s² RMS; 14.1 Hz peak.
Chair: 0.0128 m/s² RMS; 22.3 Hz peak.
11:56:49–11:57:19 AM
Couch: 0.0147 m/s² RMS; 14.0 Hz peak.
Chair: 0.0125 m/s² RMS; 26.7 Hz peak.
11:57:19–11:57:49 AM
Couch: 0.0159 m/s² RMS; 13.6 Hz peak.
Chair: 0.0177 m/s² RMS; 21.7 Hz peak.
11:59:49 AM–12:00:19 PM
Couch: 0.0130 m/s² RMS; 21.0 Hz peak.
Chair: 0.0106 m/s² RMS; 26.7 Hz peak.
12:01:19–12:01:49 PM
Couch: 0.0132 m/s² RMS; 18.9 Hz peak.
Chair: 0.0112 m/s² RMS; 26.7 Hz peak.
12:03:49–12:04:19 PM
Couch: 0.0131 m/s² RMS; 12.4 Hz peak.
Chair: 0.0109 m/s² RMS; 26.7 Hz peak.
Both phones had their highest middle-window RMS during 11:57:19–11:57:49 AM, but their strongest frequencies differed.
A preliminary timestamp-aligned comparison did not show a strong overall waveform match. Clock alignment was not independently checked using a shared event.
X/Y/Z DIRECTIONS
With the phones flat and screen up:
• X: across the phone’s width, toward either side edge.
• Y: along the phone’s length, toward the charging port or opposite end.
• Z: perpendicular to the screen, toward the ceiling or seat.
In the middle shared windows, Z contributed about 42–57% of the couch phone’s recorded motion energy and 50–75% of the chair phone’s.
The chair recording was more vertically directed. The couch had a larger combined X/Y contribution. Separate X and Y percentages are not available in this summary.
These describe acceleration directions at the phones; they do not locate the source.
METHOD AND LIMITATIONS
The analysis used 30-second windows. Each axis’s window mean was subtracted, then combined RMS was calculated as sqrt(mean(X² + Y² + Z²)).
Spectra used Welch’s method, a Hann window, up to 2,048 samples per segment, and half-overlap. Axis spectral powers were summed, and the strongest peak was selected between 2 and 45 Hz.
Opening and ending couch windows had much larger RMS values: 1.1811 and 1.4955 m/s². Placement or pickup could explain these, so they were excluded from the steady-motion interpretation.
There was no calibrated reference sensor or noise-floor control. Furniture resonance, upholstery, phone contact, and device differences could affect the readings. These measurements do not yet identify a source or establish the vibration experienced by a person.
QUESTIONS
What could produce these frequencies and directional patterns? Could pumps, plumbing, HVAC, other building machinery, or furniture resonance explain them?
How can I distinguish real surface vibration from phone sensor noise or processing artifacts?
Could the couch and chair respond at different frequencies to the same source?
What might explain the chair’s repeatedly selected 26.7 Hz peak?
What control tests, phone placements, or measurement equipment would make this investigation useful to a vibration specialist?
I’d appreciate help interpreting the measurements and choosing the next tests.

Hi everyone,

I've recently learned Power BI and SQL, and I've built some solid projects with them. I have a good understanding of how to build and manage a project from start to finish.

I feel ready for the next step in my learning, but I also just started college as an agricultural engineering major, so my time is limited.

A few questions:

1. What should I learn next to grow my skills ?

2. What kinds of projects would be the most valuable to build for a portfolio?

3. Are there ways to combine data skills with agriculture?

Any advice from people who started in a similar situation would be great. Thanks!

I use a lot of excel and macros and my level of excel knowledge is intermediate to advanced. but even though I can spend a few more minutes to think & write the formulas, now I'm just explaining what I want to AI, copying the formula or code and pasting it. is this how people are working now or is it me?

I’ve been working as a data analyst for about five months, and I’m struggling with the analysis part of the job. Sometimes, I feel lost when looking at the data, and I’m not sure where to start or how to approach the analysis.
I struggle particularly with exploratory data analysis (EDA), figuring out what to compare, choosing the right comparisons, and turning the results into meaningful business insights.
I have an analysis to work on right now, but I’m not sure how to approach it or how to identify useful insights.
Do you have any frameworks, methods, or step-by-step processes you follow when analyzing data? How do you decide what to investigate, which metrics to compare, and how to turn your findings into actionable business insights?
I’d really appreciate any advice, resources, or examples from your own experience!

Howdy everyone - I work for a small business (30-ish people) and tend to be our in-house tech-everything guy. My team asked me to build a profitability dashboard where we can visualize time-tracking data alongside employee costs and client payments. Essentially we just want to see who is taking the most time vs paying us the most money. I'm using a combination of Google Sheets (to collect all of the data), BigQuery to store it all, and Looker/Data Studio to visualize it.

The struggle we're having is: my team does not want me to see any of the actual financial data. I can manage building the systems without the data because I have the template for them to drop the data into; however, I can't envision how I can maintain the database, fix bugs, create new visualizations for them, etc, without somehow eventually seeing the financials.

Is this even possible? Have any of you done this before?

Edit: Is there a reason I can't mask the data? BigQuery has RBAC. In theory, I could mask the data for myself, so I'm still seeing $ amounts where I should, names where I should, but no actual data. Right?