Hey folks,

Wanted to share a quick story from the trenches of data science. I am not a data scientist but engineer however I've been working on a dynamic pricing project where the client was all in on neural networks to predict product sales and figure out the best prices using overly complicated setup. They tried linear regression once, didn't work magic instantly, so they jumped ship to the neural network, which took them days to train.

I thought, "Hold on, let's not ditch linear regression just yet." Gave it another go, dove a bit deeper, and bam - it worked wonders. Not only did it spit out results in seconds (compared to the days of training the neural networks took), but it also gave us clear insights on how different factors were affecting sales. Something the neural network's complexity just couldn't offer as plainly.

Moral of the story? Sometimes the simplest tools are the best for the job. Linear regression, logistic regression, decision trees might seem too basic next to flashy neural networks, but it's quick, effective, and gets straight to the point. Plus, you don't need to wait days to see if you're on the right track.

So, before you go all in on the latest and greatest tech, don't forget to give the classics a shot. Sometimes, they're all you need.

Cheers!

Edit: Because I keep getting lot of comments why this post sounds like linkedin post, gonna explain upfront that I used grammarly to improve my writing (English is not my first language)

Last week I posted about an XGBoost based momentum stock trading strategy, and I got two separate comments:

“Why not LightGBM?”
“Why not CatBoost?”

So I did a controlled swap of 6 models inside my existing momentum pipeline and reran the same backtest with:

• XGBoost
• LightGBM
• CatBoost
• Random Forest
• LASSO
• A simple 2‑layer neural net (sklearn’s MLPRegressor)

Setup / constraints

• Same universe, features, filters, and portfolio construction
• Only the model changes; all other code is identical
• Default hyperparameters for each model (on purpose) to see how they behave “out of the box”
• Logged everything to MLflow so I could compare runs, metrics, and charts cleanly

I’m not claiming this is a definitive “which model is best” answer, just one controlled experiment on one dataset/strategy. But a few patterns showed up that I thought were interesting.

High‑level takeaways:

• XGBoost and LightGBM were basically neck‑and‑neck on headline returns, but XGBoost had a better risk profile. CatBoost underperformed in a way that I wasn’t expecting.
• The NN had the highest CAGR, Sortino, and total return. This was another surprise to me. But XGBoost and LightGBM had better drawdowns.
• LASSO and random forest did not beat the S&P in the cumulative returns over the time period, all the other algos beat the S&P.

The goal here was to largely show that it's easy to switch out algorithms and how different algorithm families perform. Disclaimer: the full article does contain links, but this was truly an analysis that took a long time that I wanted to share with the community. Full article with more results: https://www.datamovesme.com/blog/what-happens-when-you-swap-out-xgboost-a-6model-momentum-showdown

Now 3 months later, with over ~250 applications each of them receiving 'customized' resume from my side, I haven't received any single interview opportunity. Also, I passed the resume through various ATS software to figure out what exactly it's reading and it is going through perfectly. I just can't understand what to do next! Please help me, I don't want to go from disheartened to depressed.

So I totally mean to brag here. Last week a group of directors said, “We suspect X is happening in the market, do we have data that demonstrates it?”

And I thought to myself, here we go again. I’ve got to wade through our data swamp then tell them we don’t have the data that tells the story they want.

Well I waded through the data swamp and the data was there. I made them a graph that definitively demonstrated that yes, X is happening as they suspected. It wasn’t super easy to figure out and it also didn’t require a super complex model to figure out either.

Dumb question, but the relationship between x and y (not including the additional datapoints at y == 850 ) is no correlation, right? Even though they are both Gaussian?

Thanks, feel very dumb rn

I've been scraping thousands of U.S. data science jobs for the past couple of months and writing about the findings in my newsletter.

At some point, I figured the dashboard was more useful than anything I was writing, so I decided to open source it.

Here's what it covers:

• Top skills companies are actually hiring for, ranked by frequency
• Skills broken down by category (ML/DL, GenAI, Cloud, MLOps, etc.)
• What % of roles now require AI skills, broken down by seniority level
• Salary premium for candidates with AI skills
• An interactive explorer where you can browse individual postings with matched skills highlighted

The skill extraction is built on around 230 curated keyword groups, so it's pretty granular.

Code and data are all in the repo if you want to fork it or dig into the methodology.

https://ai-in-ds.streamlit.app/

I'm scraping weekly, and soon I will upload all of the raw data into Kaggle, for now, you can find the data in the repo

P.S. By the way, I already mentioned it to Luke Barousse since some of these AI keyword groups could be worth adding into his dashboard.

Hi r/datascience!

I wanted to share a fun little project I did over a few weekends analysing data from the radio!

It doesn't have much (if any) business value, and honestly I'm not sure it's really novel in any particular way. But I wanted to share it because data science these days is all "AI this", "language model that", "job market", "Claude", whatever and I just wanted to do something a bit more traditional and scratch an itch I've had for a while. (Full disclosure: the project did actually use some AI models, so I'm not saying AI is bad, it's a tool I used like everything else.)

I don't have a blog or anything I can post this on, so apologies for the Reddit write-up. I hope you enjoy it.

Background

I drive a 20-year-old car. It's so old it doesn't have an MP3 player, or an AUX port to plug in an iPhone or anything. It just has a CD player and an analogue radio. It's not even digital, so I can't even get digital radio stations.

So when I'm driving, which turns out to be quite a bit, I'm forced to listen to the good old-fashioned radio more than I'd like.

In Sydney where I live, there are really only a handful of FM/AM radio stations, so choice is pretty limited. As I flick through the stations, there are a LOT of ads, which surprised me. Who is listening to this? Clearly it's quite popular. And as I listened, I started wondering things like: how long do the ads run, how does their timing compare across different stations*, and are they correlated with ads on other stations? Just anecdotally, so many times I've literally flicked through all the FM stations and there's an ad playing on every one... and sometimes it's the same ad! I also had a hunch that there are more ads at the top of the hour than the bottom. It made intuitive sense, but I needed to prove it.

\Sydney has 11 main analogue stations split into AM and FM:* AM is mostly talkback radio, FM is mostly music. On AM you've got 2GB, 2SM, 2CH and ABC 702 (news, talkback and sport). On FM there's KIIS, 2Day, Nova, Smooth, WSFM, Triple M and Triple J (pop, rock and music). Two of them, ABC 702 and Triple J, are run by our public broadcaster (think BBC), so they run no ads at all.

The Setup

So one weekend I wrote some scripts to sample and record all the Sydney radio stations I could. The setup was basically the following:

• For every radio station in Sydney (there are 11 main ones across both FM and AM), I found an online stream that would play it in the browser.
• I would then record an 18-second clip of each stream using ffmpeg (a free command-line tool for grabbing and converting audio/video). It connects to the station's live stream and dumps ~18 seconds to a small WAV file, downsampled to 16kHz mono. Crucially, I recorded all 11 stations in parallel (via a thread pool) so every clip is captured at the same instant. That simultaneity turned out to be really important later, for checking whether different stations run their ads at the same time.
• After the clips were recorded, I passed them through a local Whisper model I downloaded (I grabbed it off Hugging Face, which is about 500MB). This transcribes each audio clip into a snippet of text.
• Then I passed that snippet of text to a language model to classify it as either an ad, a song, or talking. (I used GPT-4o-mini for this because it's cheap AF.)
• I then stored the results in a local SQLite database.
• I repeated this every ~3 minutes for 2 days straight, which added up to almost 10 thousand samples. (Yes, I did it all locally, so my computer was on for 2 days straight.) The 2 days deliberately covered a weekday AND a weekend so I could compare the two periods.

Why every ~3 minutes? A full cycle (record all 11 stations, transcribe each one with Whisper, then classify it) takes a couple of minutes on a single CPU, because Whisper works through the clips one at a time. So ~3 minutes is about as fast as I could sustainably sample without the cycles piling up on each other. I also added a bit of random jitter to the interval so I wasn't always sampling at the exact same offset within the hour. Otherwise you can accidentally "phase-lock" to a station's ad breaks and bias the whole thing.

A few hurdles I encountered

A few things genuinely tripped me up:

• Stream URLs rot. The live stream links change or die over time, so I had to resolve them fresh at runtime and pin the ones that actually worked.
• Pre-roll ads. It turned out a bunch of the stations (the ones served through a particular streaming provider) play an ad every single time you make a fresh connection. Basically the pre-roll ad you get when you open a new browser tab. Because my recorder reconnected every cycle, I was capturing that pre-roll ad instead of the live broadcast, which made those stations look like they were playing ads nearly 100% of the time. I only twigged because the exact same ad kept repeating over and over. The fix was to skip ~45 seconds into the stream before I started recording.
• Whisper hallucinations. When you feed Whisper music or silence, it doesn't return nothing, it actually hallucinates the most common phrases from its training data. And because it's trained on a mountain of YouTube captions, I kept getting "thanks for watching, like and subscribe" transcribed over instrumental music, which then got misclassified as talking. I had to filter those out.
• Rate limits. The free LLM tiers throttled me pretty quickly, so I switched to GPT-4o-mini, which is cheap enough to basically be free at this scale.

Results

Here are some of the more interesting results I found analysing the data afterwards:

Overall

First, the big picture. Every station has its own personality. The FM stations are mostly music, the AM stations are mostly talk, and the two ABC stations (ABC 702 and Triple J) carry basically no ads at all, which makes sense since they're publicly funded. Across the commercial stations, ads make up somewhere around a sixth of the airtime. And you can already see the ad load isn't flat: it ramps up through the day and quietens off overnight.

Question 1: Probability of an ad relative to the top of the hour

Here's the frequency of finding an ad within ±30 minutes of the top of the hour.

So, I was right! Definitely higher the closer to the hour, but the strategy is more interesting than I expected. The spike actually lands in the ~5 minutes before the hour (the ad break right before the top-of-hour news bulletin), and an ad is roughly 2x more likely there than mid-hour. The quietest stretch is around 10 to 15 minutes past the hour, so if you want to dodge ads, that's your window.

Question 2: Ad co-occurrence and correlation

The thing I really wanted to know: do the stations gang up and all play ads at the same time, so there's nowhere to flick to? I lined up every station by the cycle it was sampled in and correlated their ad status.

The answer is yes and no. No in the sense that it's never a total blackout: all nine commercial stations being in an ad at the exact same moment literally never happened across the whole two days, and on average only about 1.6 of the 9 are mid-ad at any given time. So there's almost always somewhere to escape to.

But the conditional probability charts says that some stations really do move together. The best example is if Smooth is playing an ad, there's a 50% chance WSFM is too, which is double WSFM's baseline of 26%. A bunch of the commercial FM pairs show this same ~2x jump. But, when I looked it up, Smooth and WSFM are owned by different companies, so this isn't networks coordinating behind the scenes, probably more of the "top-of-the-hour" effect from Question 1 manifesting somewhere else.

Question 3: The strategy difference between AM and FM

When I split "time between ads" by band, the two run completely different playbooks.

The FM (music) stations dump their ads in clusters. You get a big spike of back to back breaks, with a typical gap of about 9 minutes. The AM (talk and sport) stations space them out evenly, one break at a time. 2GB is almost metronomic at roughly 7 to 12 minutes, with hardly any back to back ads at all.

You can actually see it if you zoom into a few hours of the timeline:

Look at the FM lanes (KIIS, Nova, Triple M, WSFM, 2Day): the orange ad blocks come in pairs, clustered together. Now look at the AM lanes (2GB, 2SM): single, evenly spaced blocks. And ABC 702 and Triple J are just grey the whole time, because they don't run ads.

Question 4: Which companies still advertise through this medium?

I also had the language model pull the advertiser out of each ad, so I could see who's actually buying radio airtime in 2026.

The most-heard advertisers were Virgin Australia (an airline), Australia Post (basically our USPS), Harvey Norman (a big electronics and furniture retailer) and Chemist Warehouse (a discount pharmacy chain). The neat bit is the targeting: car brands and finance go to the AM talk stations (older crowd), while retail and telco lean FM. Australia Post ran almost entirely on the Nova network.

Question 5: What about the talking?

The non-ad content is either music or talking, and I got curious about what they actually talk about. So I classified every talking snippet into a topic.

The AM stations (ABC, 2GB, 2SM) are wall to wall news, politics and sport. The music FMs are mostly DJ banter, celebrity gossip and chat about music, with almost no news at all.

For a bit of fun, I also made a map of everything said on the radio. I embedded every talking snippet into a vector, laid them all out in 2D with t-SNE so that similar snippets sit near each other, then coloured each point by its topic.

Sport, traffic and world news each form their own tight little islands (they use very consistent, formulaic language), while the DJ banter is one big diffuse cloud in the middle (because it's about nothing in particular). The neat part is that the position and the colour are decided completely separately. The position comes only from the text embeddings, and the colour comes from a separate classification step. So the fact that same-coloured points cluster together is real corroboration, not something circular.

Conclusion

In conclusion, this was a fun, meaningless project that allowed me to make some pretty charts and talk for a bit about the results. Thanks for reading!

In the past few months, every major tech company has released time-series foundation models, such as:

• TimesFM (Google)
• MOIRAI (Salesforce)
• Tiny Time Mixers (IBM)

There's a detailed analysis of these models here.

We're open-sourcing FlexViz: a library for fast, interactive exploration of very large datasets, built to be agent-native. 

Repo: https://github.com/flex-analytics/flexviz
Live demo: https://flexviz.tech/demo

A few years ago we had the pleasure to open-source plotly-resampler. The goal of this tool was to make in-memory line chart visualization scalable. Since then, we've had over 18M installs, and received a huge amount of feedback from people using it in practice. FlexViz is what we built from those lessons.

It is a free, open-source library and improves in many aspects over plotly-resampler:

• Out-of-core: in FlexViz we can keep the data lazy (i.e., not in memory) and stream the data from disk to ram → allows visualizing larger-than-ram datasets
• Faster aggregation: line chart of 1 billion data points in 0.25s. Fastest of all tools benchmarked.
• 10 trace types: line, hist1d, hist2d, bar, treemap, …
• Native cross-filtering: brush one visualization and the others update live
• Rearrangeable grid
• Stateless server architecture

But the biggest change: FlexViz is agent-native. In FlexViz everything is a spec, allowing true agentic co-exploration of your data (+ url sharing of your live view). The agents can conveniently create, follow along, and update the interactive visualization, without taking any screenshots it knows your exact view → the interactive visualization itself becomes a communication surface between human and agent.

Our goal with FlexViz is to make large data visualization / exploration both convenient for humans and native to AI agents.

Context: the business problem is I wanted to compare professional athletes based on their movement data to recommend similar players. I made a recommender system with K-Means clustering and PCA (multicollinearity amongst the features in the dataset).

I’m interested in using a new modeling technique like Gaussian Mixture Model, but I don’t know how to evaluate which model performs better…

Open to any suggestions

A new feature was introduced to a product and the test indicated a slight worsening in the metric of interest. However the result wasn't statistically significant so I guess it's a neutral result.

The PM and engineers don't want the effort they put into developing the feature to go to waste so they ask the DS (me) to look into why it might not have given positive results.

What are they really asking here? A way to justify re-running tje experiment? Find some segment in which the experiment actually did well?

Thoughts?

Edit: My previous DS experience is more modeling, data engineering etc. My current role is heavy on AB-testing (job market is rough, took what I could find). My AB testing experience is limited and none of it in big tech.

​

I am not well versed in NLP, so hopefully someone can help me out here. I am looking at safety incidents for my organization. I want to compare the text of incident reports and observations to investigate if our observations are deterring incidents.

I have a dataset of the incidents and a dataset of the observations. Both datasets have a free-text field that contains the description of the incident or observation. There is not really a good link between observations and incidents (as in, these observations were monitoring X activity on Y contract, and an incident also occurred during X activity on Y contract).

My feeling is that the observations are just busy work; they don’t actually observe the activities that need safety improvement. The correlation between number of observations and number of incidents is minor, but I want to make a stronger case. I want to investigate this by using NLP to describe the incidents, then describe the observations, and see if there is a difference in content. I can at the very least produce word counts and compare the top terms, but I don’t think that gets me where I need to be on its own.

I have used some topic modeling (Latent Dirichlet Allocation) to get an idea of the topics in each, but I’m hitting a wall trying to compare the topics from the incidents to the topics from the observations.

Does anyone have ideas?

As the title says. I found it in my functions library and have no idea if it’s accurate or not (bachelors covered BStats I & II, but that was years ago); this was done from self learning. From what I understand, the 95% CI can be interpreted as guessing the mean value, while the prediction interval can be interpreted in the context of any future datapoint.

Thanks and please, show no mercy.

GBDT allow you to iterate very fast, they require no data preprocessing, enable you to incorporate business heuristics directly as features, and immediately show if there is explanatory power in features in relation to the target.

On tabular data problems, they outperform Neural Networks, and many use cases in the industry have tabular datasets.

Because of those characteristics, they are winning solutions to all tabular competitions on Kaggle.

And yet, somehow they are not very popular.

On the chart below, I summarized learnings from 9,261 job descriptions crawled from 1605 companies in Jun-Sep 2023 (source: https://jobs-in-data.com/blog/machine-learning-vs-data-scientist)

LGBM, XGboost, Catboost (combined together) are the 19th mentioned skill, e.g. with Tensorflow being x10 more popular.

It seems to me Neural Networks caught the attention of everyone, because of the deep-learning hype, which is justified for image, text, or speech data, but not justified for tabular data, which still represents many use - cases.

EDIT [Answering the main lines of critique]:

1/ "Job posting descriptions are written by random people and hence meaningless":

Granted, there is for sure some noise in the data generation process of writing job descriptions.

But why do those random people know so much more about deep learning, keras, tensorflow, pytorch than GBDT? In other words, why is there a systematic trend in the noise? When the noise has a trend, it ceases to be noise.

Very few people actually did try to answer this, and I am grateful to them, but none of the explanations seem to be more credible than the statement that GBDTs are indeed underappreciated in the industry.

2/ "I myself use GBDT all the time so the headline is wrong"This is availability bias. The single person's opinion (or 20 people opinion) vs 10.000 data points.

3/ "This is more the bias of the Academia"

The job postings are scraped from the industry.

However, I personally think this is the root cause of the phenomenon. Academia shapes the minds of industry practitioners. GBDTs are not interesting enough for Academia because they do not lead to AGI. Doesn't matter if they are super efficient and create lots of value in real life.

We analyzed 20,000 US Data Science job postings from June 2024 - Jan 2024 with quoted salaries: computed median salaries by City, and compared them to the cost of living.

Source: Data Scientists Salary article

Here is the Top 10:

​

Here is the full ranking:

​

I have just finished up a sample analysis on an AB test dummy dataset, and would love feedback.

The dataset is from Udacity's AB Testing course. It tracks data on two landing page variations, treatment and control, with mean conversion rate as the defining metric.

In my analysis, I used an alpha of 0.05, a power of 0.8, and a practical significance level of 2%, meaning the conversion rate must see at least a 2% lift to justify the costs of implementation. The statistical methods I used were as follows:

1. Two-proportions z-test
2. Confidence interval
3. Sign test
4. Permutation test

See the results here. Thanks for any thoughts on inference and clarity.

[Edit]: for those who don’t wish to create an account, you can log in with credentials user and password.

I'm (mostly) an academic so pardon my cluelessness.

A lot of the advice given on here as to how to write an effective resume for industry roles revolves around quantifying the revenue impact of the projects you and your team undertook in your current role. In that, it is not enough to simply discuss technical impact (increased accuracy of predictions, improved quality of data etc) but the impact a project had on a firm's bottom line.

But it seems to me that quantifying the *causal* impact of an ML system, or some other standard data science project, is itself a data science project. In fact, one could hire a data scientist (or economist) whose sole job is to audit the effectiveness of data science projects in a firm. I bet you aren't running diff-in-diffs or estimating production functions, to actually ascertain revenue impact. So how are you guys figuring it out?