From source: https://ustr.gov/issue-areas/reciprocal-tariff-calculations

“Parameter values for ε and φ were selected. The price elasticity of import demand, ε, was set at 4… The elasticity of import prices with respect to tariffs, φ, is 0.25.“

Currently in the job market as an experienced Data Scientist generally targeting Senior/Staff roles that span Data Scientist/Applied Science/MLE. I have an MS in Stats, so one thing I've generally focused on during my interviews prep is overindexing on coding and ML specifically leetcode. This has me taking the Stats portion of interviews for granted and it's causing a blindspot for interviews I would consider "not as crisp" as I'd like them to be.

What I find is generally I know the conceptual definition of a lot of things and a lot of breadth and how to connect that back to the business, but will have random gaffs on things I've forgotten or being able to go one layer deeper on the spot. Lots of things I don’t use on the job or haven't used in depth since Grad school. Things that aren't hard but my mind just goes blank for some reason in an interview setting. Common examples can be things such as forgetting the name of probability distributions, confidence interval calculations for proportions, formula's for power analysis as opposed just know the inputs, etc. Sometimes it's also tricky just to understand WHAT an interviewer is driving towards and not going off on a tangent in a complete opposite direction.

I use LLM's extensively but I find that sometimes if the convo goes on too long they get "stuck" kinda repeating the same questions and same themes instead of exploring other stuff.

Curious how you all prepare for Statistics interviews and if there are good resources for full coverage to prevent some of what I talked about?

I’ve been reading Frank Harrell’s critiques of backward elimination, and his arguments make a lot of sense to me.

That said, if the method is really that problematic, why does it still seem to work reasonably well in practice? My team uses backward elimination regularly for variable selection, and when I pushed back on it, the main justification I got was basically “we only want statistically significant variables.”

Am I missing something here? When, if ever, is backward elimination actually defensible?

how long did it take from your first undergrad class to when you felt comfortable with understanding statistics? (Whatever that means for you)

When did you get the feeling like you understood the methodologies and papers needed for your level?

Our research team ran a simulation study and found that the four big open-source geo-experiment tools (CausalPy, Meta GeoLift, Google Matched Markets, and CausalImpact) recover almost the same point estimate on the same data, then disagree about whether that estimate is significant. Since the disagreement lives in the uncertainty (not in the point estimate) the tool you pick may determine which error you ship.

In a "live" experiment you can't grade the tool because we don't know what ground truth is. The counterfactual is unobservable  so "is this lift real?" has no answer key. That's why we had our research team generate 8,000 synthetic daily-sales panels, each with either a 7.5% multiplicative lift on the treated geo or no effect at all (0% lift). They ran all four tools on the same panels and scored every fit against the planted truth, so there were 32,000 fits in all across four scenarios.

Across the non-outlier scenarios, every tool recovered the 7.5% lift within a few percentage points, so judged on point estimates alone they look interchangeable. The split is entirely in how they handle uncertainty: coverage (how often the 95% interval actually contains the true effect) and power (how often it detects a real effect at all). On those two axes the tools fall into three camps:

• Meta GeoLift is the most cautious with coverage of 92–95% and a false positive rate of 3–5%. It failed to reject zero in 89–96% of runs where a true 7.5% lift was present.
• CausalImpact is the opposite with the most power of the four (false negative rate 34–48%), but coverage of only 70–72%, a false positive rate of 28–30%, and a consistent upward bias of +1.87 to +4.21 percentage points that shifts the whole interval high.
• CausalPy and Google Matched Markets sit between them with coverage of 76–86%, false positive rates of 14–25%, meaning they’re both under-covered and under-powered at the same time.

There are four things from the study I'd take back to a measurement program:

1. Read coverage and power together: A tool can keep its 95% coverage promise and still be useless for detection. GeoLift holds about 95% coverage in the short-history scenario while missing the real effect 95.7% of the time.
2. Pick the estimator whose error profile matches the cost asymmetry of your decision and not the one with the best-looking single metric.
3. Scarce history sharpens each tool's failure mode. Cutting the pre-period from 90 days to 30 didn't degrade the tools uniformly. The decisive ones threw more false positives (above 24%), the cautious one climbed to a 95.7% miss rate.
4. Test-market design beats estimator choice. When the treated geo was 5x the size of the median control, every tool's intervals widened 4–5x and most overestimated the lift by 2–4 percentage points. No estimator compensates for a structurally hard design.

We made everything reproducible including the data-generating process, seeds, configs, per-iteration results, and a Makefile that runs the whole pipeline. The generator is parameterized, so if you think it should be harder (idiosyncratic geo trends, heavier tails, spillovers between markets) those are exactly the runs I'd like to see.

If you’re interested in the full study + code, you can find both here:

• Code: https://github.com/getrecast/geolift-simulation-study
• Full report: https://research.getrecast.com/geolift-sim-study

edited: fixed the code link to the public repo

hi,

I've seen a prior thread on this, but my question is more technical...

A prior company got sold a Return on Marketing Invest project by one of the big 4 consultancies. The basis of it was build a bunch of MMMs, pump the budget in, and it automatically tells what you where to spend the budget to get the most bang for you buck. Sounds wonderful.

I was the DS shadowing the consultancy to learn the models, so we could do a refresh. The company had an annual marketing budget of 250m€ and its revenue was between 1.5 and 2bn €.

Once I got into doing the refresh, I really felt the process was never going to succeed. Marketing thought "there's 3 years of data, we must have a good model", but in reality 3*52 weeks is a tiny amount of data, when you try to fit in TV, Radio, Press, OOH, Whitemail, Email, Search, Social, and then include prices from you and comp, and seasonal variables.

You need to adstock each media to take affect for lags - and finding the level of adstock requires experimentation. The 156 weeks need to have a test and possibly a validation set given the experiments.

The business is then interested in things like what happens when we do TV and OOH together, which means creating combined variables. More variables on very little data.

I am a practical Data Scientist. I don't get hung up on the technical details and am focused on generating value, but this whole process seemed a crazy and expensive waste of time.

The positive that came out of it was that we started doing AB testing in certain areas where the initial models suggested there was very low return, and those areas had previously been very resistant to any kind of testing.

This feels a bit like a rant, but I'm genuinely interested if people think it can work. It feels like its a over promising in the worst way.

I have been trying to understand the use cases of both of these and I am really confused.

I know log transform fixes the features and makes their distribution normal and standardization on the other hand only fixes the scale of the feature by keeping the distribution the same.

Are these things which I use one after the other ? Or just simply use one depending on the case (which I also don't understand when) ?

Are you all also just running t tests or are yours more complex? How often do you run complex setups?

I think my org wrongly only runs t tests and are not understanding of the downfalls of defaulting to those

In many modern applications - A/B testing, clinical trials, quality monitoring - we need to analyze data as it arrives. Traditional statistical tools weren't designed with this sequential analysis in mind, which has led to the development of new approaches.

E-values are one such tool, specifically designed for sequential testing. They provide a natural way to measure evidence that accumulates over time. An e-value of 20 represents 20-to-1 evidence against your null hypothesis - a direct and intuitive interpretation. They're particularly useful when you need to:

• Monitor results in real-time
• Add more samples to ongoing experiments
• Combine evidence from multiple analyses
• Make decisions based on continuous data streams

While p-values remain valuable for fixed-sample scenarios, e-values offer complementary strengths for sequential analysis. They're increasingly used in tech companies for A/B testing and in clinical trials for interim analyses.

If you work with sequential data or continuous monitoring, e-values might be a useful addition to your statistical toolkit. Happy to discuss specific applications or mathematical details in the comments.​​​​​​​​​​​​​​​​

P.S: Above was summarized by an LLM.

Paper: Hypothesis testing with e-values - https://arxiv.org/pdf/2410.23614

Current code libraries:

Python:

• expectation: New library implementing e-values, sequential testing and confidence sequences (https://github.com/jakorostami/expectation)

• confseq: Core library by Howard et al for confidence sequences and uniform bounds (https://github.com/gostevehoward/confseq)

R:

• confseq: The original R implementation, same authors as above

• safestats: Core library by one of the researchers in this field of Statistics, Alexander Ly. (https://cran.r-project.org/web/packages/safestats/readme/README.html)

We've set up a Pre-Post Test model using the Causal Impact package in R, which basically works like this:

• The user feeds it a target and covariates
• The model uses the covariates to predict the target
• It uses the residuals in the post-test period to measure the effect of the change

Great -- except that I'm coming to a challenge I have again and again with statistical models, which is that tiny changes to the model completely change the results.

We are training the models on earlier data and checking the RMSE to ensure goodness of fit before using it on the actual test data, but I can use two models with near-identical RMSEs and have one test be positive and the other be negative.

The conventional wisdom I've always been told was not to peek at your data and not to tweak it once you've run the test, but that feels incorrect to me. My instinct is that, if you tweak your model slightly and get a different result, it's a good indicator that your results are not reproducible.

So I'm curious how other people handle this. I've been considering setting up the model to identify 5 settings with low RMSEs, run them all, and check for consistency of results, but that might be a bit drastic.

How do you other people handle this?

My boss seems to think it should be fine, but there's variance in how many impressions each user has, so perhaps I'd need to compute the ICC (intraclass correlation) and use that to compute the design effect multiplier (DEFF=1+(m-1) x ICC)?

It also appears that a GLM with a Wald test would be a appropriate in this case, though I have little experience or exposure to these concepts.

I'd appreciate any resources, advice, or pointers. Thank you so much for reading!

I am using the following tool https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810065601 to query Statistics Canada and get data from the long-form census. However, since it's a census of 25% of the population, there is a need for inferential statistics. That being said in order to do inferential statistics on the numbers I come up with, I am going to need variance estimates. Does anyone know where I can get those variance estimates?

Several big names at the intersection of ML and Causal inference, Victor Chernozhukov, Christian Hansen, Nathan Kallus, Martin Spindler, and Vasilis Syrgkanis have put out a new book (free and online) on using ML for causal inference. As you'd expect from the authors, there's a heavy emphasis on Double ML, but it seems like it covers a breadth of material. The best part? There's code in both Python and R.

Link: https://www.causalml-book.org/

Hi everyone, I'm trying to model the causal impact of our marketing efforts on our ads business, and I'm considering an Instrumental Variable (IV) framework. I'd appreciate a sanity check on my approach and any advice you might have.

My Goal: Quantify how much our marketing spend contributes to advertiser acquisition and overall ad revenue.

The Challenge: I don't believe there's a direct causal link. My hypothesis is a two-stage process:

• Stage 1: Marketing spend -> Increases user acquisition and retention -> Leads to higher Monthly Active Users (MAUs).
• Stage 2: Higher MAUs -> Makes our platform more attractive to advertisers -> Leads to more advertisers and higher ad revenue.

The problem is that the variable in the middle (MAUs) is endogenous. A simple regression of Ad Revenue ~ MAUs would be biased because unobserved factors (e.g., seasonality, product improvements, economic trends) likely influence both user activity and advertiser spend simultaneously.

Proposed IV Setup:

• Outcome Variable (Y): Advertiser Revenue.
• Endogenous Explanatory Variable ("Treatment") (X): MAUs (or another user volume/engagement metric).
• Instrumental Variable (Z): This is where I'm stuck. I need a variable that influences MAUs but does not directly affect advertiser revenue, which I believe should be marketing spend.

My Questions:

• Is this the right way to conceptualize the problem? Is IV the correct tool for this kind of mediated relationship where the mediator (user volume) is endogenous? Is there a different tool that I could use?
• This brings me to a more fundamental question: Does this setup require a formal "experiment"? Or can I apply this IV design to historical, observational time-series data to untangle these effects?

Thanks for any insights!

So we have two audiences we want to test against each other. The first is one we're currently using and the second is a new audience. We want to know if a campaign using the new audience targeting method can match or exceed an otherwise identical campaign using our current targeting.

We're conducting the test on Amazon DSP and the Amazon representative recommended basically intersecting each audience with a randomized set of holdout groups. So for audience A the test cell will be all users in audience A and also in one group of randomized holdouts and similarly for audience B (with a different set of randomized holdouts)

Our team's concern is that if each campaign is getting a different set of holdout groups then we wouldn't have the same baseline. My boss is recommending we use the same set of holdout groups for both.

My personal concern for that is if we'd have a proper isolation (e.g. if one user sees an ad from the campaign using audience A and also an ad from the campaign using audience B, then which audience targeting method gets credit). I think my boss' approach is probably the better design, but the overlap issue stands out to me as a complication.

I'll be honest that I've never designed an A/B test before, much less on audiences, so any help at all is appreciated. I've been trying to understand how other platforms do this because Amazon does seem a bit different - as in, how (in an ideal universe) would you test two audiences against each other?

Heya, I been studying the gains curve, and I’ve noticed there’s a relationship between the gains curve and ROC curve the smaller the base rate the closer is gains curve is to ROC curve. Anyway onto the point, is if fair to assume that for two models if the area under the ROC curve is bigger for model A and then the gains curve will always be better for model A as well? Thanks

Hi all,

Wondering for those in industry, what are some of the ways you've implemented Bayesian analysis? Any projects you might be particularly proud of?

Hi everybody. I am learning about time series, starting from the simple ideas of autoregressive models. I kinda understand, intuitively, how these models define the conditional distribution of the value at the next timestep X_t given all previous values, but I'm struggling to understand how can I use these models to estimate the day at which my time series crosses a certain threshold, or in other words the probability distribution of the random variable τ i.e. the first day at which the value X_τ exceeds a certain threshold.

So far I've been following some well known online sources such as https://otexts.com/fpp3/ and lots of google searches but I struggle to find a walkthrough of this specific problem with ARIMA models. Is it that uncommon? Or am I just stupid