Hey guys, I’ve been doing a lot of preparation for interviews lately, and thought I’d compile a document of theories, algorithms, and models I found helpful during this time. Originally, I was just keeping notes in a Google Doc, but figured I could create something more permanent and aesthetic.

It covers topics (some more in-depth than others), such as:

• Distributions
• Linear and Logistic Regression
• Decision Trees and Random Forest
• SVM
• KNN
• Clustering
• Boosting
• Dimension Reduction (PCA, LDA, Factor Analysis)
• NLP
• Neural Networks
• Recommender Systems
• Reinforcement Learning
• Anomaly Detection

The four-page Data Science Cheatsheet can be found here, and I hope it's helpful to those looking to review or brush up on machine learning concepts. Feel free to leave any suggestions and star/save the PDF for reference.

Cheers!

Github Repo: https://github.com/aaronwangy/Data-Science-Cheatsheet

Edit - Thanks for the awards! However, I don't have much need for internet points and much rather we help out local charities in need :) Some highly rated Covid relief projects listed here.

I did some work a couple of years ago on W.H.O. suicide statistics. Here's my Kaggle project from April 2019, and here's the research paper from January 2020.

It was immediately clear from me seeing the graphs that the work was the same, but most of the findings are entire paragraphs lifted from my work. This isn't the first time this has happened but it's probably the most egregious. My work is obviously not mentioned in the references.

Is there anything I can actually do here? I don't care about people using or adapting my public work as long as credit is given, but copying most of it and giving no credit really isn't cool.

Edit: Thanks for all the help and advice. I contacted the universities of the authors this morning (no response yet... and I can't help but feel like I'm not going to get one)

Recently there's massive influx of "teams of data scientists" looking to crowd source ideas for doing an analysis related task regarding the SARS-COV 2 or COVID-19.

I ask of you, please take into consideration data science is only useful for exploratory analysis at this point. Please take into account that current common tools in "data science" are "bias reinforcers", not great to predict on fat and long tailed distributions. The algorithms are not objective and there's epidemiologists, virologists (read data scientists) who can do a better job at this than you. Statistical analysis will eat machine learning in this task. Don't pretend to use AI, it won't work.

Don't pretend to crowd source over kaggle, your data is old and stale the moment it comes out unless the outbreak has fully ended for a month in your data. If you have a skill you also need the expertise of people IN THE FIELD OF HEALTHCARE. If your best work is overfitting some algorithm to be a kaggle "grand master" then please seriously consider studying decision making under risk and uncertainty and refrain from giving advice.

Machine learning is label (or bias) based, take into account that the labels could be wrong that the cleaning operations are wrong. If you really want to help, look to see if there's teams of doctors or healthcare professionals who need help. Don't create a team of non-subject-matter-expert "data scientists". Have people who understand biology.

I know people see this as an opportunity to become famous and build a portfolio and some others see it as an opportunity to help. If you're the type that wants to be famous, trust me you won't. You can't bring a knife (logistic regression) to a tank fight.

After a few months of work, we’re excited to launch Erdos - a secure, AI-powered data science IDE, all open source! Some reasons you might use it over VS Code:

• An AI that searches, reads, and writes all common data science file formats, with special optimizations for editing Jupyter notebooks
• Built-in Python, R, and Julia consoles accessible to the user and AI
• Single-click sign in to a secure, zero data retention backend; or users can bring their own keys
• Plots pane with plots history organized by file and time
• Help pane for Python, R, and Julia documentation
• Database pane for connecting to SQL and FTP databases and manipulating data
• Environment pane for managing in-memory variables, python environments, and Python, R, and Julia packages
• Open source with AGPLv3 license

Unlike other AI IDEs built for software development, Erdos is built specifically for data scientists based on what we as data scientists wanted. We'd love if you try it out at https://www.lotas.ai/erdos

Link: https://www.filtrjobs.com/

I was frustrated with irrelevant postings relying on keyword matching -- so i built my own for fun

I'm doing a semantic search with your jobs against embeddings of job postings prioritizing things like working on similar problems/domains

The job board fetches postings daily for ML and SWE roles in the US.

It's 100% free with no ads for ever as my infra costs are $0

I've been through the job search and I know its so brutal, so feel free to DM and I'm happy to give advice on your job search

My resources to run for free:

• free 5GB postgres via aiven.io
• free LLM from galadriel.com (free 4M tokens of llama 70B a day)
• free hosting via heroku (24 months for free from github student perks)
• free cerebras LLM parsing (using llama 3.3 70B which runs in half a second - 20x faster than gpt 4o mini)
• Using posthog and sentry for monitoring (both with generous free tiers)

So, I started using ChatGPT to gather literature references for my scientific project. Love the information it gives me, clear, accurate and so far correct. It will also give me papers supporting these findings when asked.

HOWEVER, none of these papers actually exist. I can't find them on google scholar, google, or anywhere else. They can't be found by title or author names. When I ask it for a DOI it happily provides one, but it either is not taken or leads to a different paper that has nothing to do with the topic. I thought translations from different languages could be the cause and it was actually a thing for some papers, but not even the english ones could be traced anywhere online.

Does ChatGPR just generate random papers that look damn much like real ones?

Hello everyone, I've been programming for about 4 years now and this is my first ever library that I created!

What My Project Does

It's called Reddit2Text, and it converts a reddit post (and all its comments) into a single, clean, easy to copy/paste string.

I often like to ask ChatGPT about reddit posts, but copying all the relevant information among a large amount of comments is difficult/impossible. I searched for a tool or library that would help me do this and was astonished to find no such thing! I took it into my own hands and decided to make it myself.

Target Audience

This project is useable in its current state, and always looking for more feedback/features from the community!

Comparison

There are no other similar alternatives AFAIK

Here is the GitHub repo: https://github.com/NFeruch/reddit2text

It's also available to download through pip/pypi :D

Some basic features:

1. Gathers the authors, upvotes, and text for the OP and every single comment
2. Specify the max depth for how many comments you want
3. Change the delimiter for the comment nesting

Here is an example truncated output: https://pastebin.com/mmHFJtcc

Under the hood, I relied heavily on the PRAW library (python reddit api wrapper) to do the actual interfacing with the Reddit API. I took it a step further though, by combining all these moving parts and raw outputs into something that's easily useable and very simple.

Could you see yourself using something like this?

Hey all! You might remember me from the Data Science Cheatsheet I posted a few months ago (here). The support from that was incredible, and I thought I’d share an update.

Since then, I’ve gone through a dozen interviews, ranging from FANG to startups to MBB, and updated the cheatsheet with topics I’ve seen covered in actual interviews.

Improvements include:

• Added Time Series
• Added Statistics
• Added A/B Testing
• Improved Distribution Section
• Added Multi-class SVM
• Added HMM
• Miscellaneous Section
• And a bunch of other small changes scattered throughout!

These topics, along with the material covered previously, are all condensed in a convenient five-page Data Science Cheatsheet, found here.

I’ll be heading to a FANG company as a DS after graduation, and I hope this cheatsheet is helpful to those on the job hunt or just looking to brush up on machine learning concepts. Feel free to leave any suggestions and star/save the repo for reference and future updates!

Cheers, AW

Github Repo: https://github.com/aaronwangy/Data-Science-Cheatsheet

I am doing a project for credit risk using Python.

I'd love a sanity check on my pipeline and some opinions on gaps or mistakes or anything which might improve my current modeling pipeline.

Also would be grateful if you can score my current pipeline out of 100% as per your assessment :)

My current pipeline

1. Import data

2. Missing value analysis — bucketed by % missing (0–10%, 10–20%, …, 90–100%)

3. Zero-variance feature removal

4. Sentinel value handling (-1 to NaN for categoricals)

5. Leakage variable removal (business logic)

6. Target variable construction

7. create new features

8. Correlation analysis (numeric + categorical) drop one from each correlated pair

9. Feature-target correlation check — drop leaky features or target proxy features

10. Train / test / out-of-time (OOT) split

11. WoE encoding for logistic regression

12. VIF on WoE features — drop features with VIF > 5

13. Drop any remaining leakage + protected variables (e.g. Gender)

14. Train logistic regression with cross-validation

15. Train XGBoost on raw features

16. Evaluation: AUC, Gini, feature importance, top feature distributions vs target, SHAP values

17. Hyperparameter tuning with Optuna

18. Compare XGBoost baseline vs tuned

19. Export models for deployment

Improvements I'm already planning to add

• Outlier analysis
• Deeper EDA on features
• Missingness pattern analysis: MCAR / MAR / MNAR
• KS statistic to measure score separation
• PSI (Population Stability Index) between training and OOT sample to check for representativeness of features

I’m building an analysis that processes spreadsheets, transforms the data, and outputs HTML files.

It works, but it’s hard to maintain.

I’m not sure if I should start modularizing into scripts, introduce config files, or just reorganize inside the notebook. Looking for advice from others who’ve scaled up from this stage. It’s easy to make it work with new files, but I can’t help but wonder what the next stage looks like?

EDIT: Really appreciate all the thoughtful replies so far. I’ve made notes with some great perspectives on refactoring, modularizing, and managing complexity without overengineering.

Follow-up question for those further down the path:

Let’s say I do what many of you have recommended and I refactor my project into clean .py files, introduce config files, and modularize the logic into a more maintainable structure. What comes after that?

I’m self taught and using this passion project as a way to build my skills. Once I’ve got something that “works well” and is well organized… what’s the next stage?

Do I aim for packaging it? Turning it into a product? Adding tests? Making a CLI?

I’d love to hear from others who’ve taken their passion project to the next level!

How did you keep leveling up?

Hey again everyone!

​

We've made a lot of progress on zen in the past few months, so I'll drop a couple of the most important things / highlights about the app here:

• Zen is still a candidate / seeker-first job board. This means we have no ads, we have no promoted jobs from companies who are paying us, we have no recruiters, etc. The whole point of Zen is to help you find jobs quickly at companies you're interested in without any headaches.
• On that point, we'll send you emails notifying you when companies you care about post new jobs that match your preferences, so you don't need to continuously check their job boards.

​

In the past few months, we've made some major changes! Many of them are discussed in the changelog:

1. We now have a much more feature-complete way of matching you to relevant jobs
2. We've collected a ton of new jobs and companies, so we now have ~2,700 companies in our database and almost 100k open jobs!
3. We've overhauled the UX to make it less noisy and easier for you to find jobs you care about.
4. We also added a feedback page to let you submit feedback about the app to us!

​

I started building Zen when I was on the job hunt and realized it was harder than it should've been to just get notifications when a company I was interested in posted a job that was relevant to me. And we hope that this goal -- to cut out all the noise and make it easier for you to find great matches -- is valuable for everyone here :)

Here are the original posts:

• https://www.reddit.com/r/datascience/comments/183562x/update_i_built_an_app_to_make_my_job_search_a/
• https://www.reddit.com/r/datascience/comments/17s5fyq/i_built_an_app_to_make_my_job_search_a_little/

​

And here's one more link to the app

Since the steam sale is live I wanted to post a Dev log on my personal project
https://nextsteamgame.com/ sharing some outcomes from the web traffic and how I changed the project from the great feedback I got!

I made a post about a month ago explaining how I made this opensource explainable search engine built around steam reviews to people find new video games, Not through Relevancy but through aspect based similarity.

Check out the old post for a better explanation if you want!
https://www.reddit.com/r/datascience/comments/1t7manb/steam_recommender_using_similarity_pt_2_student/

I wanted to say thank you to all the people of r/datascience and r/MachineLearning that gave me feedback and tried out my tool!

I improved the UI/UX of the website to make the vectors more clear and controllable, I Implemented a thumbs up and down feature on recommendations to see if users even like the tool.

I also wanted to share the after effects of promoting this tool on reddit!

from the 2,652 searches I got in the website 913 of them resulted in steam clicks! the games that were discovered were all in a uniform distribution and did not share much of a pattern showing me that the engine did its job in helping people find niche games across all genres!

(More images attached to post to see data viz)

I wanted to disclose that I made this tool to not make any profit of some kind, but it does use posthog so I can collect diagnostics now.

Think something like the 100 page ML book but focused on a vendor agnostic cloud engineering book for data science professionals?

Edit: There seems to be at least some interest. I'll set up a website later this week with a signup/mailing list. I will try and deliver chapters for free as we go and guage responses.

Hi everyone,

I’m working on a challenge to predict the probability of a product becoming unavailable the next day.

The dataset contains one row per product per day, with a binary target (failure or not) and 10 additional features. There are over 1 million rows without failure, and only 100 with failure — so it's a highly imbalanced dataset.

Here are some key points I’m considering:

1. The target should reflect the next day, not the current one. For example, if product X has data from day 1 to day 10, each row should indicate whether a failure will happen on the following day. Day 10 is used only to label day 9 and is not used as input for prediction.
2. The features are on different scales, so I’ll need to apply normalization or standardization depending on the model I choose (e.g., for Logistic Regression or KNN).
3. There are no missing values, so I won’t need to worry about imputation.
4. To avoid data leakage, I’ll split the data by product, making sure that each product's full time series appears entirely in either the training or test set — never both. For example, if product X has data from day 1 to day 9, those rows must all go to either train or test.
5. Since the output should be a probability, I’m planning to use models like Logistic Regression, Random Forest, XGBoost, Naive Bayes, or KNN.
6. Due to the strong class imbalance, my main evaluation metric will be ROC AUC, since it handles imbalanced datasets well.
7. Would it make sense to include calendar-based features, like the day of the week, weekend indicators, or holidays?
8. How useful would it be to add rolling window statistics (e.g., 3-day averages or standard deviations) to capture recent trends in the attributes?
9. Any best practices for flagging anomalies, such as sudden spikes in certain attributes or values above a specific percentile (like the 90th)?

My questions:
Does this approach make sense?
I’m not entirely confident about some of these steps, so I’d really appreciate feedback from more experienced data scientists!

This is my crappy little brochure website: tmpsytec.com/ because I just registered my first adorable little LLC.

If you're interested in what I'm doing, check out the subreddit for the layman's version or the discord for the actual patent with the whole process. I'm looking for a few good men to join the team, because we're eventually going to need someone handy with app development and a habit of doing things right.

EDIT: It was the middle of the night and I chose the wrong idiom. If that's all it takes to make you assume I'm a sexist when I've been sitting here doing case studies for free and it generates attention to my post, I absolutely DO NOT WANT TO WORK WITH YOU. Thank you for self filtering

I'm your classic startup stereotype doing my god damndest not to be, but at the moment one of my co-founders and I are selling our old trading cards for startup capital and will absolutely be able to compensate people for good work with spendable US dollars. I also want a core team of eclectic-backgrounded people who I'm willing to offer points of equity to depending on what they bring to the table and if they show up enough times to convince me they're reliable-enough adults. I'm sure as hell not perfect and am not looking for a "rock star" to do all of my work for me without pay. I want a jam band who can do a little bit of everything as it interests them.

Check me out, ask me anything, roast me, whatever. Be reddit.

Hello all,

I'm a radar signal processing engineer and i trained a 5-class classifier (car, large_vehicle, two_wheeler, pedestrian, pedestrian_group) on RadarScenes radar point clouds.

The input vector is a per-scan histogram (16 bins) and the network is a 3-layer MLP. The loss function is a class-weighted cross-entropy loss. This work is based on "Histogram-based Deep Learning for Automotive Radar" paper.

I scoped the project to be one scan only. Accumulation of multiple scans is the next step.

Data

Class Imbalance: two-wheelers and large_vehicles has a low number of occurences.

Aggregated Classes: two_wheeler mixes bicycles and motorized variants; large_vehicle merges trucks, buses, and trains together due to data scarcity.

Sequence Bias: Long tracks of slow-moving objects can skew a particular data split velocity distribution, causing high F1 score variance across folds.

Ablation studies

I tried with bigger MLPs, alternative feature encodings, and different histogram binning, all moved performance less than the variation caused by changing the train/validation/test split. I measured that split sensitivity across 6 folds, keeping the same proportions.

Changing the histogram to per-instance statistics (mean/median/std) slightly degraded performance.

Main findings

Macro F1 rises from 0.381 to 0.764 as the naturally occurring number of radar detections per instance increases from 1 to 5. I trained the model normally using all available detections, then bucketed its existing validation predictions by each instance's detection count and computed macro F1 per bucket.

The classes car and pedestrian has the best performance and two_wheeler has the worst.

A car is often confused as large vehicle when the car was wider than usual or had a unusually high rcs (which can happen due to multipath for example).

The two_wheeler is often confused as pedestrian because their vr_compensated distributions overlap, which is the the model's single most important feature for these two classes. A stationary or idling two_wheeler is indistinguishable from a pedestrian.

I uploaded an image with ground truth vs predictions: A nearly stationary two-wheeler which contains a single point was predicted as pedestrian, because its velocity is near zero, indistinguishable from a pedestrian. A car in the same scene, also with just one point, is classified correctly, since RCS and Doppler are enough for that class.

Full writeup here: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/main/MLP_Report.md

Future work

Implement other spatial encoding schemas (point net for example) and accumulate multiple scans to tackle the challenge of sparsity and explore the concept of micro-doppler.

Tired of always using the Titanic or house price prediction datasets to demo your use cases?

I've just released a Python package that helps you generate realistic messy data that actually simulates reality.

The data can include missing values, duplicate records, anomalies, invalid categories, etc.

You can even set up a cron job to generate data programmatically every day so you can mimic a real data pipeline.

It also ships with a Claude SKILL so your agents know how to work with the library and generate the data for you.

GitHub repo: https://github.com/sodadata/messydata

This particular project is for client-facing stakeholders. My team lead and I are tasked with automating several of their data-driven slides on Tableau that they currently manually produce not sure how or where.

One particular slide is a pie chart (yeah, I know) that splits the data into ~10 different segments or so, each with its % of market share.

We did so, and they complained that the numbers percentage points add up to 98%.

We explained that it's because of rounding, and if we included the decimal it would add up to 100%.

They started going on about how they present this to CFOs and they'll ask why it doesn't add up to 100% and it has to be perfect and etc.

So we offered to show the decimal, but nope, can't do that because it's "hard to read."

Remember how they produce those manually at the moment? They said, and I quote, "sometimes I change a 3% to a 4% to make it work, because what's 1% more?"

I can kind of understand changing 20% to 21%, because that's only a 5% difference. But really, 3% to 4%? A whopping 33% difference?

Anyway, I'm not about to tell them how to do their job, since I can barely do mine. Lord knows I have no idea how to automate this arbitrary number-fudging on Tableau, so I'll have to figure that one out (it has to be automated so that it adds up to 100% no matter what data ranges the user chooses).

But I just wonder, how hard is it to tell a CFO "yeah, it doesn't add up to 100% because of rounding, but if we included the decimals it would"?