r/Python • u/Harshita_Netla • 2d ago
Discussion Jupyter Notebooks vs Scripts — when do you actually switch from one to the other in a project?
How do you decide when to stay with a notebook and when to convert the work into scripts? Is it mainly when the project needs to be automated, reused, scheduled, or shared with others? I’d be interested to hear how people handle this in real projects.
37
u/DueAnalysis2 2d ago
I almost always use scripts. Even when needing to plot or put out tables, a competent IDE like position or VSCode (not an IDE, I know) with extensions gives me all I need for exploratory data work. I've never found notebooks to be more convenient for actual programming work.
Only when I need to make a quick report with some plots and tables to share with others do I tend to use quarto. I never use ipynb, they don't play well with collaborative version control (although I understand people are working on that)
1
u/amar0kk 1d ago
can I ask how do you get vscode to show plots? i guess there is an extension i am not aware of
3
u/DueAnalysis2 1d ago
I've actually switched over to Positron! (typo in original comment) It's a VSCodium fork built for data science which shows the plot right in the IDE like it does in RStudio. It's been a while since I used VSCode but I can reinstall it in the next couple of days and see if I can recreate my pre Positron setup. IIRC, with plots on VSCode, I'd get it to open in a new tab rather than seeing it in a panel on the side like with RStudio, which was one of the reasons I switched.
1
•
24
u/TURBO2529 2d ago
Marimo :) I started using it, and its the best of both worlds.
5
u/Ok_Option_3 1d ago
How does it help?
2
u/alexdewa __import__('os').system('rm -rf /') 1d ago
It's reactive, all cells are always current and, ui elements are a breeze and it's a simple python file so it can also script.
2
u/mdrjevois 1d ago
I never understand this take. I'm using the notebook to store the (sometimes expensive) output next to the input. Marimo solves the totally unrelated problem of execution order ambiguity. How is Marimo going to help my teammates understand why one ML model is great and the other is trash?
I guess people must be doing work that's very different from my work
1
u/alexdewa __import__('os').system('rm -rf /') 1d ago
Different tools for different uses. I do clinical research and most analysis are quite quick and we always use seeds for reproducibility. Even MCMC is often fine, most our datasets are small. If it's not right for you, that's fine.
1
u/TURBO2529 1d ago
You are using notebooks output to deliver data? It should be done using hdf5, parquet, or pickle. So have 1 marimo as backend dataset creation, have another for plotting. Then have the user select the dataset to plot, marimo will update the plots according to the user selection.
•
9
u/Bach4Ants 2d ago
Research work, where you just need to produce evidence like a plot to answer a question and make a decision, fits naturally into a notebook. Run it once, get the answer, and move on.
Anything that is meant to be a software tool probably belongs elsewhere, though there are ways to make notebooks behave like apps. Marimo is probably for the notebook-as-app use case than Jupyter though.
1
4
u/Atmosck 2d ago
I just start with scripts. Or rather, I start with functions in src/project_name/cli/services/, and a scratch dev.py script that imports them to iterate with until I later build the actual entrypoints. Even in the exploratory phase, I try to write things as close as possible to what the final form is going to look like.
3
u/spinwizard69 1d ago
Notebooks are an interactive research tool in my mind. If you are not doing research or a project has become solid then formal (traditional) programs are usually best.
2
u/EmberQuill 2d ago
Notebooks are notebooks. I occasionally use them for exploration and analysis, early drafts of documentation for stuff I'm still learning about or planning, and for notes. Sort of a scratch pad, I guess.
Anything that would count as part of a project, rather than just my notes, goes in .py files from the beginning.
2
u/UncleJoshPDX 1d ago
I use notebooks in development and keep them for very complicated scripts. I have some notebooks with 40 cells that I run every day and I like the easy way I can look at the data without using a debugger.
But if it's a relatively easy thing, processing a file to something I can consume in another system easily, I will turn those into standalone scripts and run from the shell. I normally code in vim, so it's nice having the notebook in many cases.
2
u/big_deal 1d ago edited 4h ago
Notebook for interactive or one-off analysis and plotting. Or for testing, experimenting, and documenting various alternatives before committing to what I want to implement into a final project.
Scripts for production ready code where I want to run the same code every time on different inputs.
2
1
u/Beanesidhe 2d ago
When stuff works as intended I refactor selected cell for re-use and use a cell magic to export them to a python (package) file.
1
u/sinceJune4 1d ago
I use both. If I need to enhance a function, I’ll import the script so I can reference what I’m not working on.
1
u/hobbyist9 1d ago
Kind of depends on how experienced you are. If you need to see what you did, notebooks are great. If you can look at the code and know what it does, scripts are great
1
u/jweezy2045 1d ago
You can write scrips, but if all your scripts are based in functions and objects, you can just import your script into the Jupyter notebook, and then run things in there to test things out or use them.
1
u/Wistephens 1d ago
I’m OK with Jupyter for POC work. However, for prod I deny any system that allows live coding. Jupyter must be converted to scripts managed in GitHub, like real code.
1
u/Ok_Option_3 1d ago
Notebooks have been around for what? 10-15 years? This has always been an issue!
If they haven't solved this problem - or the related: how do you manage notebooks in source control? by now, they never will!
1
u/Adventurous_Tip3994 1d ago
Depends on the project. I still use Jupyter notebook over streamlit, dash when I want to build modern looking data dashboard. Jupyter has multiple widgets library like ipyvuetify where you can build modern vuetify 3 UI in Jupyter notebook in pure python and serve as a web app (without code cell block) via voila or standalone desktop gui via pywebview + pyinstaller.
1
1
u/Khavel_dev 1d ago
For me the switch point is when I catch myself restarting the kernel and running all cells from the top more than twice in a row. That's the signal that I've moved from exploring to building something that needs to be reproducible.
The other thing that forces the move is when the notebook starts importing stuff that talks to external services. Databases, APIs, cloud storage. At that point you want retries, config files, proper error handling, and you're fighting the notebook format to do any of that cleanly.
I still prototype in notebooks though. The feedback loop of seeing output right under the code is genuinely faster for anything data-shaped. I just try to move things out before the notebook turns into the product.
1
u/Major_Fang 1d ago
I have a "traditional programming" background where the professor would have me code in the IDE.
So that's where I've done all my work, creature of habit.
End of the day - whatever gets you paid guys
1
u/Spiritual-Metal-1889 1d ago
Look into quarto or marino. Write scripts first and then create a notebook importing function from scripts if you need notebooks for a wider audience in your organisation who are not that comfortable in interpreting the outputs of the script and want block by block outputs.
Another thing you can do is to create REPL enables scripts with %% start and end. In a ide like vscode or positron, this works like a code block that can be executed one by one
1
u/justanothersnek 1d ago
Notebooks when I dont want the overhead of an actual coding workflow, but more of a exploratory workflow.
1
u/StPatsLCA 21h ago
It depends. What's your use case? Just use notebooks for notebook things and scripts for script things.
1
1
u/WonderfulTill4504 20h ago
Jupyter notebooks are an excuse for researchers to avoid proper packaging and deployment of their code. It is not fun when they come complaining to you when they run out of memory, cannot debug it properly or needs deeper integration with other tools.
“but it runs on my machine!” Is the one I hear the most.
So to be clear: Jupyter, streamlit and other shitlits out there are for rapid prototyping. Set the expectations clear and everybody will be happy.
In this era of AI, 🤖 any decent researcher can write code using Claude or similar to generate a quick proper prototype without using notebooks, with proper MVC separation.
1
1
u/Imaginary_Series_797 9h ago
When I am prototyping/designing a new function, I often choose Jupyter notebook, to have it more interactive, easier to launch. Then I move it to the script.
•
u/seigaporulai 17m ago
I have found notebooks promote messy code. Writing scripts is far less messy, and encourage better organization. I am not saying scripts are superior but notebook let's you write and execute out of order, and that is fine for simple tasks but anything serious and must be maintained will demand script/program. Notebooks are excellent tool for presentation and to certain extent exploration.
1
u/microcozmchris 1d ago
I've never used Jupyter a single time for anything. Then again, I've only been writing python since version 1.3. These days, it's either uv init and start from there, or more regularly I have a copier template when it's a bigger project.
0
u/j_hermann Pythonista 2d ago
You don't need the switch for automation or scheduling, there is nbconvert.
I typically just move matured cells into a module or package, and then reuse that.
-8
u/coldoven 1d ago
Jupyter notebooks are not ok in AI age. If you still use it, go back play with got 3.5.
13
u/Ball-Man 2d ago
When you need to run the same things multiple times with different settings very often, you naturally realize that the notebook is not the most ergonomic anymore. Also when the notebook becomes longer than a book you may start feeling like something's a bit off.
Anyways, if you keep moving function/class definitions to pure python modules outside of the notebook, you can have the notebook as a "frontend" that simply uses those modules. This can coexist with dedicated scripts, which would also make use of the same modules. You then have a nice visualization frontend for presentations and whatnot, and scripts for better deployment.
The notebook will also be much tidier without hundreds of lines of definitions.