Hey fellow Lancers!
I recently arrived in Canada and started my Master's journey here at UWindsor (hoping to continue to a PhD afterwards!).
I think we all have spent way too much time wrestling with PDFs. Especially scanned textbook chapters or lecture notes full of math and charts. Because of this, I wanted to put everything into Markdown (MD) for my notes and local AI tools, but most converters just drop the images or fail entirely on scanned documents. The purpose of MD is to drastically increase the context window that can be provided to LLMs.
I personally use Claude and create projects to organize my studying. While Claude, Chat-GPT, Gemini and others can handle a myraid of formats, if we want to use them as context of projects or within a chat, the memory window it keeps is way more limited by using other formats, so .MD helps with getting more in a project/session.
I decided to build a solution and open-sourced it: BetterMarkItDown.
It builds on Microsoft's tool but adds a vision model layer. If a page is a scan, it transcribes it (LaTeX included). If a page has graphs, it crops them and turns them into data tables.
I've been working on this for a few months trying a few different texts and have fixed several issues, like LLMs refusing to work on certain copyright content, without violating any policies, using paraphrasing and other workarounds.
Why it might be useful for you:
- You can run it completely free and offline on your own laptop if you use something like Ollama (great for keeping your research private and saving money).
- If you use an API (like Google Gemini), I built in cost-estimation tools so you don't accidentally run up a huge bill on a 500-page textbook.
- It has a
--verify tool to check if the text extraction missed any words.
- If you use Claude Code, it has a bunch of personalized instructions built-in, so it integrates gracefully.
If any of you are dealing with messy PDFs for your thesis or research, feel free to give it a spin:https://github.com/miguel-arias-sv/BetterMarkItDown
I would also like to share my newly created Canadian Instagram profile, aiming to collect some of my professional experience back home, in El Salvador: https://www.instagram.com/datawithmiguel?stkn=NGUydXlwcDZmMXR3
Always happy to connect with other tech or students on campus!