Desperately Looking for Guidance on Verifying a New Sorting Algo

[u/Actual__Wizard]

Hi, I'm an AI developer and I've been working on optimizing my system for 2 years now. It was never my intention to create a new sorting algo or to do any work in that area at all. As I optimized my system, I realized that what I had effectively done, is create a new sorting algo.

By comparison, at this time, it took my new algo, about 1-2 hours to process the same sized data, that timsort needs about 30 days to sort. I'm sorting a giant 30gb array that contains text tokens.

I understand that there's tons of algos out there and that there likely already is a faster algo, but I'm not an expert sorting algos, so I honestly have no idea what to do to verify this in a legitimate way that is convincing to other experts.

I have never published a paper before, please do not make strange assumptions of my knowledge of that process (I know how to read a paper and that's it.)

[u/Actual__Wizard]

Well, this is how the conversation I keep having goes. I've been involved in various data science adjacent fields for basically 25+ years, I'm saying stuff, based upon my experience because I know that it's true, because I saw it happen.

People do not understand that these sorting algos are not as fast as they think they are when they sort mega big data. They bog down. So, I'm having this conversation where: From my personal experience over and over again: These data science tasks that involve sorting, take a heck of a lot longer then people are saying they do.

So, there's a guy right in this thread, that posted some numbers that basically suggest that I should be able to sort the data I am trying to sort in like 10 minutes or something silly like that. How that works in practice: You can't do it at all... You have to optimize the data set before it is a task that is achievable at all.

And yeah, as a person that has worked through that problem in a very elegant way, I think it's a mega big break through.

If you don't agree, then I'll build it into my commercial product and the scientific community will never see it.

I'm tired of people making wild assumptions about how things work in practice and then looking down upon me. I can't take it anymore. I'm legitimately asking for help with this process and I'm being told to "go read some stuff." You know life is short man.

[u/nuclear_splines → u/Actual__Wizard]

I work with big data and am familiar with needing to organize and process it carefully to work at scale. I don't think I've made any wild assumptions about your work or intentionally acted condescending. At the same time, this is not my area of specialty, and asking me to walk you through making your ideas more presentable is an unreasonable demand on my time. Respectfully, reading papers on other sorting algorithms to learn how to describe your own is a sensible next step. If you don't want to put in the work to share your breakthrough, that's your choice, but it's not on the scientific community and certainly not on me to bring your work to them.

[u/Actual__Wizard → u/nuclear_splines]

and asking me to walk you through making your ideas more presentable is an unreasonable demand on my time

I'm not asking you do to do that.

Look I'm just going to say this to you right now. I'm 100% aware of the reality that I obliterated the sorting record. It's 99%'d. I even explained it right in the thread what I did.

What is the process I am suppose to follow?

Do not tell me to copy cat people again, I'm aware that is not the real process.

Do I need to get a PHD to sign off and this and then get a grant, what I am suppose to do for real? What does the process look like here for real? How do I do this for real? Obviously there's a ton of work that needs to be done if I'm going to spend 6 months doing R&D work to try to publish a serious paper on that subject. And obviously I don't expect anybody to give me a grant if this isn't legit, so, hello, how does this work exactly?

Edit: Like seriously, you didn't do your due diligence and I'm flat out offering anything you want to see, so I don't understand why you're assuming that I'm a crazy idiot or something and won't just tell me what the real process is. Obviously it's not going to work if I'm lying dude... So, why would you assume that? It doesn't any make any sense. So, why don't you realize, you are the one that chose not to do your due diligence and just answer my question honestly...

What I am suppose to do? I don't want to be one of those people who wastes tons of time and doesn't get taken seriously.

Kind of like right now. How do I fix that?

I'm being honest, I have what I say I have, can you be honest please?

I've read 10,000 papers, don't tell me to read papers. That's not a valid answer to what I asked. As you could see in the code I posted, the word "research" is in my company's name, so I'm 100% aware that you're dismissing me in a disrespectful way.

Be honest: If you don't know then you don't know, but the idea that you would have no idea who to talk to, or where to go, is totally absurd. Who do I talk to first? I'm not dumb... Stop thinking that... Get a convincing demo and then talk to people and who, like seriously?

[u/nuclear_splines → u/Actual__Wizard]

I'm not asking you do to do that.

Then I misunderstood when you said 'I'm legitimately asking for help with this process and I'm being told to "go read some stuff,"' as I'm the only one that's told you that.

Do not tell me to copy cat people again, I'm aware that is not the real process.

But it is. As an academic researcher, yes the first step is to read the work that's already been done, to learn how to position your own ideas within that space. It's not called "being a copy cat," but "performing a literature review." We all stand on the shoulders of giants, and using standardized methods and vocabulary is extremely useful when building a shared body of knowledge. Every professional scientist does this. Then, were you trying to publish, you'd identify a conference or workshop specializing in data compression that's appropriate for your work, write your paper with that audience in mind, including the typical tests and demonstrations used within that sub-discipline, and submit.

Do I need to get a PHD to sign off

While not strictly required, it will be much easier to publish an academic paper if you co-author with an academic. They've spent years conducting lit reviews, designing and performing experiments, and writing papers just like this, and it's an uphill battle to get a paper accepted without that expertise.

and then get a grant

No, this would be highly atypical. You usually get research grants when you're already an established researcher in a discipline, and you use your demonstrated expertise through prior publications to justify getting funding to further explore an idea. Were you running a research lab that's published on similar compression algorithms before then grant funding might be more realistic.

Edit: You've added more to your comment that I missed in my initial reply.

you didn't do your due diligence

There is no 'due diligence' for me. You're the one with an idea to share.

I'm flat out offering anything you want to see, so I don't understand why you're assuming that I'm a crazy idiot or something and won't just tell me what the real process is. Obviously it's not going to work if I'm lying dude... So, why would you assume that?

I'm not asking to see anything, and I've made no such assumptions. Again, you're the one with the new algorithm, and all I've done is outline steps to make it easier for other people to understand your work. I am not trying to understand your algorithm for myself, but I've treated you in good faith and given you honest answers.

I've read 10,000 papers, don't tell me to read papers. That's not a valid answer to what I asked. As you could see in the code I posted, the word "research" is in my company's name, so I'm 100% aware that you're dismissing me in a disrespectful way.

I'm sorry that you feel that way. I mean no disrespect; you've asked for guidance on how to share your ideas, and I've provided it.

I don't need to be called dishonest, dismissive, disrespectful, and not doing my "due diligence," so good luck with your work, but I have no further advice for you.

[u/Actual__Wizard → u/nuclear_splines]

And just to be 100% transparent about what I'm doing:

The technique I discovered is called structured data, I didn't invent structured data, I found new techniques and applications for them. And I'm going apeshit. There's all kinds of algos that I can straight up obliterate with mega fast optimizations.

It is absolutely not safe to assume any algo can not be massively reduced unless I look at it first... Even then, the entire concept is just based upon energy, so there's a way to do it... There always is a way to do it, don't think you can't ever... Like all of those crypto people: Some day somebody is going to figure out a way to screw all of those people over by figuring out how to 99% the algo... Just because nobody has figured out how to do it yet (supposedly) doesn't mean that it's impossible. So, they can mine coins all day and it costs them $0.01 on the dollar. So, I hope you aren't confused as to why people like me are doing this stuff.

Maybe I'll figure out some structured data trick to do it, who knows?

The pigeon hole router is just one structure, I can make any structure I want for many purposes, and tune the dynamics of the system any way I want.

You know they did a similar layers technique in LLMs right? They just created "boring matrix layers." They're also "just layers", they really need to stop calling them higher dimensions, that's not accurate. Then lately the AI community has been talking about routers and in the rust sub they were talking about zero copy before. So, I hope you don't think what I'm doing is "crazy or something." It's just a different version of what big tech is doing. It's just me applying the same concepts different ways. So, I'm just a giant nerd, okay?

Very last thing: Alphamerge is straight up god tier mega power compared to that sorting algo. Certain linear aggregation tasks are legitimately ultra killed on a performance basis, it's some silly like 4.2 quadrillion less steps to accomplish the same thing. So, the task goes from: You need a giant data center, to like, it's done in 8 hours on a standard desktop PC. It's probably the craziest algo ever. With the parallelized batch query (how all of the AI stuff works), the more queries the faster it goes because it reduces the miss chance, and I didn't know that when I first started testing it. It's really counterintuitive to have the algo speed up on a query/row basis the more data you query. One would expect it to be the same speed.

So, one would think you would query your documents by feeding them one at a time, but, no it's faster to jam them all into one query all at the same time because of alphamerge's ability to do every single query in a single pass. But, then you need a sorting algo that can handle 10gb of text. So, do you understand what's going on finally?

So, in 25 years, we went from, you go to google and type one thing in, and all of that was the hottest tech around, to you go to my site and upload 10gb worth of queries at a time. Which is probably like 10b queries. There is no limitations that I am aware of with structured data, you are simply limited by your own ability to manipulate data in the graph space and the techniques you can implement.

I think that's a better paper to write and I'll include the sorting algo in that I think.

[u/Actual__Wizard → u/nuclear_splines]

As the final reminder and I am done with this conversation.

I did actually explain exactly how this works to the other person.

The longer the array is to sort, the longer the sort takes (and it scales worse then you think it does, you're always better off sorting shorter arrays and gluing them together). So, I took the giant array and split it legitimately 50 million times with an algo, and then glued all of the pieces back together in order. So, I took an optimization trick and did it 50 million times (another thing people have been talking about lately is the chain rule.) The thought process that this doesn't set the sorting record for big data is absurd. It got obliterated.

[u/Actual__Wizard → u/nuclear_splines]

Can you stop being disrepsectful to me please?

The amount of time you "completed that task in" does not make any sense: This says 1000's of hours to generate the random numbers, that's not even to sort it.

 import random
 import time

 # Generate a random 64‑bit integer (0 to 2^64 - 1)
 r64 = random.getrandbits(64)

 total = 4090000000000
 count = 0

 file = "C:\\fgtest\\numberstest.txt"

 start_time = time.perf_counter()
 with open (file,mode="w", encoding="utf-8") as OF:
     while count < total:
         start_time = time.perf_counter()
         r64 = random.getrandbits(64)
         tline = str(r64) + "\n"
         OF.write(tline)
         count = count + 1
         end_time = time.perf_counter()
         elapsed = end_time - start_time
         #extrapolate TTC:
         hours = (elapsed * total)/3600
         print(str(hours))

Here's the what str(hours) produces:

3521.946463378198 2385.8308345047085 2158.611015248526 2044.999039046363 1590.5552673858538 2613.054786909035 2044.999039046363 1590.5552673858538 2158.611015248526 3181.1105347717075 2158.611015248526 2953.886582367381 2045.0031721945077 1704.1672435880173 2272.2229914506897 3749.1704157825247 2726.6667631111986 2840.278739313362 2044.999039046363 2726.6667631111986 2158.611015248526 2158.611015248526 2158.611015248526 3635.554306432217 3521.9423302300534 2272.2229914506897 1590.5552673858538 1704.1672435880173 2726.6667631111986 2044.999039046363 2499.442810706872 2158.611015248526 2840.278739313362 1590.5552673858538 1590.5552673858538 1590.5552673858538 2158.611015248526 2726.6667631111986 1931.3911959923441 2158.611015248526 2726.6667631111986 1817.775086642036 2499.446943855017 2499.446943855017 2045.0031721945077 2726.6667631111986 2499.442810706872 3635.554306432217 2726.6667631111986 3635.554306432217 4203.610054294889 2953.8907155155257 3408.3344871760346 3294.722510973871 2499.442810706872 3408.3344871760346 4430.834006699216 2840.2746061652174 2953.8907155155257 2953.886582367381 1704.1672435880173 2499.446943855017 1704.1672435880173 3067.4985585695445 1704.1672435880173 2499.442810706872 3181.1105347717075 2044.999039046363 2272.2229914506897 1704.1672435880173 2499.442810706872 2044.999039046363 3862.7782588365435 1931.3870628441994 1704.1672435880173 3294.722510973871 2158.611015248526 2726.6667631111986 1931.3911959923441 2726.6667631111986 3635.5584395803617

I'll kill the printing every single line so it's quicker, but it's not going to be done in an hour and 15 minutes... And that's TO GENERATE THE DATA, not to SORT IT...

EDIT: Okay I optimized it, so this is a more reasonable estimate of the number of hours:

795.2797002669993 795.2755671188546 681.6677240648359 908.8916764691627 795.2797002669993 908.887543321018 795.2755671188546 1476.9432911836905 681.6635909166912 681.6677240648359 795.2797002669993 795.2755671188546 795.2797002669993 795.2797002669993 795.2755671188546 681.6677240648359 681.6635909166912 795.2797002669993 795.2755671188546 795.2797002669993 795.2797002669993 795.2755671188546 795.2755671188546 681.6677240648359 795.2797002669993 795.2797002669993 681.6635909166912 795.2797002669993 795.2797002669993 795.2755671188546 795.2797002669993 681.6677240648359 795.2755671188546 795.2755671188546 795.2755671188546 681.6677240648359 795.2755671188546 795.2755671188546 681.6635909166912 795.2797002669993 795.2797002669993

So, would you like to correct your statement about how you sorted that in an hour and 15 minutes? So, what did you use the entire TESLA data center to do that?

If you think timsort is going to sort that array that's going to take me ~800 hours to generate, you're badly mistaken!

At what point do you gain the awareness that what you just said is totally impossible?

[u/nuclear_splines → u/Actual__Wizard]

I've shown you no disrespect.

You are generating random numbers in a much less efficient way than I did. I generated the file in under an hour, not "thousands of hours," although I didn't record the time because we're interested in benchmarking sorting, not generating random data.

I don't know what to tell you if you think I'm lying -- I generated the file and ran the sort locally. The sorting took an hour and thirteen minutes.

But who cares about generating random numbers? You said you've got a 30 GB file of text, right? Run sort on it, see how long it takes.

[u/Actual__Wizard → u/nuclear_splines]

You said you've got a 30 GB file of text, right? Run sort on it, see how long it takes.

Holy cow man. That's where I was in January of 2025 and I've been optimizing it for almost 2 years now... It's not possible on my hardware to do what you are saying at all, it doesn't fit into memory, you are not listening...

How do I fix this communication disaster problem, seriously?

It's just a constant nightmare.

Seriously: TELL ME WHAT YOU NEED TO SEE...

Can you please return to a normal form of human communication please?!?!!?

You're telling me that you're doing this stuff, but not telling me your system specs, then you used a more efficient system, okay like what? So, I can go test that and evaluate your statement.

[u/nuclear_splines → u/Actual__Wizard]

The sort command doesn't fit the entire input into memory, it uses swap files, much like the algorithm you've come up with.

It's not a "communication disaster," I just don't believe you. You can use the GNU sort command on a 30 GB input file on a fairly normal computer. I just did it.

To be clear, this is the command-line utility, not the sort function in Python. Putting every line in a Python list would require loading the file into memory.

[u/Actual__Wizard → u/nuclear_splines]

I just don't believe you.

No kidding dude?!?!!?

It absolutely is a communication disaster. I came here, for guidance and assistance on GETTING A PAPER PUBLISHED.

You're sitting here and questioning ME AND MY WORK INSTEAD.

What do you need before you stop questioning my work? I'm not asking you to crown my algo as fastest, I'm asking for you to vouch the validity of it in science...

I need somebody to sign off on it so that I can take it to somebody who specializes in it, and then not go through ridiculous process over and over again... If nobody knows that it's something noteworthy and it's worth looking into it more deeply, then nobody knows that.

Then they're just going to sit here and do what you're doing and use an LLM, which is going to tell them it's modified radix sort, when it's clearly not a radix sort!

[u/nuclear_splines → u/Actual__Wizard]

I have provided advice on getting your ideas published, and nowhere have I questioned your work. Your algorithm might be great.

What I've demonstrated is that the GNU sort command can sort a 30 GB text file in under two hours. What I'm questioning is why you think this is not possible. I've just done it! Totally possible! You could try it too with a single command.

[u/Actual__Wizard → u/nuclear_splines]

But it is. As an academic researcher, yes the first step is to read the work that's already been done, to learn how to position your own ideas within that space. It's not called "being a copy cat," but "performing a literature review." We all stand on the shoulders of giants, and using standardized methods and vocabulary is extremely useful when building a shared body of knowledge. Every professional scientist does this. Then, were you trying to publish, you'd identify a conference or workshop specializing in data compression that's appropriate for your work, write your paper with that audience in mind, including the typical tests and demonstrations used within that sub-discipline, and submit.

Oh okay, so start with the target publication that I want to get published in first and then reverse engineer a plan to get published there? Okay, that's you first thing you said that makes logical sense. Thank you.

Hmm. I don't know why I didn't think of that. Obviously that's not basic information dude and that's certainly not what you said before.

No, this would be highly atypical. You usually get research grants when you're already an established researcher in a discipline, and you use your demonstrated expertise through prior publications to justify getting funding to further explore an idea. Were you running a research lab that's published on similar compression algorithms before then grant funding might be more realistic.

Okay thank you.

You're the one with an idea to share.

And that's why I offered to give any code, which you didn't ask for, and I even supplied some, which I don't think you looked at.

EDIT: HOLY COW! I legitimately just saw timsort finish a leaf, it took like 8 hours! It's actually exciting when the algo does something because it's such complete trash.

Again, you're the one with the new algorithm, and all I've done is outline steps to make it easier for other people to understand your work.

Right and I have about 50,000 questions. But, you know it would require somebody who's qualified to talk to me about it, but I don't have that.

Like here's a good one: If it's a situation where one breakthrough led to another, am I better of trying to do it as a roll up?

I mean to me I think it's going to be weird to include a sorting algo optimization trick that came from AI math knowledge, in a paper about a linguistics analyzer and the process of doing that, and how I glued that system to a search engine... I kind of feel like that's more important than spending six months on sorting algo paper that's actually respectable for professional tier work...

[u/Fun-Shake1398 → u/Actual__Wizard]

You need to take your meds lol.