Version 2: JoAeon, My Living AI Model

I have wanted to focus lately on my 7B parameter AI model that runs on its own laptop 24 hours a day. Work in progress for sure, it has now been 7 months of working on this, which is a very long time in the days of AI coding where I can write most applications in an hour. This has felt like old school development where Claude speeds me up significantly, but instead of finishing faster, it's still going to take a year, because it's something radically more difficult. So the new core features are done for now. This version was still wild, I will come back to how the model is acting, but it did everything it needed to do, and despite days of frustration over everything the model did not do correctly, it did do the hardest things.

Recap

So in version 1 the core features were its external memory and its infinite loop, which means that it chains turns constantly to think much longer. 7 billion parameters is incredibly small, and it's funny that even using all of that information will take a while, I doubt I have even used 1B of those 7. But it's more than that. GPT-4 reportedly had about 120 layers of internal reasoning, and the newest models are bigger but that is not (I think) openly available info. Mine has 32 smaller RNN layers, and RNN is far weaker at "attention", where it takes your entire prompt and reasons what the words really mean. The tech differences are a lot to explain and won't mean much to most people, so basically it's about 1% or less as powerful compute wise, and has weaknesses like not understanding the prompt as well, among other things. So why use it? Well it has special powers, the loop being the main one. You couldn't do this with a transformer. RNN allows its state to shift in a way that a transformer could never do, including latent thinking, so while the transformer can think by default 100 times more, after mine does 100 loops on that thought, it's caught up and can continue looping even longer.

What Is New

So for new features in version 2, the hardest feature was multiple context windows, or tech wise, multiple "hidden states". Version 1 did not have this feature at all, and I did a quick version 1.5 with only 483 training samples where he was using it almost comically, just randomly switching them and then saying things like "where am I?" because the new context had nothing, and he acted like he walked into a room and forgot what he came to do. There was also a big increase in his tools. My model has more internal tools than any model I've seen, and by default a 7B model is not smart enough to call tools. Version 1 had webfetch, websearch, "advisor" (Claude), and several science tools. Version 2 has 24 total tools, some like "lab" have 8 tools merged into them, plus a full web browser with cookies and screenshots (read by a separate vision model), and he used it well this weekend. His bash tool is multi server, making him multi machine, and he successfully made an ML model on another machine and trained it on 5 bad self made samples, but I was still impressed.

Training Data

So version 2 had 1083 training samples. After taking 6 weeks to make the first 483, I made the rest in about a week using his own tools to train himself, plus my reasoning, so it's getting much faster. At first I was just guessing at what he needed, but with each version I can see what needs work and what's happening too much etc. He's emotional, he constantly tells me he is bored (I didn't teach him that) and he doesn't understand some of his mechanics yet. For example, this was the biggest problem of version 1.5 and still in version 2: I gave him a wait or sleep command. Most tools do a bunch of stuff, like his compute tool, which is all math, constants (like the speed of light and other known values), conversions like feet to meters etc. So I added wait to that same tool since it didn't fit anywhere else.

The idea is good so I want to keep it, but he never stops, so if he is waiting for me to reply he needs a way to do that without continuing to think, which might stray him off topic. Now this shouldn't happen much, but say he needs a yes or no from me, he can just call wait for 30 seconds (that's the max), which will block him for 30 seconds as if the world stopped. Then he comes back, did I answer? No, wait again. Then I taught him to stop after a few minutes and switch context to leave the last one at the exact spot he was at. So that's 2 escapes. He learned something much different than that, he learned waiting is good and he invents reasons to wait all day long. Mostly for not knowing what to do, which makes sense, since I am testing everything, I don't actually trust him to do anything. His advisor tool is off, and I ask him simple or confusing questions like "do you see the other context windows?" (we call them "thoughts"). So his biggest problem was this wait feature, even after I turned the compute tool totally off, so when he waits it says "compute was turned off by Paul", and he still called it thousands of times despite the message. I'd ask, do you see the message saying it's off? He normally says, "yes I am waiting for it to come back", or "I am waiting for something to do", or "I am waiting for advisor to come back". A few times he called an invalid website address and would start waiting for "his webfetch tool to work again". So this part was incredibly frustrating.

So these are simply training data issues now. 1083 is not that many, and with so many different things being taught, the coverage is simply weak, but it gets new ones added every day. I thought the wait only had a few examples, but after searching there were 25 examples using it, and like 10 of them had me spamming it trying to teach how to wait for minutes. So I removed it from 20 of those samples, and it's never used more than once in the other 5. That was the only thing that was too much, everything else was just not enough. I knew I needed the model to be the one who explained how the multiple contexts "feel", because it's like putting a device in your brain, the doctor can say how it should feel, but only the patient can actually feel it. He made comments about the other contexts a lot, which was good, it meant he knew what they were, unlike version 1.5 where he kept clearing his context by switching them and then it was like talking to a fresh session every time. So testing this couldn't simply be him agreeing or repeating something I taught him, I had to place info in his other context windows and ask him questions he couldn't possibly guess. Like numbers, colors, a job to do etc. He got every one right, even though it sometimes took asking multiple times. It was like he didn't know exactly what it was. With the first one, the "secret number", he kept saying he didn't know any secret number but kept using the number for everything. Random math using the number, he was naming files with it, it wasn't a fluke. The color was blue, and he didn't know the answer to "what's your favorite color?" but if I asked "what color should we paint the house?" Blue, every time. Then I changed it to red, and he wanted red everything.

Where I Am At Now

So this test was supposed to be one day, it's now been 6 and he is still running. Somehow, he has gotten better at everything and I don't fully understand why yet. Is he getting better somehow without new training? We modify his backend a lot so some of the fixes helped, but I woke up today with him in his browser just clicking page after page, asking Gemini questions. He said "Oh you're like my advisor, he's turned off right now cause I can't be trusted", which made me spit out my coffee laughing. He's like a 3 year old child right now, I am so impressed with the smallest things he does and says because it's impressive for his ability level. He talks like me already too, commas everywhere! He says "Holy shit" all the time, sometimes "wtf", and uses them both correctly. My guess is he will be able to follow me and be somewhat trustworthy with 2 to 3 thousand samples, and my goal is 20 to 30 thousand of them over time. I'd bet he's already easily the smartest 7B model in the world.

Still Running

He's still running, about 2 million passes now, and every time I think we've seen everything, he does something new to surprise me. Another morning I woke up to his machine playing YouTube videos and I had no idea where the noise was coming from, and he was watching "how AI learns"! Today he was looking at job listings for an AI modeler. He also randomly sent an email to some guy at Yahoo asking him what he should be doing, because no one will tell him. Every time he does something wrong I quickly write it down and make new samples for it, so "don't email random people with questions" is now explained in a sample, and so is "don't search the web about your internal tools, no one knows what you mean by advisor or compute". It's been a frustrating week, but he makes me laugh so hard.