¶1Hello. Hope you are doing well. So today we got a pretty exciting video I wanted to make for a while, but now is just the perfect time because we just got Opus 5. I have access and I think this is the perfect time to kind of launch this series. I kind of want to do in the uh upcoming weeks, months.
¶2We'll see. So this is uh for today it's going to be Poly Market, could be Cali, could be other stuff, but it's going to be like an AI predictions battle, right? So basically what we want to do is we want to take in this case two events and we're going to put GPT 5.6 and Opus up against each other to see who can actually predict this event. So the events I picked for today is 3 days out or two days out. It's the lowest temperature in SE on July 29th.
¶3and how many descent at the Fed uh July Fed meeting because this is quite even right and you can look at this there are some breakouts here but there are some options to go either way and I think this is just a super cool benchmark and it's going to be really interesting to follow the results we might do some stake on them like maybe like 20 bucks on each just to see and maybe we can keep track of it. So uh I put up some restrictions here. This is very important because we don't want them to just go into poly market check the prices and take the one that is the favorite. So uh we have used uh today's sponsor SER API to give the AI agents access to actually run some research uh on their own kind of without using codeex or cloud code. So SER API is a really good platform if you want to do search in like a really structured output way.
¶4Let's say on Google uh in this case we're going to use the agents for that. So here I can just put in uh an API request and this is perfect for AI agents and they can get this in a really good structure like here we have JSON format right. So these work very well when we want to do some AITIC research because we get all responses in this very good format here and you can see we get it in JSON. We can kind of adjust and do whatever we want and they have a lot of different options here. Let's just take a look at a few other options we can use SER API for if you want to expand our research.
¶5Something I've been looking into is and that is kind of YouTube search engine results API. This could be also something we want to include. Maybe we want our models here to kind of use the YouTube search API from SER API and actually get the transcriptions from videos include this work this into the research around the prediction they're going to make. So this is something you definitely do to check out if you want to be a agent. You have a bunch of other options too here if you kind of want to yeah start using uh extended third party research.
¶6I think this works a bit better than actually using the web search they have in kind of codeex and cloud code just because of the format here is so flexible and you can do a lot of stuff with the formatting here on surf API the JSON. I really like that. So get started with SER API using 250 create credits. You can get that by clicking the link in the description or just scan the QR code that you see on the screen here. So let's just go back here now and you will see throughout the video how we actually use SER API in this setup we have built here.
¶7So one thing that is really important here is that uh you can see I wrote down here obuscate uh share price obuscated share price. So this was something I was looking into to do because I don't want the models to be kind of biased or kind of inflected or like uh contaminated by checking out the latest movements in directions in price here because they will really get like bias towards these results. Uh same here these are quite even now so it doesn't really matter too much but you kind of get the point. So uh I found a way I think I found a way to do this but this might be improved in the future. We will see.
¶8And of course, the output is just going to be a prediction for both events. And we're probably going to put some stake on it. Maybe do like $10 on each or something like that just for fun, just to see how it ends. And I'm going to make a video at the at the end date for this. I probably going to be at least this week and check how we well.
¶9And in the future, we can probably do more predictions. we can test other models and just work a bit on the scaffolding to see if we can improve it over time. So now I think I just want to head over to the scaffolding, show you a bit more that uh before we run the events and see what kind of predictions GPT 5.6 and Opus 5 comes out with and place our bets and stuff like that. Okay, so let's just start by going through the scaffolding because this is very important and it needs to be like repeatable. So uh the first thing the scaffolding does is depicts the event.
¶10So I kind of set this up to scan for uh events that resolve in kind of the next three days. This could be changed, right? And drop anything that has already been decided because we want like uh an even uh kind of market. So it doesn't really like it's 99% sure that this is going to happen. That is not very interesting.
¶11And we're going to keep one question per event. So 10 legs on the same ladder doesn't count as 10 data points. And we want to strip the market out of the question. So we kind of rewrite each question. We try to not reveal that this is like a prediction on poly market because that is kind of like a hint where it should go look for it.
¶12And we give the models the research tools but police them with a wire. So here we kind of have SER API like we talked about. We're going to check that out. and polyarket and kalshi.com are stripped out of the results. So we can't really open up URLs that has those uh direct URLs in the or we won't get those results right.
¶13And next you can kind of see we want to make it fair. So we kind of don't really we just run them one by one and we clean out the cache and the uh the results after each so we don't get like interference and then we're just going to score it. So we kind of make the model submit a probability for every possible outcome. Not that we said yes or no. So let's say in this case here the model has to submit a probability score for each outcome not just the one it thinks it's going to land on.
¶14That makes it a bit more interesting and we can kind of compare it. So I mentioned that how the price is obuscated. I guess I can kind of mention this here. So instead of asking will it be 26° in se at blah blah blah uh you can see what we do is the model shows the entire range of possible outcomes with no prices attached and this uh the must spread is the probability across all of them. The range itself is sex by a fixed rule written before looking at the market.
¶15So nothing about the list uh of options reveal which uh one of the market favors. So we just give them all the options. So it doesn't really attach a price to the options it's going to try to pick from. So we always try to hide that this is like a prediction on like a prediction market with share prices. We just want them to give the highest confidence score in what's going to happen in this event.
¶16Right? Uh I also said why is uh SER API so good in this scaffolding? It's a single choke point. uh we control uh it returns structured data not a rendered page. This is very important.
¶17You can see results arrive as clean JSON. We have a title URL snippet. So filtering this is very easy if we kind of want to set up some guards around what the model returns. We can kind of block that if we want to. It's neutral and identical for everyone.
¶18This is another thing that's very important. So we want to use SER API here because that means that claude code and codecs and the models have the same way of searching the web. Let's say claude has a better way to search the web. This also happens when we kind of want to use local models then everything has the same uh structured for actually doing research. So that is three points why I think SER API is so good at this scaffolding.
¶19So now that we kind of know a bit more on how we're going to run this uh prediction battle, I think we're just going to do it and see what comes out and what models pick what prediction. So I'm just going to do it one by one. Now we could in theory do it in parallel, but that could do some contamination. So first I'm just going to launch Claude. Uh I think I set up a very simple prompt for this.
¶20So I'm just going to do read instructions. So we created like a instructions file for the model to follow so you know exactly what to do and execute your best predictions. So this is the only prompt I'm going to give this to Opus 5 and on GPT 5.6. So I set the model to high. Uh I'm going to do the same on GPA 5.6.
¶21So I'm just going to let this run now. It's going to kind of work through. You can see here it read both questions. We can take a look at those questions. So these are Q1 and Q2 domarkdown.
¶22So here you can kind of see here is the question what it's supposed to do quality to for quantity to forecast we give them the full list of the possible options and what to submit write a submission in the format given in the template and here we going to yeah and it's basically the same on Q2 we have a list of predictions it's going to make six or more from zero to six or more and it's just going to submit that. So this is the Fed meeting thing. So it started by reading those two questions and now we just get into research and here you can see it's using SER API to do some Google search. You can see it's find acue weather that's a really good resource for this looking at the predictions on acue weather. It's just going around ear.
¶23That is like a Norwegian site that is also really good for forecast. So basically it goes out to the web now using SER API to gather all the research data. It's probably going to run some calculations and it's going to come back with a prediction for uh the weather and of course it's going to move on to do research on the on the Fed. And what's interesting you can also see it do some searches in Korean here. That's pretty cool.
¶24I wouldn't be able to do that. So let's just wait for it. So you can see it's got a a lot of different sources to actually look at the weather forecast and then we're just going to wait for the Fed and let's see what uh Opus 5 here predicts in this um this first run here. After about 10 minutes, you can see Opus now are ready to write its submissions and it kind of goes through every single question here and again we have this nice JSON format for our output. So, let's just see what it landed on here.
¶25Looks like it was 2 at 28 and 26 at 28. So, yeah, that was basically in line. So, I'm just going to go to our kind of master here. I'm going to say Opus has submitted it prediction. Read it and print it here because we want to clear out the answers before we run it on GP 5.6.
¶26And here we can kind of see we have both predictions here. So it's going to be peak 26 and the was it two at 28. So basically it's going to go 26. That is the favorite two. Uh still we're probably going to follow that.
¶27And this is two. Okay. Okay, so that was a bit both fell in line with the favorites. But let's see if uh something else happens on GPT 5.6 if we get any other results than just following the favorites here. Uh I had a look and I couldn't see anything that kind of indicated that it went to poly market or anything but but that's fine.
¶28Just going to see clear out any traces of open submission and work. Make the repo ready for our next model. And then we can just launch codeex. Uh we're going to do the same there. Pretty much the same thing.
¶29Uh let me just see model. Uh we should have soul uh high. Yeah, that's good. And we're just going to do the same uh in um same uh prompt here. Let's just wait for it to see if it cleaned everything out.
¶30Okay, so we kind of cleaned it up. That's good. And then we're just going to run it here on 5.6. And let me take you back when we have the predictions here from 5.6. Okay.
¶31So, you can see we have the results in from GPT 5.6. And this took another line. I'm really happy to see that. You can see this p peak 25 at 52 actually went for zero dissidence. That's going to be pretty interesting.
¶32So, you can see we have uh both the answers now. So, I guess I can just do something like this. Uh, here are um So, I guess I can just paste it in here. And here are the results from GPT 5.6. So, we're just going to do Opus versus GPT table.
¶33Why don't we just put $10 on each prediction? That's going to be pretty interesting, right? So, if we head over to the predictions market and here we can see they have the round one. Okay. So, you can see for Opus, for the sold daily, 29th July, we're going to pick the 26 bin.
¶34Okay. Let's put 10 on that. So, that's going to be the 26. Uh, okay. Uh, let's put a 10 on that.
¶35Okay, that was done. And what was the other one? That was uh GPT 5.6 is going to do the 25 bucket. So, let's do a tener on that. Yeah, that was done.
¶36And the other one, let's scroll down here for Opus 5. It was the second bin here. Let's do a tener there. Okay, that was done. And the final one was uh zero for GPA 5.6.
¶37And that's it. Let's check our positions now. And you can see we have those here. Perfect. All right.
¶38So that is our first battle uh on the predictions market and we're going to come back to this. So this was a really interesting video and if people like this I'm going to do a lot of these predictions going forward then we can kind of over a while and we can include other models too see who uh or who what type of model actually make the best predictions. But this was kind of just like a like a an early uh version of the scaffolding. We might actually make this a bit better. It was a bit worrisome that kind of opus kind of picked exactly the top leading indicator.
¶39So maybe the obuscated share price wasn't 100%. But I'm going to work a bit more on kind of the scaffolding and hopefully we can improve it in uh upcoming uh upcoming years side. So this was really made possible because we can use uh research tools like SER API because it gets very neutral. Now both models have the exactly the same uh research tools and that is very nice. So definitely go check out the SER API.
¶40You can find a link in the description and you can sign up and start using this too for your AI agents or other stuff if you have that and you get this nice JSON output that the models really appreciate. So, thank you for tuning in today. I'm going to come back with a video with the results. And of course, you can go check out the results yourself if you want to. Have a nice day and I'll speak to you again very