Do you mind sending me an email and maybe we could chat?
Thank you for your amazing comments. I really like your idea about indicators and saving state. I'll give that a try! Yeah, Marcos López de Prado is actually where I read about the tick bars and sampling at higher rates. You need like 2 phd's to read his books though. haha. I am doing this based on X number of ticks and not even looking at volume. I was using tick count as an indicator in that you can really see patterns when activity picks up. This sync issue is really really interesting and I'll explore this.
"The thing you’re actually doing is predicting what other people are going to predict." I think I've actually seen this in the data. In that you can see this mini-cycles almost when you really zoom into a fast moving stock. I'll check out. Both your comments are amazing and it 100% shows you know what you're talking about.
This whole project started when I was looking for an investment adviser and didn't know anything. I was looking at all these funds that had like 4% return per year and then looked at stocks like Tesla that were up like 45% return and was like this doesn't make sense. When you dig in you see that these index funds are super super low risk. However, I like extremely high risk so I was willing to explore this a little. I don't really want to say you can get better returns, because you could end up losing all your money, extremely quickly betting on single stocks, it's basically a sure thing. So, that's why I turned to building a tool that basically bets for you. So, it's more about risk. If you put everything you had into TSLA you'd be up like 50% just in the past month but that's risky as hell. So, it's more of a risk question than a return question.
Hey, thanks. Yeah, I agree with you. That's an oversight on my end. I'll tell you here though.
I'm just using go routines and channels to talk between them and then a giant mutex for locking. That's basically it. So, as new data comes in, it builds aggregates (tick based candlesticks) as needed, this then triggers the the BUY logic loop on that new data, if something is detected, that triggers a IB API order. It is dead simple and nothing complex in here. I've had upwards of 100 positions being tracked at anyone time and seems to just work. So, I haven't messed around with complex async logic too much.
I'm actually just hard coding the parameters right into the BUY loop. This probably sounds crazy but for a small setup like this they don't change that much. So, I can run some trades, tweak things, restart, and then test some more. I imagine if you were doing that in an enterprise setting you've have some formal language and hot loading and stuff. But, for me hard coding seems to work well enough.
1. Do you have anything that limits the size of a single trade or position?
2. How do you measure and manage overall volatility/risk/VAR to your portfolio?
3. What kind of safeguards do you have to avoid catastrophic bugs? (For reference, see Knight capital and how a single bug brought down the entire company: https://www.henricodolfing.com/2019/06/project-failure-case-...) Given how fast your system trades, I imagine it must be difficult to visually spot these errors. (You did mention paper trading, but I wonder if you have anything else you want to mention)
Thanks so much for sharing your knowledge publicly. It's very much appreciated!
Here's some simple rules. I basically just read about these and then stole the ideas. I wish I had data to back this up and it seems to be working.
- No single bet can be more than 5% of all money. The small bet sizes are what really saves your bacon in that even if a few lose 10+% you're still fine overall.
- I'm also limiting the amount of shares I bet and try to keep it in the low hundreds so that I get really fast fills.
- I have something that stops everything if I lose more than 1k in a day.
I don't do anything to measure overall risk or volatility. Almost everything I'm in is highly volatile. I'm basically betting as things go up and then try to cash out. Sometimes you hit the top.
Yeah, I inspect all the trades at the end of the day, well and during the day, and try to feed anything new back into the system. This is 100% manual. But, like if a single trade loses $100 or something definitely I'm in there looking at what happened.
Around 45-50% win rate but I try and keep the runners going for as long as possible and exit out of the loser trades quickly (this basically just costs you the commission on losers but sometimes more). You can tell within a minute or two if what you expect to happen is going to happen. So, even with a 50/50 win/loss rate you can still make money. I make around 100-200 trades per day. Sell everything before market close.
Does it work synchronously or asynchronously? For example, if you integrate WebSockets then you could act as soon as you get new data. If it works synchronously then you regularly request data (say, every second) and then act.
If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?
Both? I'm getting a constant stream of data via the websocket. As data comes in it gets added to the large in-memory object, and then once I have X number of trades, I'll go and build the candlestick bar. As soon as that bar builds, it triggers the BUY loop to inspect it, and see if it matches what I'm looking for. If it does, it send a buy order out to the IB API. I'm not ranking anything. I'm just looking to see if that matches and then buy. So, I have something that tracks the total money I have, and if I have any free money (and we don't already own that stock), it makes a bet. That's the logic. I wish I had some ranking logic but that's what it's doing.
That logic is happening for 5500+ stocks all in real-time which is pretty insane. But, it works really really well. Go is amazing. At market open and close there is like 60k+ events per seconds across trades and quotes. So, that loop is processing like 60k events at times, and building each stock out, and then looking to see if we should buy/sell.
All my data is from polygon.io and nothing external. I have been exploring using lookup tables of pre-computed historical values to do things like anomaly detection (ie, is this normal activity for this stock). A good example would be BXRX today [1]. I have been looking at options trading activity too and trying to use that as a signal.
Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
Go is great for network heavy apps like algo trading!
I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes:
- Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage.
- Getting caught in the middle of a pump and dump event (also frequent)
- Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %.
Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
It was my (admittedly ignorant) impression that C++ was preferred over garbage-collected languages because you'd be more likely to avoid GC pauses. Yes, of course I know that Go's is super efficient, but if you're doing a million trades a day even a tiny percentage of slowdowns that would be otherwise preventable could be a career-limiting move. Am I just behind the times?
I think you're right! Go will never beat C++ in this aspect.
I used Go because that's what I knew and for the non-professional trading I was aiming, C++ wouldn't have made much difference. My bottlenecks were network (1s+ per trade roundtrip) and chaotic unreliable crypto markets.
Just note that Java is used in HTF, but it's a different beast than our average CRUD Java. For example this article states:
"Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage."
Yep, you can with GOGC=off but then I would have to learn about how to generate less garbage in Go for long running processes. Which is good knowledge even for non-trading applications. I just didn't have the energy and time to try it.
Maybe? Every language one would have to work at to make performant, and generally pre allocation is what often happens to do so. allocation is probably more expensive with a gc’d language like go than C++, but both would want to avoid it.
I read like 5 or so books about trading, mostly the classic ones, and tried to toy around with backtracking some self-made strategies using MetaTrader: https://www.metatrader4.com/en/trading-platform
First week was funny. I kept thinking I found goldmines with fine-tunned stochastic lagging indicators, only to realize that I overfitted and an algo that made me rich when tested against a certain period of time would make me go broke when applied to another period.
I tried applying some defense against local maximum, but to no avail.
Best I could do is tie after paying exchange fees. And that took me endless weekends toying around with code, reading books and articles.
My hunch is that algos that don't rely on external signals (i.e. news) need extreme technical edge like High Frequency Trading have with their ultra low latency and premium exchange data feeds.
And I wasn't ready to dive into news sentiment analysis and other external indicators.
Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
Yeah, I've been messing around with this for 4+ years so. Got to see COVID nose dive, meme stocks, massive build up, market taking a dump, bank collapses, and now a bull run again. I have no idea if it was always like this but it has been in insane.
Yeah, I ended up taking https://github.com/gofinance/ib and rewrote my own wrapper. This took a long time but has been stable since. I'm basically only doing buy lmt, sell lmt, cancel, and updates orders though. So, the logic is pretty simple. Catching all the return messages and structuring them correctly took tons of debugging, trail, and error. Basically, mapping the messages into the correct orders for state tracking.
Thank you for your amazing comments. I really like your idea about indicators and saving state. I'll give that a try! Yeah, Marcos López de Prado is actually where I read about the tick bars and sampling at higher rates. You need like 2 phd's to read his books though. haha. I am doing this based on X number of ticks and not even looking at volume. I was using tick count as an indicator in that you can really see patterns when activity picks up. This sync issue is really really interesting and I'll explore this.
"The thing you’re actually doing is predicting what other people are going to predict." I think I've actually seen this in the data. In that you can see this mini-cycles almost when you really zoom into a fast moving stock. I'll check out. Both your comments are amazing and it 100% shows you know what you're talking about.