Currently it does: all audio is sent to the model.
However, we are working on turn detection within the framework, so you won't have to send silence to the model when the user isn't talking. It's a fairly straight forward path to cutting down the cost by ~50%.
it works well for voice activity; though it doesn't always detect end-of-turn correctly (humans often pause mid-sentence to think). we are working on improving this behavior.
Can I currently put a VAD module in the pipeline and only send audio when there is an active conversation? Feel like just that would solve the problem?
Speculating here, but I would read this as "anycast" as a concept, where each user is connected to the closest location. versus anycast as in the IP protocol. The complexity far outweighs benefits with routing each UDP packet to different servers within the same session.
> However, it's still a lowest common denominator solution with respect to what codec is being used. The broadcasting "peer" has to send using a codec that every other peer supports. Essentially that means you're stuck with the lowest common denominator codec.
This is correct, though I wouldn't call them "inferior codecs": all WebRTC capable browsers support both H.264 and VP8 as required by the specs.
It's fine to dislike Medium and run your own blog, but I find the article intellectually dishonest in comparing Medium to Facebook. One harvests your data and lets advertisers target you with ads, the other charges a subscription to provide access to content.
It's disingenuous to claim they "sell whatever advertising they please", and link to a ToS screen cap that's completely unrelated.
Companies have to make money in order to pay their developers and pay for servers. Personally, I'd rather pay them instead of seeing ads plastered everywhere.
"How quickly can we get this into people's hands? If you read the papers, you see maybe it's three years, maybe it's thirty years. And I am here to tell you that honestly, it's a bit of both."
This statement is based on what he is aware of at the time. The most pessimistic side of it is purely a data game, and Tesla's ability to have 100k cars collecting data for them will give them a significant advantage to accelerate their progress.
(btw I work at LiveKit, so let me know if we could make Agents easier to use for your use case.)
reply