Thank you, great question! Hard one to answer, thought about it a lot and it's going to be the diarisation of more than 5+ speakers per audio stream (your microphone + system audio for a max of 10). I actually spent a lot of time that went completely nowhere trying to fine tune my own diarisation model, it was fun to a degree but painful to see my output end up worse than what I currently had after days of work. Having a bot join the call would be such an easy way of diarising, the call software has already done it for you, but feels like a bit of a cop out.

Two more improvements, audio quality improvements which is currently in the works and close to release and a new document generation model. I'm currently using a custom fine tuned Phi-4 (released December 2024!) model, that's _so old_ in the grand scheme of LLMs, I just haven't had time to benchmark and properly test some new models whilst this currently does a good job as it is. There has to be some gains here, but who knows!

Interested also, which text-to-speech model do you use? For diarisation Granola uses a chrome extension instead of a bot if that can give you ideas.

I think this is just to capture the audio to ship off to their servers and do the crunching. You mentioned the word "extension" and thank you so much, I've been thinking about how to do integrations as it's something a few users have mentioned, but keep the whole "completely offline" angle. I could build standalone extensions that integrate with it if it's something a user wants. So damn obvious in hindsight!

Ahh yea as for the models:

Speech to text - Nvidia Parakeet TDT 0.6b V3

Diarisation - Nvidia Marblenet for the speech detection, TitaNet-Large for the embeddings and then using NeMo multi-scale to do clustering around them

Since the participants are known and limited, have you tried building around samples tagged with user/person names?

I tried something similar by putting together a "global ledger" of speaker identities. This was more happening during the call than having a predefined one but I just couldn't get it to work properly. The issue being that as soon as some speech gets assigned to a new label or an incorrect one, everything ends up getting misaligned and gets messy quickly.

I might take another look into doing it in a different way that gradually builds up from successful calls, I just need to think of how to do this in a simple(ish) way for non-technical users and a way that still works well enough on low to mid tier laptops.

I know this is hn and not a product development meeting, but:

If you’re targeting data for people to use on the same call then requires more intense work, while if you’re targeting data for people to use after or on an ongoing basis (eg an established business meeting with staff/vendors/etc) then having a predefined one seems good. Same interface as a CRM: picture, some audio clips for users to choose from, and confidence scores on each section of the recording for cases where people sound similar or are talking over each other. Over time the diarization gets better as users accept/reject/tag samples and that work helps them feel more aligned with the tool.

100% looking at it from an after use basis, whilst there is live transcription, trying to do anything above that on 6 year old laptop CPU just grinds stuff to a halt.

Some great suggestions, appreciate you taking the time to write it out, it's given me some things to think about.