Transcript: Auphonic - Georg Holzmann

This week we're joined by Georg Holzmann, the creator of Ophonic, which handles audio processing and uses Django.

We'll dive into all those things, how to scale it, and those should be a great discussion.

So Carlton, how are you doing?

Thank you very much.

Very well.

How how are you, Georg?

Uh I'm fine, and thanks for the invitation.

I'm curious what to speak about here.

Um I guess let's start with how did you get into programming?

And there's so much we can speak about the audio part, but also around the Django part.

Um so maybe what's your you know what's your what's your background?

How did you get into programming?

So I think I was in primary school and my dad always had an old computer, and I always managed to destroy the computer and play around with it.

And at some point, yeah, I started programming.

I don't know why actually, I think some some friend was was uh showing it to me, and the first programming language was Turbo Pascal, and then I learned C and C and all these more low-level things.

And yeah, and into Python I came actually quite late.

It was at the university.

And did you study computer science there then?

I actually studied audio engineering.

Okay, but this was uh a lot, it it was not really only audio, it was more like electrical engineering and computer science, and a lot of signal processing, machine learning, and all these things.

And so basically it was uh computer science, yes.

Okay, and this is also why I got into Python at some point because uh we were using um for for a project NumPy and SciPy and all these things when they were at their early stage.

And yeah, so so I came into Python and I loved the language a lot, and so I got into programming.

So it started as a web service.

Now there's there are also some desktop apps, but you can basically upload an audio file, and then this file is analyzed on our servers, and uh then according to the analysis, it is processed and encoded into different formats and then distributed to other services and things like that.

Um the signal processing, the cleaning up.

Yes, exactly.

Sound qualities.

And so I guess the question is is most of that done in Python or is that done in you know other languages?

So we use a lot of numpy and uh the machine learning libraries, different machine learning libraries, and yeah, also some parts are also uh written in C, so which are optimized a little bit, and for this for that we mostly use uh Cyton, if you know this, yeah, yeah, yeah.

Which which is basically um, I guess everyone on your podcast knows this, but it's basically a compiler from Python to C.

And yeah, then then we also had to build a a web system around it, and I think this was 2009 or 2010, and back then there was basically everyone was using Ruby on Rails.

Yeah, this was very popular, and I was was thinking, yeah, I don't want to get in a whole nother software stack, and I was looking at what's going on in the Python world as well, and found Django, and yeah, since then I'm a Django user.

So you've been through the long haul, you've but you know, like um you've seen the introduction of um, I don't know, let's say migrations and the big changes around 1.5 and then one seven and

This was the framework before the migrations were introduced.

Right.

And yeah, but but but my my main focus was never was never mainly on on the web tools, it was more on the uh signal processing and machine learning part and the web tools.

I just discovered them by um next to to the other things, you know, it was never my main focus.

Yeah, okay.

So of course the first thing is the the web GUI, so how you see the website or how you can create these productions, these audio productions.

Then so this is the client side, front end, then there is uh the the server side.

So after you upload an audio file, you have to process and analyze it, which which takes uh quite some processing power.

So we we uh need to have a distributed system where we can uh process all the audio files, so it's queue.

For that we use this uh DJ Celery and uh RabbitMQ.

Because I mean you've been obviously at this for quite a while now, and so you'll have you'll have seen it evolve.

You tell you must have some war stories there.

I mean, just last night I was having an issue with our podcast host, which seems far simpler uh than what you're doing.

So however you're doing it, uh it's uh it's very speedy, and I've never had an issue, which is unusual dealing with audio for me online.

Well, there were of course issues in the past.

But uh not for you.

I mean, there there were some issues at some point, also at some migrations uh there were some problems, but no major things, so I cannot tell anything else anything bad about it.

Oh, you know, I've um I've I've had scars of trying to make that work how I've wanted to, and um now I'm bit for quite a few a while now I've been firmly in the Redis camp and using that and working with that well and finding that's very scalable.

But um, you know, Rabb Rabbit MQ is still there and still has lots of adherence, so I'm you know just wondering you know, the the reason I ask is to try and draw out your your experience, which sounds very positive.

Right.

Because because if you process one audio file, so this usually takes uh it depends on your file, but it takes quite long.

It's not yeah, there are not thousands of small tasks, but there's just a few long-running tasks.

So yeah, I I I think for our use case it does not matter much if you use RabbitMQ or something else.

So it's basically the same, I guess.

Yeah, okay, interesting.

Because I imagine you have spikes of activity and then inactivity.

I wonder if uh how what does your traffic look like?

Do you have those spikes or is it more kind of level?

How does it look?

Because I think I know you're global.

So it's the average is so the the standard deviation is not so big, I would say, because yeah, you know, in Europe they are processing now, maybe, and then in the US it's it's a little bit later.

So it's it's not such a problem.

But of course, the the the infrastructure, so maybe let's speak a bit about the infrastructure.

Yeah, yeah.

So we we do not use uh Amazon AWS or things like that.

So we have um root servers which are rented because they are much cheaper for our use case because we need a lot of uh uh memory and a lot of processing power.

Yeah, therefore, of course, a lot um many root servers don't do much most of the time, but then when we have spikes in processing, we need all of them.

So, of course, these these spikes are important and determine how many uh servers we have to use, but in the end, it's still I would say it's still much cheaper compared to to AWS or other services.

Because this is one of the the sort of selling points that Amazon tell us, or you know, it's not just Amazon, it's all these providers, they sort of say, look, that you're not bus you're not having to go to the data center to change the hard disk to monitor, you know, these kind of activities that you had to do back when you were um renting space in a colo.

But the hardware is managed.

So I mean, maybe you know this hostel, it's Hetznar from Germany.

They are yeah, they are a very big hostel in Germany and very cheap for their for their root service.

Right, okay.

Because you know, to me, you're just a service that I've seen recommended and that works.

Um so I'd love to hear more about yeah, it's the company if there are other people involved and as much as you feel like you can say about that.

So uh yes, I've yeah, about five people.

Okay.

uh the the amount of audio we process?

Okay.

I mean, uh are you just has it been linear?

Have you spiking?

Um it's not linear.

I mean it is a little bit linear as well, of course, but then there are always spikes if some other user groups find out about this service and then they bring in a lot of more people, then there is a spike.

And but then it just evolves linear.

I so it's it's spikes and linear.

So I'm curious how is there what's sort of the mix of comp services that use Ophonic versus people like us who do it directly?

Do you have a sense of what that that mix is, kind of companies versus individuals using it directly?

So we have an API where for the other listeners, we have an API where uh other services can integrate our uh services and algorithms so that the users of the third-party services don't have to know that there is a phonic insight, like you did with Zencaster, for example.

Right.

And so there are some companies which are integrating our services into their systems, and yeah, then there is our website and and direct interface, like you are using now, I guess.

And so yeah, this is the share is half half, I would say.

Because I think I the only reason I found you is because I think uh I saw Tim Ferris mention that he used you directly, and I thought, oh, that sounds like what we're using with Zencaster.

Um, and then Carlton and I had a number of issues with doing it web-based, and um, so it was really kind of circumstantial that I even found Ophonic itself.

Is there a standard path for how clients kind of find you?

So this basically we don't make any marketing or things like that.

So usually it's it's word of mouth, especially in the bot.

It's kind of

You built it and they came.

So so at the beginning, so at the beginning we had um so all all started with a podcaster from Germany, because he I was listening to his podcast, and he always told, Yeah, everything is so complicated, all the audio things and generating all the file formats.

And then I thought, well, that's actually quite easy, so let's build something.

And it started with some small scripts.

And this is really one of the famous podcasters here in Germany.

So after we released it, uh, a lot of people used us already.

Right, okay.

And yeah, then we got a grant from the government here in Austria to so we got some money basically and could uh build uh a prototype or continue to build the system.

And with this money I could uh hire someone who helped to build the system, and then actually another one as well.

And then after after this this grant, uh already some some other broadcasters uh got to know our system.

I don't know why, but somehow it spread over the ocean to the US.

And there there were other people which were playing around with it.

And yeah, afterwards, after this grant, we had to to introduce a pricing model.

So in the beginning everything was free for everyone, right?

And then we had to to introduce some pricing because otherwise we could not live from it anymore.

You're not allowed to um take your time.

Yeah, really.

So it sounds like that's it's that's fantastic that you you had that uh ability instead of being pressured to introduce things faster before they were ready.

And um well the the nice thing we we had in our situation is that we did not have to do any fancy uh marketing or other kind of strategies, how to get more users.

I mean that's that's good to do if you're interested in these things, but we were all engineers basically, and we did not know anyone who can do this these other things.

So we just tried to to avoid them.

I I've seen you've been on a couple of podcasts.

Um so that that's marketing.

Of course.

Um that's yeah, that's amazing.

Well, uh so my a question I had is are there um when you created this, were there any competing services that do that?

And kind of what do you see in terms of competition?

Because it seems I I'm not aware of any, um, but it seems like such a now that we have it as part of our post-production flow, um, I can't imagine doing our podcast without it.

I mean

there are of course other audio uh software companies like Isotope, which which more build their tools for for editing.

So so basically there is no such automated way like like we do it.

So that's that's why we try to focus on this automation and workflow aspect, and and not to not try to build an editor, which would be of course also very useful.

But it's of course a lot of work if you want to build it right, and and then you get into direct competition to audition or isotope or other companies.

Like so for me to be able to upload the file and it comes back and that the with the noticeable difference in sound quality, you think, yeah, that's amazing.

That's just brilliant.

So in our system, basically you get uh this is exactly the the initial, this was exactly the initial goal when we started the system, so that users which don't have a lot of audio knowledge or also users which have a lot really lots of audio which cannot be handled manually, so that they can just use this tool and get out audio, which is okay.

But of course, if you have very specialized use cases and you want to get out every detail, then these uh these editor tools are of course much more powerful because you you can really uh work on every detail, but of course it needs more time and knowledge.

Um but I guess one thing they always talk about in product development is to focus on your particular niche and don't worry about trying to serve the other niche.

So you know that you don't have these features is it's a selling point, right?

Yeah, it's very clear what you do.

Well, so maybe we can talk about some of those features because um there's a lot that are awesome.

I wonder what would you say or what are the main features that people kind of come to you with?

I mean, I mean for us, I mean, when I'm doing the audio files, I love that you have the presets just in terms of the web base.

That's great that you know, because we do the same thing for every podcast, so I can just load our presets and then we usually do you know, basically go for almost everything.

So the compression, leveling, normalization, noise hum.

I guess that's a broad question, but as you think of the features that you have now on the site, what do you think are the core features?

And then we can get into some of the more advanced ones because I know you have some new features you've just launched.

So I would say the most important feature where everything started is our leveler.

So what

does it do?

It um levels out the audio, which means that if you have multiple speakers like like we are having a conversation now, then all these speakers can have different levels and loudness uh values, and uh some you have to balance them, so otherwise you would always have to use your volume control and adjust for the levels.

But this is actually a very complicated task because it is easy if you just have speakers, but in audio you also have then bigger parts where there is, for example, just background noise.

Then, of course, you should not amplify this background noise like you would do it with speakers.

Or then there are also music parts which should be handled quite differently because music in music you want to have more inner dynamics, and if you have speakers, you want to sound them equally loud.

But music should have some differences, of course, and therefore you you have to analyze the audio first and see where are different speakers, where are music parts, there are just background noises and things like that.

So basically, like an audio engineer would do it, and then you have to balance these different parts and use compressors and limiters and things like that.

So I when you're talking about the different parts, I'm I'm imagining, you know, my machine learning thinking and thinking, well, okay, so are you using are you tagging particular parts as this this kind of so this is speech, this is music, this is you know, some other category, and then you'll apply a different filter or set of filters depending on what it gets tagged by the machine learning algorithm.

Yes, exactly.

So basically, this algorithm first analyzes the audio and sees uh where are different background noise scenarios.

For example, if you record in a room and then we go outside, then there is another background noise scenario outside.

So we have to uh segment the audio first in these parts and then do noise reduction in the first part and then the second one.

You'd have to you'd have to do it by hand, you'd have to sort of manually identify the the segments.

We try to automate these these things by machine learning and then just apply the algorithms like you would do it in an audio editor, for example.

Wow.

Well, and then you also have I mean there's um chapter marks, which I believe is that's that's something you had before the sort of Apple podcasts and stuff had as a feature, is that right?

So this is a very complicated topic because there is so much confusion about it.

So what this chapter marks, a chapter mark is basically uh just a timestamp and a title.

So you say at this time this chapter starts like like video does it quite long already.

And the problem is with audio that there are many file formats like mp3, mp4, or opus or whatever, and usually this chapter marks were only defined for video.

So in the mp4 container, everything was defined quite well.

It is still very complicated, but at least it was defined.

And usually at the beginning, Apple was mostly using MP4 audio, so AAC audio in an MP4 container.

And the Apple podcast app always, or since very long, supported chapter marks in MP4 files.

But most podcasters use MP3 files, so they did not recognize chapters in MP3 files.

And there was a very old specification for for mp3, so for actually for ID3, which is the metadata standard for MP3, how to put chapter marks also in MP3 files, but nobody used this specification.

And yeah, then we just implemented the specification basically, and then more and more botcatchers added support for it.

And I think now since last year, uh also the Apple Podcast app supports MP3 chapters as well.

Yeah, for enough.

Well, speaking of mobile apps, I mean you have I'm so impressed by I keep finding new features that you have.

I mean, you you have mobile apps as well, right, for Alphonic for Android and iOS.

Yeah.

And so what's the or how recently was that?

How did that come about, right?

Because that's another thing to build and maintain on top of everything else.

Uh well, the thing, basically, we started the mobile app because a friend of the first developer at Alphonic uh wanted to do a project with us, and he's a web and mobile developer, and yeah, now he works at Facebook, so he's quite quite good in these things.

So he he said yes, he he wants to build a iOS app for us.

And the problem in iOS was was long that you cannot upload audio files in in a web site.

I think they changed it now, but for a long time you could not select audio files and upload them to a phonic, for example.

So the so the idea was uh to build just a simple recorder and then uh use our API in the mobile app so that you can basically export files from your phone to to a phonic.

And yeah, this this was the start basically, and then we also did an Android version.

And on Android the situation was even more complicated because there is no usable audio editor on Android, and then we thought, well, we could also build a little audio editor.

Because that's the great that that's the great challenge on Android, right?

Is it works on your Samsung that you've got in the office, but not on you know any device out there in the street.

Okay, interesting.

Yeah, so if you have problems, talk to Carlton.

Yeah, or

So thanks a lot.

But um, if you'd um so you've been using that from I mean REST Framework for well, f it wasn't quite around in 2010, but like 2012, 2013?

I think, yeah.

Right.

Because that's sort of the front door for everything.

Um I'm just curious of you know the where you where you spend your time now given that you're a scale and you're still intro um we can talk about I know you just introduced um a new a new leveling algorithm.

So sometimes we work more on the um web part or other parts, and sometimes more on the algorithms.

But yeah, uh I mean scaling.

Scaling was not so much of an issue, I would say.

So well basically we always build some we always have to fix some things if if we see uh it will get very hot.

But but yeah, actually, I mean, you know, if if the traffic go uh goes up, we just ransom more servers.

So basically that's not so complicated.

Right.

Because in in our case, I mean the website itself does not need much of scaling because we don't have hundreds of thousands of users every minute, of course.

So the the the database or the all all the web uh front end and things they are just on on one uh big root server, and then we have various other servers for for the audio processing and for the uh for the long-running tasks.

Processed audio files, I believe it's is it 30 days that you store it and then it goes away?

Is is that correct?

How did I assume that came about because you looked around and said, Oh my god, we have all this data.

Or did you have that from the beginning, uh sort of a limit on how long you would host uh processed audio?

Yeah.

There are of course also uh data protection uh reasons.

So it really is all all hardware in in in the um data center.

I always like to ask this question.

So if you if you could just w wave a magic wand and add a new feature to Ophonic, uh what would it be?

Oh because I assume you you have I mean obviously I don't have a sense of what is truly challenging or what you think your customers are demanding.

Our future feature list is very long.

Uh or maybe there's something that you don't even know how to tackle it, but it's sort of an unresolved problem in processing audio.

Uh because again Carlton and I don't know that space at all.

Yeah, okay.

What would be really cool is if you have, let's say you have um very bad audio like MP3 with uh 32 kilobits, so everything is compressed already completely and you can't hear anything about it anymore.

And then one could build an algorithm which makes the audio as the original again.

Right, okay.

But yeah, this is of course not possible for every situation, unfortunately.

Where someone can link up a third-party uh transcription

Well how and then was that something that you had again from the beginning or something that users asked for um that integration?

Because it looks it looks really nice because that that is another step in the tool chain of producing a podcast where uh do the leveling and then transcription.

Um I mean it's something we should we should add.

So if you would have a transcript, then you can search within the podcast.

So that's of course perfect.

But yeah, some years ago there were no services which produced uh reasonable output for a reasonable price.

Well said.

And yeah, now there are some there are various others other APIs as well which can do that.

So and back to the beginning.

So actually, we wanted to build our own speech recognition system, but then we we we we thought we cannot do everything, but it's because it's it's it's really uh very time consuming, especially if you want to build that for multiple languages.

So you have to have specialists for every language and a lot of data for every language.

That's why we decided to integrate various third-party services for speech recognition.

So we we still do our own both pre-processing and uh slice the audio into small parts, so cut out the music parts and uh take out only the speakers, and then send these slices to the external services, to the speech recognition services, and then uh combine it again so that you have the time codes in the transcript and so that you know at which time uh who is speaking what.

We should be using this.

This is this sounds amazing.

You're welcome to uh help on that.

Um yeah, I think it's uh it's so brilliant how you've you've stayed kind of within your lane yet still expanding features because uh um I mean podcasts in general are are going through quite a bit of consolidation right now.

Um, but it seems like what you're doing is so much better and so hard to do well that it's not really as maybe appealing for Spotify or someone to say, oh, we'll just do Ophonic.

Or perhaps maybe or or is that an existential threat, do you think, down the line, um, that one of these places will say, we'll just scoop that part in?

Because again, unless you really know what you're doing, I don't it seems very difficult to have a high quality podcast without using uh a tool like Ophonic.

But most but that you know that's what 1% if that, you know, everyone else who's um uh you know, because uh yeah, yeah, thanks.

Um yeah, I think our our service is just very specialized and there is just a lot of time already put into it how to how to uh really optimize it for this use case.

I mean of course someone else can build similar things, but it's it's of course a lot of work and you would need I think quite a bigger team than than we are if you don't build it for yourself.

And sure.

Yeah, right.

Is it um the size of files processed?

Is it you know uh hours?

What when you when you internally look at your metrics, what are the how do you manage growth?

Because I think that there's a number of different ways to potentially measure that.

So in our system you you have you need so two hours of audio is free for everyone in our system, and uh if you process more audio, then you can buy additional credits, and the credits are in in hours of audio.

So if you have one hour of credits, you can process one hour of audio file.

So therefore, therefore, of course, the most important measure I would say is the hours of audio we process maybe.

Was was that uh

But then if you process the file, you have to decode the file again to get the raw data.

So and then it doesn't matter if it was an MP3 or WAV file because you have the same you need the same processing amount or processing power for it.

So I'm I'm curious you because you went to university for audio engineering.

What what did you think you were gonna be doing at this point in your career and kind of what are your friends from university doing?

Because I'm sure they're not all doing startups.

So well, actually after university, I started uh so I did my master's studies here in Austria, and then I started a PhD in Germany, in Berlin.

Also about machine learning and audio.

It's it was it is called this field is called music information retrieval, so where you try to extract information out of audio with machine learning techniques.

But yeah, then I did not finish this PhD because basically I was the only one there which did these things.

And then I started the job at a web company and get to know all these web things, also Django, so they also use Django there.

And yeah, then I thought, okay, I can also build something for myself, and I also get to know this podcaster which always had these problems with audio processing and encoding, and I just tried to build these things myself.

You know, you've got you've done the signal processing, you've done the machine learning, you've done you you're into the audio stuff, you've done the web programming, it's like you've got all the parts together and you've just created this awesome business out of it.

Well, yeah, it's the easiest lesson, isn't it?

But at that time I was interested in in in web things because I did never do big uh web projects, so I just wanted to get into this topic and yeah, for that it was quite well.

Uh there's there's some some efforts on it, but basically it has to be hands-on.

And even something like Django is largely not taught at all, and if it is taught, it's by an adjunct and it's sort of an elective.

So that's hopefully that will change, but it's it's hard to get a backbone in web technologies, even though the majority of uh undergraduate computer science graduates probably go work on something web related, so there's definitely a uh educational mismatch there, right?

So, what are you working on right now?

But we also have a desktop application with our algorithms.

So they don't include all the features of the web service.

But the the advantage here is of course that uh you don't have to do the processing in in the cloud and you don't have to upload files, so the processing is just done on your computer, which needs, of course, processing power, but you don't have to upload download.

And yeah, we also have a different uh pricing model for this desktop app, so it's just a one-time purchase.

Uh varies in the web service, you have to buy credits, so based on how much you process.

But uh yeah, what I wanted to tell is that we are working on a new uh version of the desktop software at the moment because it was very difficult to get all these Python and machine learning tools to desktop computers, and especially difficult was the GUI part because we we used in the in the desktop apps we used this WX widgets framework with with their Python, and it was so frustrating and so diffic

ult.

And now we uh did rewrite everything with web-based tools, so we use now Electron and uh combined with Python.

But yeah, Electron is still is still a very big uh framework and not so nice to handle, but at least now everything is is HTML-based, the whole user interface and the backend is all done in Python, and so as a long as a long-term goal, we could maybe use the same interface for the web and for the desktop version, and just reuse all the components and use Python as the processing engine.

Yes.

First, we are compiling everything to see with Cyton and and all these uh tools, okay.

And we are using this tool is called PyInstaller, which which creates such a bundle with Python and all the uh develop uh all the dependencies you have, and then this is bundled into one binary, and you can afterwards distribute this binary.

Okay, fantastic.

Okay, so and then the electron app just uh also takes this Python uh binary and includes it in in their framework.

Wow, yeah, that's really cool.

Yeah, yeah, it's the

Awesome.

Well, well, thank you so much for coming on and and sharing this story.

Um we love using Ophonic, and I actually what I love in a way too is that the only reason I thought that maybe you use Django is I saw your the sign-up page and you you're using slash accounts and username, email, password, and I thought, oh that looks that looks like Django.

And I just sent you an email and it's like, oh yeah, it is Django.

Yeah.

Um so there's all these, there's all these, you know, a big thing for us in this podcast is to try to highlight all the different ways Django is being used in all these different realms, um, which are largely kind of hidden because it's you know for you it's a s sort of a secondary thing, but still a key part of your you know pipeline of your process.

That is the perfect end.

That the next release isn't gonna totally change the API or you know exactly.

A lot of such things.

And those stability guarantees and the deprecation policy are important.

Well, actually, maybe maybe one last one.

If uh if there's something you could change about Django, or or what would that be?

Or or or do you have thoughts on maybe how the the future async wave um would that impact uh Ofanic at all, or is that not so much?

I I did not do so much with it already since now.

Because I mean the thing is our processing is done in a different stage, so it's done offline basically in the queue.

So here we don't would not need this async thing.

But yeah, I think for for user interface improvements it's it's of course very very very interesting, but I have to to play with around with it a little bit more, I think.

Yes, sure.

So two hours per month are for free.

And if you have any questions or feedback, we are always very, very happy to get feedback and also error reports and whatever.

Okay, thank you so much for coming on and giving us the time.

Thank you.

Bye bye.

Thank you, ciao.