Transcript: Auphonic - Georg Holzmann

the creator of Ophonic, which handles audio processing and uses Django. We'll dive into

all those things, how to scale it, and it should be a great discussion. So Carlton,

you get into programming? And there's so much we can speak about the audio part, but also around

the Django part. So maybe what's your, you know, what's your what's your background? How did you

in primary school and my dad always had a old computer and I always managed to destroy the

computer and play around with it and at some point yeah i started programming i don't know why

actually i think some some friend was was showing it to me and the first programming language was

turbo pascal and then i learned c++ and c and all these more low level things and yeah and into

python i came actually quite late it was at the university okay did you study computer science

there then not really i actually started audio engineering okay but this was a lot it was not

really only audio it was more like electrical engineering and computer science and a lot of

signal processing machine learning and all these things and so basically it was computer science

for for a project numpy and scipy and all these things when they were at the early stage and yeah

so so i came into python and i loved the language a lot and so i got into programming okay and um

also about audio processing and some other things so it started as a web service now there's there

are some desktop apps but you can basically upload an audio file and then this file is analyzed on

our servers and then according to the analysis it is processed and encoded into different formats

and then distribute it to other services and things like that.

that you studied at school, right?

The signal processing, the cleaning up.

Yes, exactly.

The sound qualities. Yeah, okay.

And so I guess the question is, is most of that done in Python

or is that done in, you know, other languages?

are mainly built in Python.

So we use a lot of NumPy and machine learning libraries, different machine learning libraries.

And, yeah, also some parts are also written in C, which are optimized a little bit.

And for that, we mostly use Cyton, if you know this.

Yeah, yeah, yeah.

Which is basically, I guess everyone on your podcast knows this, but it's basically a compiler from Python to C.

And yeah, then we also had to build a web system around it.

And I think this was 2009 or 2010.

And back then, there was basically everyone was using Ruby on Rails.

And this was very popular.

And I was thinking, yeah, I don't want to get in a whole nother software stack.

And I was looking at what's going on in the Python world as well.

And found Django.

And yeah, since then, I'm a Django user.

So you've been through the long haul.

You've seen the introduction of, I don't know,

let's say migrations and the big changes around 1.5 and then 1.7.

This was the framework before the migrations were introduced.

Right.

And yeah, but my main focus was never mainly on the web tools.

It was more on the signal processing and machine learning part

and the web tools.

I just discovered them next to the other things.

It was never my main focus.

No, you go ahead, Carlton.

Well, I was going to say, so the main uses for Django here

is to build the API around the data processing pipelines that you have.

So, of course, the first thing is the web GUI.

So how you see the website or how you can create these productions, these audio productions.

So this is the client side, front end.

Then there is the server side.

So after you upload an audio file, you have to process and analyze it, which takes quite some processing power.

So we need to have a distributed system where we can process all the audio files.

So at DasQ, for that, we use this DJ Celery and RabbitMQ.

Because, I mean, you've been obviously at this for quite a while now, and so you'll have seen it evolve.

You must have some war stories there.

I mean, just last night I was having an issue with our podcast host, which seems far simpler than what you're doing.

So however you're doing it, it's very speedy and I've never had an issue, which is unusual dealing with audio for me online.

Nope.

Yeah, so, yeah, for us, RabbitMQ and Celery worked quite well.

I mean, there were some issues at some point.

Also, at some migrations, there were some problems, but no major things.

So I cannot tell anything bad about it.

It's more like as you scale out these things.

Well, Carlton has feelings on Celery.

no and on rabbit on amq you know i've um i've had scars of trying to make that work how i've wanted

to and um yes now i'm a bit for quite a few a while now i've been firmly in the redis camp and

using that and working with that well and finding that's very scalable but um you know rabbit mq is

still there and still has lots of adherence so i'm you know just wondering you know the reason i ask

is to try and draw out your your experience which sounds very positive yeah so our setup is maybe

process one audi file so this usually takes it depends on your file but it takes quite long

it's not yeah there are not thousands of small tasks there's just few long running tasks so

So yeah, I think for our use case, it does not matter much if you use Revit MQ or something else.

So it's basically the same, I guess.

Because I imagine you have spikes of activity and then inactivity.

I wonder, what does your traffic look like?

Do you have those spikes or is it more kind of level?

How does it look?

Because I know you're global.

so the the standard deviation is not so big i would say because yeah you know in europe they

are processing now maybe and then in the us it's it's a little bit later so it's it's not such a

problem but of course the the infrastructure so maybe let's speak a bit about the infrastructure

yeah yeah so we we do not use uh amazon aws or things like that so we have um root servers which

are rented because they are much cheaper for our use case because we need a lot of uh a memory and

a lot of processing power yeah therefore of course a lot many root servers don't do much

most of the time but then when we have spikes in processing we need all of them so of course

these these spikes are important and determine how many servers we have to use but in the end

it's still i would say it's still much cheaper compared to to aws or other services and so i

is one of the sort of selling points that amazon tell us or you know it's not just amazon it's all

these providers they sort of say look that you're not you're not having to go to the data center to

change the hard disk to you know these kind of activities that you had to do back when you were

we have to manage the whole software so from the operating system up to all the stacks you have to

install but the hardware is managed so i mean maybe you know this hostel it's heads now from

germany they are yeah they are very big hostel in germany and very cheap for for their for their

sort of traffic you're dealing with now versus i guess 10 years ago or what eight nine years ago

when you when you started because you know to me you're just a service that i've seen recommended

and that works um so i'd love to hear more about yeah it's the company if there are other people

involved and as much as you feel like you can say about that yes sure so uh yes we are about five

linear as well of course but then there are always spikes if some other user groups find out about

this service and then they bring in a lot of more people, then there is a spike.

And, but then it just was linear.

So, so it's.

so when we when we started using this podcast we started using a service called zencaster

which i found out uses you so i'm curious how is there what's sort of the mix of services that

use ophonic versus people like us who do it directly do you have a sense of what that

that mixes kind of companies versus individuals using it directly okay yeah this is maybe half

can integrate our services and algorithms so that the users of the third-party services don't have

to know that there is a phonic inside like you did with zancaster for example right and so there

are some companies which are integrating our services into their systems and yeah then there

is our website and direct interface

like you are using now, I guess.

And so, yeah, the share is half-half, I would say.

Because I think the only reason I found you

is because I think I saw Tim Ferriss mention

that he used you directly.

And I thought, oh, that sounds like

what we're using with Zencastr.

And then Carlton and I had a number of issues

with doing it web-based.

And so it was really kind of circumstantial

that I even found Ophonic itself.

is there a standard path for how clients kind of find you no i i don't know so there's basically

from germany because he i was listening to his podcast and he always told yeah everything is

so complicated all the audio things and generating all the file formats and then i thought well

that's actually quite easy so let's build something and it started with some small scripts

and this is really one of the famous podcasters here in germany so after we released it a lot of

and then we got a grant from the government here in austria to so we got some money basically and

could build a prototype or continue to build the system and with this money i could hire someone

who helped to build the system and then actually another one as well and then after after this this

grant already some some other podcasters got to know our system i don't know why but somehow it

spread over the ocean to the us and there there were other people which were playing around with

it and yeah afterwards after this grant we had to to introduce a pricing model so in the beginning

everything was free for everyone right and then we had to to introduce some pricing because

otherwise we could not live from it anymore yeah because you initially you were funded by the

take a i mean you seem to have a long-term approach to this whereas i mean i used to be

out in san francisco and if you take venture capital money you pretty much within 18 months

and really 12 months you have some very high goals you need to hit you're not allowed to

um take your time yeah really it sounds like that's that's fantastic that you you had that

ability instead of being pressured to introduce things faster before they were ready yeah sure

marketing or other kind of strategies how to get more users i mean that's that's good to do if

you're interested in these things but we were all engineers based kelly and we did not know anyone

who can do this these other things so we just try to to avoid them well this is marketing now i've

that's amazing well so my a question i had is are there um when you create this were there any

competing services that do that and kind of what do you see in terms of competition because it

seems i'm not aware of any but it seems like such a now that we have it as part of our post-production

flow i can't imagine doing our podcast without it there are no direct competitors which which

also other spoken word recordings i would say like conference recordings and lectures and things like

that i mean there are of course other audio software companies like isotope which which

more built their tools for for editing so so basically there is no such automated way like

like we do it so that's that's why we try to focus on this automation and workflow aspect

and and not to not try to build an editor which would be of course also very useful

but it's of course a lot of work if you want to build it right and and then you get in the

direct competition to audition or isotope or other companies yeah and those tools those tools

beginner and there's just no way i can apply the right filters and you know in anything like a time

efficient way like so for me to be able to upload the file and it comes back and the with the

noticeable difference in sound quality you think yeah that's amazing that's just brilliant yeah so

the the initial this was exactly the initial goal when we started the system so that users which

don't have a lot of audio knowledge or also users which have a lot really lots of audio which cannot

be handled manually so that they can just use this tool and get out audio which is okay but of course

if you have very specialized use cases and you want to get out every detail then these

this editorial tools are of course much more powerful because you you can really work on

every detail but of course it needs more time and knowledge yeah um but i guess one thing they always

trying to serve the other niche so you know that you don't have these features is it's a selling

features because um there's a lot that are awesome i wonder what would you say what are the main

features that people kind of come to you with i mean i mean for us i mean when i'm doing the audio

files I love that you have the presets just in terms of the web base that's great that you know

because we do the same thing for every podcast so I can just load our presets and then we usually do

you know basically go for almost everything so the compression leveling normalization noise hum

I guess that's a broad question but as you think of the features that you have now on the site what

do you think are the core features and then we can get into some of the more advanced ones because I

know you have some new features you've just launched yes so i would say the most important

means that if you have multiple speakers like like we are having a conversation now then all

these speakers can have different levels and loudness values and some you have to balance

them so otherwise you would always have to to to use your volume control and adjust for the levels

but this is actually a very complicated task because it is it is easy if you just have speakers

but in audio you also have then bigger parts where there is for example just background noise

then of course you should not amplify this background noise like you would do with the

speakers or then there are also music parts which should be handled quite differently because music

in music you want to have more inner dynamics and if you have speakers you want to sound them

equally loud but music should have some differences of course and therefore you you have to analyze

the audio first and see there are different speakers where music parts there are just

background noises and things like that so basically like an audio engineer would do it and then

you have to balance these different parts and use compressors and limiters and things like that so

parts i'm imagining you know my machine learning thinking and think okay so are you using are you

tagging particular parts as this this kind of this is speech this is music this is you know some other

category and then you'll apply a different filter or set of filters depending on what gets tagged by

things like different speakers music bands and different noise parts or if the background noise

changes for example this yes this is another algorithm we we have a noise it's called noise

reduction so basically this algorithm first analyzes the audio and sees where are different

background noise scenarios for example if you record in a room and then we go outside

then there is another background noise scenario outside so we have to segment the audio first in

these parts and then do noise reduction in the first part and then the second one right and

have to do it by hand you'd have to sort of manually identify the the segments yes in in

steps you have to do by hand we try to automate these these things by machine learning and then

just apply the algorithms like you would do it in an audio editor for example okay okay fantastic

something you had before the sort of Apple Podcasts and stuff had as a feature.

Is that right?

So this is a very complicated topic because there is so much confusion about it.

So what this chapter marks, a chapter mark is basically just a timestamp and a title.

So you say at this time, this chapter starts like video does, it's quite long already.

and the problem is with audio that there are many file formats like mp3 mp4 or opus or whatever

and usually these this chapter marks were only defined for video so in the mp4 container

everything was defined quite well it is still very complicated but at least it was defined

and usually at the beginning apple was mostly using mp4 audio so aac audio in an mp4 container

And the Apple podcast app always, or since very long, supported chapter marks in MP4 files.

But most podcasters use MP3 files, so they did not recognize chapters in MP3 files.

And there was a very old specification for MP3, so actually for ID3, which is the metadata standard for MP3,

how to put chapter marks also in MP3 files, but nobody used this specification.

and yeah then we just implemented the specification basically and then more and more podcatchers added

support for it and i think now since last year also the apple podcast app supports mp3 chapters

I keep finding new features that you have.

I mean, you have mobile apps as well, right?

For Auphonic, for Android and iOS.

And so what's the, how recently was that?

How did that come about, right?

Because that's another thing to build

and maintain on top of everything else.

because a friend of the first developer at Auphonic

wanted to do a project with us

and he's a web and mobile developer

and yeah now he works at facebook so he's quite quite good in these things so he he said yes he

wants to build a ios app for us and the problem in ios was was long that you cannot upload audio

files in in a web site i think they changed it now but for a long time you could not select

audio files and upload them to a phonic for example so the so the idea was uh to build just

simple recorder and then use our API in the mobile app so that you can basically export files from

your phone to to aphonic and yeah this this was the start basically and then we also did an android

version and on android the situation was even more complicated because there is no usable audio

editor on android and then we thought well we could also build a little audio editor right

the great challenge on android right is it works on your samsung that you've got in the office but

not on you know yeah device out there in the street yeah but actually there are not so many

from i mean rest framework for well it wasn't quite around in 2010 but like 2012 2013 i think

the audio part or how much time do you spend on just scaling up the the web part because that's

sort of the front door for everything um i'm just curious of you know where you where you spend your

time now given that you're a scale and you're still interested we can talk about i know you

we work more on the web part or other parts and sometimes more on the algorithms but yeah i mean

scaling scaling was not so much of an issue i would say so well basically we always build some

we always have to fix some things if if you see uh it will get very hot but but yeah actually i

mean you know if the traffic goes up we just rent some more servers so basically that's not so

complicated right because in our case i mean the website itself does not need much scaling because

we don't have hundred thousands of users every minute of course yeah so i'm imagining you could

and give you lots of spare capacity.

And then the need to scale is the back-end processing units.

So the database or all the web front-end and things,

they are just on one big root server.

And then we have various other servers for the audio processing

and for the long-running tasks.

and then it goes away?

Is that correct now?

Yes, 21 days.

I assume that came about because you looked around

and said, oh my God, we have all this data.

Or did you have that from the beginning,

sort of a limit on how long you would host processed audio?

because otherwise you would need a lot of storage.

Yeah.

And yet, there are, of course, also data protection reasons.

So it really is all hardware in the data center.

Yes, exactly.

Fantastic.

So if you could just wave a magic wand and add a new feature to Ophonic, what would it be?

Oh.

Because I assume you have, I mean, obviously, I don't have a sense of what is truly challenging or what you think your customers are demanding.

I mean, but yeah, I'm curious where you see that need or, you know, if you had all this time, you would spend it.

Our future feature list is very long.

Or maybe there's something that you don't even know how to tackle it, but it's sort of an unresolved problem in processing audio.

Because, again, Carlton and I don't know that space at all.

What would be really cool is if you have, let's say,

you have very bad audio like Ampli3 with 32 kilobits,

so everything is compressed already completely

and you can't hear anything about it anymore.

And then one could build an algorithm

which makes the audio as the original again.

right okay this would be great yeah that really is a magic wand but yeah at some point you you

will lose information and then it's difficult to restore the information again but there are um

what the missing data is and interpret like that and they you know sometimes come up with good

badly encoded audio but yeah this is of course not possible for every situation

not using which people have asked for is you you link in with speech recognition right where

someone can link up a third-party transcription so this this service that does use so a perfect

speech recognition system yeah what and was that something that you had again from the beginning

or something that users asked for um that integration because it looks it looks really

nice because that is another step in the tool chain of producing a podcast where do the leveling

and then transcription um i mean it's something we should we should add yes this this was not from

podcasting you know podcasting are not search podcasts are not search able so if you would

have a transcript then you can search within the podcast so that's of course perfect but

yeah some years ago there were no services which produced a reasonable output for a reasonable

price well said yes it was actually i think two years ago or three years ago when

the first api was the no it was not the first one but one one of the first ones

was the the google cloud speech api which which had an acceptable quality and also a reasonable

price and yeah now there are some there are various others other apis as well which can do that

So and back to the beginning. So actually, we wanted to build our own speech recognition system. But then we thought we cannot do everything. Because it's really very time consuming, especially if you want to build that for multiple languages. So you have to have specialists for every language and a lot of data for every language.

That's why we decided to integrate various third-party services for speech recognition.

So we still do our own pre-processing and slice the audio into small parts,

so cut out the music parts and take out only the speakers,

and then send these slices to the external services, to the speech recognition services,

and then combine it again so that you have the time codes in the transcript

and so that you know at which time who is speaking what.

Why aren't we using this?

This sounds amazing.

You're welcome to hop on that.

Yeah, I think it's so brilliant

how you've stayed kind of within your lane

yet still expanding features

because, I mean, podcasts in general

are going through quite a bit of consolidation right now.

But it seems like what you're doing is so much better

and so hard to do well, that it's not really as maybe appealing for Spotify or someone

to say, oh, we'll just do Ophonic. Or perhaps maybe, or is that an existential threat? Do

you think down the line that one of these places will say, we'll just scoop that part

in because again unless you really know what you're doing i don't it seems very difficult to

have a high quality podcast without using a tool like ophonic well that's not true if you're an

there is just a lot of time already put into it how to how to really optimize it for this use case

i mean of course someone else can build similar things but it's it's of course a lot of work and

you would need i think quite a bigger team than than we are if you don't build it for yourself

and sure and yeah i don't think that there is so much money in it that you will throw a

20 people team on it but i don't know right so it's a nice niche and that if you as long as you

the files processed is it you know hours what when you internally look at your metrics what are the

how do you manage growth because i think that there's a number of different ways to potentially

the length of of the audio file so in our system you you have you need so two hours of audio is

free for everyone in our system and uh if you process more audio then you can buy additional

credits and the credits are in in hours of audio so if you have one hour of credits you can process

one hour of audio file so therefore therefore of course the most important measure i would say is

can be i mean yeah very different in size was was that uh yes so that seems smart from a marketing

perspective but i as you know for someone like you who really knows the audio i mean the costs

could be wildly different for one hour of um you know one file versus another so yes the size of

with a very big bit size,

then the MP3 file is very small, of course.

But then if you process the file,

you have to decode the file again to get the raw data.

And then it doesn't matter if it was an MP3 or WAV file

because you need the same processing amount

or processing power for it.

So I'm curious, because you went to university

for audio engineering,

what did you think you were going to be doing

at this point in your career

And kind of what are your friends from university doing?

Because I'm sure they're not all doing startups.

So, well, actually, after university, I started, so I did my master's studies here in Austria.

And then I started a PhD in Germany, in Berlin, also about machine learning and audio.

It was, it is called, this field is called music information retrieval.

so where you try to extract information out of audio with machine learning techniques but yeah

then i did not finish this phd because basically i was the only one there which did these things

and then i started the job at the web company and get to know all these web things also django

so they also use django there and yeah then i thought okay i can also build something for

myself and i also get to know this podcaster which always had this problem with audio processing and

encoding and i just tried to build these these things myself yeah no i mean it sounds like the

you've done you you're into the audio stuff you've done the web programming it's like you've got all

the parts together and you've just created this awesome business out of it that's exactly i still

place and then later said oh i can combine these these two no it was not about audio it was

to big web projects so i just wanted to get into this topic and yeah for that it was quite well

learn web uh in school uh there's sure some some efforts on it but basically it has to be hands-on

and even something like django is largely not taught at all and if it is taught it's by an

adjunct and it's sort of an elective so that's hopefully that will change but it's it's hard to

get a backbone in web technologies even though the majority of uh undergraduate computer science

graduates probably go work on something web related so there's definitely a uh yeah educational

of course and you just have to do it and and learn it by yourself yes spoken like an engineer

part and et cetera.

But we also have a desktop application with our algorithms.

So they don't include all the features of the web service.

But the advantage here is, of course, that you don't have to do the processing in the

cloud and you don't have to upload files.

So the processing is just done on your computer, which needs, of course, processing, but you don't have to upload, download.

And, yeah, we also have a different pricing model for this desktop app.

So it's just a one time purchase.

There is in the web service, you have to pay credits.

So based on how much you process.

But, yeah, what I wanted to tell is that we are working on a new version of the desktop software at the moment.

because it was very difficult to get all these

Python and machine learning tools to desktop computers.

And especially difficult was the GUI part

because we used in the desktop apps,

we used this WX widgets framework with their Python bindings.

And it was so frustrating and so difficult.

And now we did rewrite everything with web-based tools.

So we use now Electron and...

Okay, so it's Electron.

...combined with Python.

But yeah, Electron is still a very big framework

and not so nice to handle,

but at least now everything is HTML-based,

the whole user interface,

and the backend is all done in Python.

And so as a long-term goal,

we could maybe use the same interface

for the web and for the desktop version

and just reuse all the components

are various ways how you can do that first we are compiling everything to c with titan and and all

these tools and we are using this tool is called by installer which which creates such a bundle

with Python and all the dependencies you have.

And then this is bundled into one binary

and you can afterwards distribute this binary.

And then because that's Electron-based,

that will be on Windows and Mac and Linux as well?

Okay.

And then the Electron app just also takes this Python binary

and includes it in their framework.

it's good for building application it doesn't have this story about desktop integration or

the missing part is the gui part yeah yeah awesome well well thank you so much for coming on and

is that the only reason I thought that maybe use Django is I saw your, the signup page and

you're using slash accounts and username, email, password. And I thought, Oh, that looks,

that looks like Django. And I just sent you an email. It's like, Oh yeah, it is Django.

So there's all these, there's all these, you know, a big thing for us in this podcast is try to

highlight all the different ways Django is being used in all these different realms, which are

largely kind of hidden because it's you know for you it's a sort of a secondary thing but still a

key part of your you know pipeline of your process yeah and and what i find quite uh important is

start today i would say that is the perfect end this is of course very important if you are a

small company like we are so you cannot rebuild everything from scratch after five years or yeah

You know, the next release isn't going to totally change the API or, you know.

Exactly.

All the such things.

And those stability guarantees and the deprecation policy are important.

Well, actually, maybe one last one.

If there's something you could change about Django, what would that be?

Or do you have thoughts on maybe how the future async wave, would that impact Ophonic at all?

Or is that not so much?

processing is done in a different stage so it's done offline basically in the queue so here we

don't would not need this async thing but yeah i think for for user interface improvements it's

of course very very very interesting but i have to to play with around with it a little bit more i

system so two hours per month for free and if you have any questions or feedback we are always very

very happy to to get feedback and also error reports and whatever great okay well thank you