mp3 test

There is already a thread about mp3's here but this is something I read yesterday.

I came to think of it when I read one of voodoo's remarks (the fact that you can hear the difference between mp3 and wav) in the 'audio loudness' thread.


Yesterday I read an article about this in a normal, mainstream pc magazine. Not a specialized audio magazine.

Anyway, the title was: 'mp3 as good as cd'. Of course I had to read that one if they made such a statement. Well, they tested 5 sound experts. They had to listen to music and tell if it was an mp3 or a wav version. They also coded three different songs (three different styles, pop, techno and classical music). To make the test a bit more accurate.

First, they started with 128kbit/s, then 192, 256 and finally 320kbit/s.
The test results were like this: everybody was totally sure that they heard the difference with the mp3 coded at 128kbit/s.

Then the next ones:

192kbit/s, 98% sure
256          60%sure
320          0%sure


They heard a difference in how 'deep' the soundscape was, the basstransients being more spread out, some clarity, etc.

At 320 kbit/s, they heard a difference but they couldn't tell which one was the mp3.

I just think this test is a little bit ridiculous. Especially the title... mp3 is NOT as good. You throw away information while coding so it IS worse of course.

It's also ridiculous since they coded at such high bitrates...It's obvious that it will start to sound better then! But then you start to lose the whole purpose of coding since the files start to get much bigger. Like the one at 320 kbit/s which was only about 4 times as small as the original wav file.

These mainstream magazines seem to want to 'justify' mp3 for the masses. Saying that they are right when claiming that it is as good, while the sound experts are wrong when saying the opposite.

FWIW... one night in the studio, myself and a band I was working with started talking about MP3's.  I extracted several songs off of CD's to my harddrive, made copies and converted them to MP3 (at various rates).  We sat and listened on 3 different sets of monitors and EVERY SINGLE PERSON (the drummer didn't last long, due to a damaged tempanic) could hear the difference.  We did the blind test, volume up/volume down, etc... and we could all hear it!  Up to that point, I wasn't particularly against the format, but proof is in the pudding.  By design... the format literally dumps valuable audio data out of the file.  That's proven science!  I do agree that MP3 has it's use.  You can use it to make a file small enough to email... but, don't confuse convienence with quality!  Convienent maybe... quality, no and if a person wants to consider himself/herself a audiophile, they owe it to themselves and others to understand the difference!  

Were the tests in the magazine comparative (A/B)?  In other words, were they listening to the wave version and then the mp3 version and saying which was which?  That sort of testing is fine up to a point, but a more real-life scenario is to play them say 20 different tracks, some from wave and some from mp3 - might be one mp3 and 19 wave, or whatever - and then see how confident they are.  Obviously when it came to a really low bit rate the mp3's would be easy to spot, but at anything half decent, it's hard.  I remember the thread on the Synt forum where nobody claimed they could tell if a decently prepared mp3 was that or a wave.

Without a basis for comparison you can get away with quite a lot - at risk of repeating myself, I recorded a big band CD using an eight-track minidisc machine, and the only review I saw was quite complimentary about the sound.  I've used little bits of minidisc backup material on internationally-released classical CDs too.  Nobody complained.  What offends my ear much more quickly is poor mic technique, or poor mics!  In other words, if you use superb equipment into mp3 or MD format, you'll get a better result than using cheap equipment into 24/96 format, IMHO.

I do have plenty of issues with any lossy compression scheme when it comes to multiple encode/decode cycles and archiving etc, but as a convenience delivery format, mp3 and the like has a place.

But we've been here before.....

- Ozpeter

Unless the test is blind, individual, and controlled, the results are questionable.

On something as blatantly different as a 128k MP3 and a WAV, the exercise might be valid just so everyone is exposed to how different the supposedly "CD quality" MP3 is.  But if people were immediately discriminating well encoded 256k & 320k files from the equivalent WAV, I suspect there was something else going on.  They're not THAT different that everyone can casually tell on several monitoring setups.  These take some serious attention from me.  I dunno; maybe my hearing is going...

Also, on the magazine test, people wrote how "sure" they were, but were they RIGHT?  If  these results weren't tabulated, the test was useless.  We think we hear things that aren't there all the time.  I sold many pairs of speakers using this concept.

I'd love to see some good double blind tests on these things.

Without a basis for comparison you can get away with quite a lot

exactly.  if you have a cd with nothing but mp3s, all encoded at the same bitrate, does it make sense to compare the quality with a wav?  and the majority of people who actually use the mp3 format AREN'T audiophiles, so they probably couldn't hear, or wouldn't care about the difference in quality.

i must admit...i do carry around a cd filled with mp3s myself.  i listen to it at work mostly.  it enables me to be able to listen to hours of music, without having to lug around millions of cds.  the mp3 format has it's uses.  i think that the biggest problem with the format itself, is people comparing it to the .wav format.

just my $.02

When they were sure, they also got it right. I forgot to mention that.

And they went from 128kbit/s up to 320kbit/s.

Oz, the tests were indeed comparative.

And they got to listen to these songs in an acoustically treated, expensive studio with excellent monitors of course. So that makes it also a bit easier to hear differences between the wav and the mp3. Someone who's sitting on the train listening to a cd with mp3's on won't really hear those subtles differences I guess

I think of it this way.  CD or higher standards are great for sound quality but if you want to carry around a lot of tracks in a small space, they're not much good (at present).  Mp3 is great where space is at a premium but no use as a recording or archiving medium.  I doubt whether anyone here would disagree.  So comparing the two is like running a motoring test to see how much shopping you can get into the back of Forumula 1 race car compared to a Ford Focus, and how quickly Ford Focus gets around Monza compared with the F1 car.  There's very little point!

[TROGDOR!!!]

I think of it this way.  CD or higher standards are great for sound quality but if you want to carry around a lot of tracks in a small space, they're not much good (at present).  Mp3 is great where space is at a premium but no use as a recording or archiving medium.  I doubt whether anyone here would disagree.  So comparing the two is like running a motoring test to see how much shopping you can get into the back of Forumula 1 race car compared to a Ford Focus, and how quickly Ford Focus gets around Monza compared with the F1 car.  There's very little point!

Well spoken!  FWIW, I have listened to my fair share of mp3s, and I still keep a few CDs of my favorites for listening enjoyment.  However, the more I develop my ear, the more the format grates on me...

...for that matter, so do my speakers, CD players, headphones, radio recievers...

Josh

Well.... we all know my thoughts, so you can just imagine when I opened my "Father's Day" gift from my 11 year old and found a sporty little MP3 player (complete with software and a USB cable) for storing music to listen to.  Of course, I acted totally estatic.... it's still in a bag, hiding behind my recliner!  

Well.... we all know my thoughts, so you can just imagine when I opened my "Father's Day" gift from my 11 year old and found a sporty little MP3 player (complete with software and a USB cable) for storing music to listen to.  Of course, I acted totally estatic.... it's still in a bag, hiding behind my recliner!  

now that's funny!  you should make him/her read the forums.  they'd know better next time.  

I think you are missing out on an opportunity here, VooDoo -

Mrs VooDoo - "Hey, VooDoo, get over here and change the baby's diaper!"

No response.

Mrs VooDoo - "HEY VOODOO, GET OVER HERE AND.......

Mr VooDoo - "Sorry my dear, couldn't hear you, I had the mp3 player earphones on and the volume waaaayyyyy up...."

Hey Oz..... I think you might be onto something there!  (now let us see.... where did I put that little MP3 thing...)!!    

Hehe, did you get an mp3 player??    A bit funny considering what you think about mp3, voodoo.

Anyway, I said this on the syntrillium forum too: it's simple...
It's just quality against filesize. If you want quality, it's big. If you want something small, you get less quality.
It's just a matter of what you prefer or need at the moment.

When we first started using MP3's to transfer radio commercials between stations, I took a morning and set up a test.  I took a 30" bit of music chosen at random, and copied and converted it to the whole series of MP3 bit rates.  Then A/B'd the source and converted file to count the number of artifacts I could find.

My conclusions were pretty much the same as the original findings above.  Anyone could hear the 128kbs difference, and the higher the bit rate, the harder it became to find processing artifacts.  It took me about 5 minutes of critical listening skills (learned from the Communications Research Council ... part of the NRC here in Canada) to even find one artifact at 320kbs.  Note this was LISTENING and not watching spectral analysis or a phase scope.

Are MP3's perfect?  No.  Do they affect the sound?  Depending on the bit rate yes.  Does the average listener care?   Not in the least.  They're more concerned on if they like the song or not.