Ping Caelin Gabriel RE: Design of Experiments for audio applications

[Puma Cat]

I recently mentioned in a thread a few weeks back in the acoustical treatments forum about using a statistical approach known as Design of Experiments back in 2011 to optimize the integration of my REL Sub into my 2-channel system with Dyanudio Contour S3.4s.

Caelin Gabriel mentioned in that thread that he'd be interested in seeing this approch, so I'm reposting it for Caelin and any others to read that may be interested. Heads up: it's pretty statistical....

The other principal reason I'm posting it is that I think this approach could be very useful to Caelin in designing Shunyata power cords using his DTCD measurement system (in fact I did a "mock DOE" some years ago for designing power cords using DTCD and DOE, just to show how it might work.)

To start with here is an introduction to Design of Experiments.

Description
Design of experiments (DOE) is a powerful tool that can be used in a variety of experimental situations. DOE is particularly powerful because it allows for the examinination of multiple input factors to be characterized with resepect to their effect on a desired output (called the "response"). By evaluating multiple factors at the same time, DOE can identify important interactions that may be missed when experimenting with one factor at a time. All possible combinations can be investigated (full factorial) or only a portion of the possible combinations (fractional factorial). Fractional factorials will not be discussed here.

When to Use DOE
Use DOE when more than one input factor is suspected of influencing an output. For example, it may be desirable to understand the effect of temperature and pressure on the strength of a glue bond.

DOE can also be used to confirm suspected input/output relationships and to develop a predictive equation suitable, called a transfer function, for performing what-if analysis.

DOE Procedure
Acquire a full understanding of the inputs and outputs being investigated. A process flow diagram or process map can be helpful. Utilize subject matter experts as necessary.

Determine the appropriate measure for the output. A variable (continuous datas) measure is preferable. Attribute measures (pass/fail) should be avoided. Ensure the measurement system is stable and repeatable.

Create a design matrix for the factors being investigated. The design matrix will show all possible combinations of high and low levels for each input factor. These high and low levels can be generically coded as +1 and -1. For example, a 2 factor experiment will require 4 experimental runs:



Note: The required number of experimental runs can be calculated using the formula 2n where n is the number of factors.

For each input, determine the extreme but realistic high and low levels you wish to investigate. In some cases the extreme levels may be beyond what is currently in use. The extreme levels selected should be realistic, not absurd. For example:

Enter the factors and levels for the experiment into the design matrix. Perform each experiment and record the results. For example:



Calculate the effect of a factor by averaging the data collected at the low level and subtracting it from the average of the data collected at the high level. For example:

Effect of Temperature on strength:
(51 + 57)/2 - (21 + 42)/2 = 22.5 lbs

Effect of Pressure on strength:
(42 + 57)/2 - (21 + 51)/2 = 13.5 lbs

The interaction between two factors can be calculated in the same fashion. First, the design matrix must be amended to show the high and low levels of the interaction. The levels are calculated by multiplying the coded levels for the input factors acting in the interaction. For example:



Calculate the effect of the interaction as before.

Effect of the interaction on strength:
(21 + 57)/2 - (42 + 51)/2 = -7.5 lbs

The experimental data can be plotted in a 3D Bar Chart.






This simple-minded example above shows that there is an interaction between temperature and pressure in the strength of the glue bond. This is one feature of DOEs that is particularly useful when looking at the effect of a number of factors and their effect on the critical functional response.

Now that that is out of the way as intro, let's look at the specific experiment I had in mind in the next post.

The experiment I was trying to reproduce was the one based on the graph from research from JBL showing that listeners perceived a room as having flat room reponse when the measured frequency actually looked more like this. This is presumably because our hearing is not linear at lower frequencies. Bear in mind, you may or may not agree that this frequency response is the ideal or preferred one. Either way, it doesn't matter, I merely used it as a "jumping off" point for this Design of Experment to create a model. A model is just that: a model. All statistical models need to be confirmed in the real world to see if they hold up.



Given this, I set out to to perform a set of experiment to see if I could determine the optimal settings of speakers, REL sub settings and other factors, like grilles on or off, speaker toe-in, port plugs, etc. that would optimize the in-room response.


With that in mind, I set out to see what I could do with my set up to emulate that via DOE.

System used at the time was: Conrad-Johnson Premier 17LS Preamp, Conrad-Johnson LP70S, Dynaudio Contour S3.4 speakers, REL R-305, M-Audio MobilePre USB mike preamp, Dayton calibrated measurement microphone with calibration file, and Room EQ Wizard Software. The Room EQ Wizard software generates and simultaneously measures the frequency sweep from 10 Hz to 20,000 Hz.

The desired responses were to maximize 20 and 50 Hz repsonse in dB, and minimize the 150 and 500 Hz responses. These 150 and 500 Hz responses were nodes that I wanted to minimize, if possible.

An example trace as my starting point for reference is shown. The purple trace is where I started with the sub in, the brown trace with the sub out, before optimizing things using the DOE approach. The following is the real-time, IN-ROOM measured response for the right speaker:



The goal here was to maximize the 20-50Hz node to range, while smothing the 70 Hz, and minimizing the 105, 155 and 500 Hz node peaks, so as to emulate the JBL graph.

There were 3 factors I examined in the DOE: 1) sub gain (as clicks up from zero), 2) sub crossover, likewise clicks up from zero, and 3) plug or no plug in the speaker reflex port. There were four responses being measured as the result of 3 factors at 2 different levels (low, high as in the examples above shown). So, in DOE parlance, this was a 3-factor, 2-level full factorial DOE.

Setting up a full-factorial DOE in JMP (a stastical package), here are the experimental matrix I ran per JMP's output for the experimental design and the ACTUAL measurements for each frequency (as measured in dB by room Eq Wizard).



The next post will be the analysis of the data...

[Puma Cat]

Here is the result of analyzing the 20 Hz response of the DOE (warning: large image)




We can see from the actual by predicted plot that the R^2 and r^2 adjusted are strongly correlated at 0.97 and 0.94, respectively, indicating that our results fit our model pretty well. Also the leverage plots (the factors affecting the response (20Hz output), show that Sub gain has a large effect, and quite possibly that the interaction of sub gain and sub crossover may have an effect, as the p-value is just barely above p>-.05, which tends to suggest there may be a significant effect if we gather more data or relax our confidence level from 95% to 90%. In, fact, if we relax our alpha from 5% (the chance we are willing to accept that we are wrong) to 10%, then the Sub crossover point becomes significant. In addition, our analysis of variance with probability of F >0.0016, suggests that that this result occurred from our null hypothesis, that the model does not predict sub 20 Hz performance is very low.

The interaction profile plots also suggest that there is a likely interaction between sub XO setting and sub gain, and the plot lines are not perfectly parallel, but appear to intersect.

Looking at the Prediction Profiler from JMP, comparing Sub gain with Sub XO, we can see that the maximum 20 Hz response is obtained by the Sub gain at 12 clicks up, and sub set at zero clicks, and we could expect a level of 75.1875 dB.





JMP is also cool because it will show you a 3-D image of two factors at one time as they affect the desired response in a 3-D cube, in this case, the 20 Hz response. As you can see, the upper left corner of the rectangle shows, which is max 20 Hz response is at a sub crossover of zero, and a gain setting of 12.



The above is just the prediction profiler for one frequency, in actuality I used JMP to provide the sub gain, crossover and port plug settings for the desired response for four frequencies (it turns out the fifth, highest frequency, is not impacted by the sub settings with any statistical power).


These analyses show how you can utitlize DOE to predict with good confidence the response from setting the factors (crossover point and gain) to give the desired response at 20 Hz.

I used this approach then to measure the BEFORE and AFTER responses, in this graph, BEFORE is BLUE and AFTER is GREEN. Note how much smoother the overall frequency response is for the green trace.



I then confirmed the DOE model by extensive listening tests, including having some audio buddies come over and evaluate the low frequency response on recordings they know well.

[Shrinky Smurf]

Outstanding experiment design and analysis. Your effort required a tremendous amount of knowledge, forethought, equipment and time, not only to perform, but also to collate results and share. Glad you were able to hear practical results as well! This is not so overly complicated that it would be out of reach for someone with the interest to take the time and use DOE on various other aspects of a system. Thank you for the post, PumaCat.

Best,
Shrinky

[Puma Cat]

Originally Posted by Shrinky Smurf

Outstanding experiment design and analysis. Your effort required a tremendous amount of knowledge, forethought, equipment and time, not only to perform, but also to collate results and share. Glad you were able to hear practical results as well! This is not so overly complicated that it would be out of reach for someone with the interest to take the time and use DOE on various other aspects of a system. Thank you for the post, PumaCat.

Best,
Shrinky

Thanks, Shrinky! Nice to have input from someone who understands it.

I teach DOE to scientists, so I understand it pretty well, though there is always more to learn.

I agree it is not so complicated that it's out of reach for others to do. The biggest effort is in setting up the mike and just running the different combinations of factor settings.

The analysis in JMP just takes a few minutes. In fact, JMP has made considerable improvements to their DOE platform since 2011, and now there is no longer a reason to do fractional factorials anymore as there are more efficient and informative DOE designs one can do instead.

[PHC1]

Stephen, very interesting idea and approach. I'm having a hard time imagining how a power cord can be developed from it? When you have "measurable" results from variables it is understood. What if you can not measure any difference in any meaningful way? Cables and power cords all too often fall into exactly that category. Difference can not be measured but our ears and/or imagination tells us otherwise?

[Puma Cat]

Originally Posted by PHC1

Stephen, very interesting idea and approach. I'm having a hard time imagining how a power cord can be developed from it? When you have "measurable" results from variables it is understood. What if you can not measure any difference in any meaningful way? Cables and power cords all too often fall into exactly that category. Difference can not be measured but our ears and/or imagination tells us otherwise?

Using Shunyata Research's DTCD metrology: Dynamic Transient Current Delivery, which is plotted as Current (amps) on the Y-axis vs Time on the X-axis. This is a key attribute for power cords: to be able to deliver as much current as quickly as possible when an amplifier or other component is pulling current due to dynamic, transient swings in the music. Think about the very big orchestral dynamics in the opening measures in Albeniz' Asturias, for example. The other metrics that could be measured would be RFI and EM noise rejection, for example, as these would impact S/N (usually measured in dB); this could possibly be measured as "residual noise", which I've seen plots from in data sets from Shunyata.

A lot of "power strips/surge suppressors", e.g., a Monster Cable one I had years ago, etc. as an example: when amps were plugged into it, the amps sounded slow, dark, and muddy. It is because these "power strips" dramatically negatively impacted DTCD.

[PHC1]

Originally Posted by Puma Cat

Using Shunyata Research's DTCD metrology: Dynamic Transient Current Delivery, which is plotted as Current (amps) on the Y-axis vs Time on the X-Axis. This is a key attribute for power cords: to be able to deliver as much current as quickly as possible when an amplifier or other component is pulling current due to dynamic, transient swings in the music. Think about the very big orchestral dynamics in the opening measures in Albeniz' Asturias, for example.

I am hard to impress with this idea since I worked in radiology, specifically with MRI machines that use Superconducting Magnets. The windings which have ZERO electrical resistance and expulsion of magnetic fields occurring when cooled below critical temperature (cryogenically cooled with liquid helium). The MRI noise you hear is actually the current pulsing in those wires/windings of zero resistance. A "power cord" doesn't get better than that.

[Antonmb]

Very interesting Stephen. I can’t say I followed all the detail, but I grasp the concept of DOE, and while I never thought of it in the context of audio setup or equipment design, it certainly seems to make sense. I’m neither a scientist nor engineer, but saw the positive results our R&D team realized from the use of DOE in the design of high-performance synthetic ropes.

[Puma Cat]

Originally Posted by Antonmb

Very interesting Stephen. I can’t say I followed all the detail, but I grasp the concept of DOE, and while I never thought of it in the context of audio setup or equipment design, it certainly seems to make sense. I’m neither a scientist nor engineer, but saw the positive results our R&D team realized from the use of DOE in the design of high-performance synthetic ropes.

Hi Tony,
Glad hear you have some awareness/experience with DOE. It's also used other industries like plastic molding, printing, and tire development, etc.

I can totally see how it would be useful in designing high-performance synthetic ropes. For example, you could examine things like: fiber type, fiber tensile strength, no. of twists/unit length in the rope, thickness or gauge, different construction techniques, etc....I'm sure you know of others that I don't know about.

[Puma Cat]

Originally Posted by PHC1

I am hard to impress with this idea since I worked in radiology, specifically with MRI machines that use Superconducting Magnets. The windings which have ZERO electrical resistance and expulsion of magnetic fields occurring when cooled below critical temperature (cryogenically cooled with liquid helium). The MRI noise you hear is actually the current pulsing in those wires/windings of zero resistance. A "power cord" doesn't get better than that.

Absolutely correct, Serge, but power cords are used in conditions like folks' living rooms, not where they can be cooled cryogenically or where they become "superconducting". They also have design constraints, in that they cannot be so large a gauge that they are impractical for use in real-world environments.