[visionlist] signal detection query

Thu Jul 22 18:13:07 GMT 2010

As Todd says, its important that test is same condition for all items.
Hence the difference in hit rates for high and low TMS must be due to something that happened at study.
Hence I now agree with others that the difference in z hits must be due to better discrimination.
Its still the case that the actual false alarm rate is non-informative, since it drops pout of the difference in d'. However, the new items are still essential to ensure that people don't say yes all the time. Having a single block with all 3 kinds of trials, seems to be the most efficient design, as participant maintains same criterion throughout.
However, there are arguments of a more lengthy split design, as the criterion is known to be NOT fixed, but an average over trials where participant changes criterion only after an error, if there were different valences of association according to the previous study condition then people might move their criteria differently according to the type of error. In that case the two single condition studies might have less criterion variance and hence higher d'. They would not necessarily have the same false alarm rate, as more positive associations in one condition might lead to more 'old' responses. IN that event  one would use the condition appropriate fa rates and just might get a different difference in d'.  This is possible by pretty far-fetched and unlikely. So the single block design with less trails overall should be effcitient
Best
Diana

On 22/07/2010 18:04, "Todd S. Horowitz" <toddh at search.bwh.harvard.edu> wrote:

Diana

If I understand Joseph's design correctly, the two different TMS conditions are at study, not at test. So TMS+ and TMS- are two different sets of pictures which were studied either with or without TMS, but they're being tested mixed in the same block of trials, all with no TMS at the time of testing.

IF there were two different testing conditions, then there would be two accompanying sets of new images to provide corresponding false alarm rates.
But since both the TMS+ and TMS- stimuli are being tested in the same block, I think it's reasonable to assume that the FA rates are the same.

thanks
Todd

On Jul 21, 2010, at 8:06 PM, Kornbrot, Diana wrote:

HI Joseph,
As Todd & others have clearly described there is no difficulty with problem 1.

As for problem 2, unfortunately the outside TMS fa rate CANNOT provide any information whatsoever as to whether differences in hit rate under strong and weak TMS are due to differences in d' or differences in bias.
Simple algebra shows why:
d'(strong TMS) = z(hit, strong TMS) - z(fa, no TMS)
d'(weak TMS) = z(hit, weak TMS) - z(fa, no TMS)
Hence
d'(strong TMS) - d'(weak TMS) = z(hit, strong TMS) = z(hit, weak TMS),
the contribution form the fa rate is eliminated
If z(hit, strong TMS) IS different from z(hit, weak TMS) this could be due to better discriminability, or to people in the strong TMS state being more  [or less] biased towards 'yes'. The fa rate outside TMS cannot provide a solution. It is necessary to have separate FA rates under the two TMS stimulation conditions in order to draw any conclusion about d'
Don't shoot the messenger
Best
Diana

On 21/07/2010 16:13, "Todd S. Horowitz" <toddh at search.bwh.harvard.edu <x-msg://5/toddh@search.bwh.harvard.edu> > wrote:

Daniel, Joseph

I think we're all agreed now on point (1) :)

As to point (2), I don't think Daniel's objection is a problem for Joseph's study, since the point is not to compare old stimuli+TMS to new stimuli without TMS, but to compare old TMS and old non-TMS stimuli; the new stimuli are there simply to measure the false alarm rate.

Similarly, I think this dispenses with Daniel's other objection. It's true that the Gaussian equal-variance assumptions probably do not apply, so that d' is not independent of criterion. However, since all of the stimuli are being tested in the same block of trial, criterion should be constant, so the d's will be comparable.

However, this makes me wonder why bother to compute SDT measures at all. Since the false alarm rate should be constant for both classes of stimuli, why not just compare % correct?

thanks
Todd

On Jul 21, 2010, at 4:26 AM, Daniel Oberfeld wrote:

Hi Joseph,

Re (1) : If you use the correct formula for calculating d', then it will automatically correct for unequal numbers of old and new pictures.

Re (2): I think this is no problem for calculating the SDT statistics, but rather for the interpretation of your results - does it make sense to compare responses to old stimuli+TMS and responses to new stimuli without TMS...?

There is one very serious issue with calculating d' for your data, however. In case you collected binary responses ("Is the picture old or new?"), then for calculating d' you will have to assume that the internal distributions for "signal" and "noise" have identical standard deviations (cf. Macmillan & Creelman, 2005). It is known for a long time that this assumption is frequently incorrect for experimental data (e.g., Swets, 1986). And thus d' is not a valid measure of sensitivity because it is strongly influenced by response bias (Verde, MacMillan, & Rotello, 2006).

The simple solution (at least for future experiments) is to obtain rating responses rather than binary responses - with these responses, you can caculate for example the area under the ROC curve, which is a valid index of sensitivity even if the SDs of the internal distributions are unequal (Swets, 1986).  Again, Macmillan & Creelman (2005) explain in detail how to conduct such an experiment.

Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user's guide (2. ed.). Mahwah, NJ [et al.]: Lawrence Erlbaum Associates.
Swets, J. A. (1986). Indices of discrimination or diagnostic accuracy: their ROCs and implied models. Psychological Bulletin, 99(1), 100-117.
Verde, M. F., MacMillan, N. A., & Rotello, C. M. (2006). Measures of sensitivity based on a single hit rate and false alarm rate: The accuracy, precision, and robustness of d ', A(z), and A '. Perception and Psychophysics, 68(4), 643-654.

Best,

Daniel

________________________________
Professor Diana Kornbrot
  email:  d.e.kornbrot at herts.ac.uk <x-msg://5/d.e.kornbrot@herts.ac.uk>
   web:    http://web.mac.com/kornbrot/iweb/KornbrotHome.html
Work
School of Psychology
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
    voice:     +44 (0) 170 728 4626
    mobile:   +44 (0) 796 890 2102
    fax          +44 (0) 170 728 5073
Home
19 Elmhurst Avenue
London N2 0LT, UK
   landline: +44 (0) 208 883 3657
   mobile:   +44 (0) 796 890 2102
   fax:         +44 (0) 870 706 4997
________________________________

_______________________________________________
visionlist mailing list
visionlist at visionscience.com
http://visionscience.com/mailman/listinfo/visionlist

Todd S. Horowitz, PhD
Assistant Professor of Ophthalmology
Harvard Medical School
Associate Director
Visual Attention Lab
Brigham & Women's Hospital
64 Sidney Street, Suite 170
Cambridge, MA 02139
phone:  (617) 768-8813
fax:    (617) 768-8816
http://search.bwh.harvard.edu/ <http://search.bwh.harvard.edu/>

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

________________________________
Professor Diana Kornbrot
  email:  d.e.kornbrot at herts.ac.uk
   web:    http://web.mac.com/kornbrot/iweb/KornbrotHome.html
Work
School of Psychology
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
    voice:     +44 (0) 170 728 4626
    mobile:   +44 (0) 796 890 2102
    fax          +44 (0) 170 728 5073
Home
19 Elmhurst Avenue
London N2 0LT, UK
   landline: +44 (0) 208 883 3657
   mobile:   +44 (0) 796 890 2102
   fax:         +44 (0) 870 706 4997
________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://visionscience.com/pipermail/visionlist/attachments/20100722/c54cb8f6/attachment.htm>