reliability and validity: What's the difference

45 posts / 0 new
Last post
janfromthebruce
reliability and validity: What's the difference

 

janfromthebruce

Earlier this am I was working on posting a response in the thread concerning vapidity.
Got called away and when I went to post, the thread was closed.

The debate had turned into the reliability and validity of measurement and so this was the post I wanted to put in for what it is worth.

Let's look at reliability and later the validity of the "measures" used to ascertain both in proving which is more reliable and valid measure of vapid.

quote:

If you google "she is vapid" and "he is vapid"

AND

quote:

'he is vapid' produces 213,000 more hits than for 'she is vapid'.

Really folks this appears to be the argument.

[url=http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi2.htm]Reliab... and Validity: what's the Difference[/url]

quote:

Reliability is the consistency of your measurement, or the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects,

and is an estimate. There are two ways that reliability is usually estimated: test/retest and internal consistency.

Coming from a stats point of view, when using and not using " marks around words "he is vapid" or "she is vapid" in comparison to using no quotation marks actually shows "a lack of internal consistency." In other words, the instrument of measure "the use of quotation marks" is NOT reliable measure of these concepts, as removing the quote marks shows a complete different "estimate."

Concerning validity, as suggested in the reference page, "involves the degree to which one is measuring what one is supposed to, more simply, the accuracy of the measurement. I believe the the combined words have validity in the assertains stated here: [i]he is vapid and she is vapid[/i], but the internal consistency is lacking because inserting quotes or no quote marks should show internal consistency and don't.

Unionist you were trying to assert that only this reliability measure was valid, but [b]Test/Retest[/b], [i]is the more conservative method to estimate reliability[/i]. Essentially what Remind showed was that your "measurement" lacked "internal consistency." She used a different grouping (no quote marks) to measure the same concepts of he is vapid and she is vapid. The difference in numbers of responses in the Google search for both genders did not generate a high correlation and thus lacks reliability.

I hope that made sense, and just wanted to move from a personal debate.

oldgoat

It's been almost 30 years since I last took stats and research methodology courses. Interesting how little of it I remember, but I think I did retain that concept.

Michelle

Since this seems to be more about research methods than a discussion about rabble content, should we move this to humanities and science?

oldgoat

Ok, but if she starts talking about chi-square distributions I'm outta here.

Caissa

Michelle wrote:Since this seems to be more about research methods than a discussion about rabble content, should we move this to humanities and science?

That would be valid. [img]wink.gif" border="0[/img]

M. Spector M. Spector's picture

The difference between Googling "she is vapid" (with quotation marks) and she is vapid (without quotation marks) as a means of determining how often female entities are called vapid on the interweb is about the same as the difference between using an anemometer and a measuring cup to measure wind speed. One gives results that have a certain degree of both reliability and validity, while the other gives results that have neither validity nor reliability in any degree at all.

The alleged difference between reliability and validity therefore is [i]nihil ad rem[/i].

oldgoat

quote:


Originally posted by Caissa:
[b]
That would be valid. [img]wink.gif" border="0[/img] [/b]

She's reliable.

RosaL

quote:


Originally posted by M. Spector:
[b]The difference between Googling "she is vapid" (with quotation marks) and she is vapid (without quotation marks) as a means of determining how often female entities are called vapid on the interweb is about the same as the difference between using an anemometer and a measuring cup to measure wind speed. One gives results that have a certain degree of both reliability and validity, while the other gives results that have neither validity nor reliability in any degree at all.

The alleged difference between reliability and validity therefore is [i]nihil ad rem[/i].[/b]


I have to agree with that. I also think Sarah Palin is vapid.

janfromthebruce

quote:


Originally posted by oldgoat:
[b]Ok, but if she starts talking about chi-square distributions I'm outta here.[/b]

Ok, so now onto [url=http://stattrek.com/Lesson3/ChiSquare.aspx]chi-square distributions. [/url]

For those who are just oldgoats and really challenged, the internet provides a free [url=http://www.stat.tamu.edu/~west/applets/chisqdemo.html]Chi Square Calculator[/url] for all your computing enjoyment.

Once you get yours, we can all begin.... cause I am so vapid, she said [img]biggrin.gif" border="0[/img] [img]biggrin.gif" border="0[/img] [img]biggrin.gif" border="0[/img]

oldgoat

Golly. And at first you seemed so nice.

M. Spector M. Spector's picture

quote:


Originally posted by RosaL:
[b]I have to agree with that. I also think Sarah Palin is vapid.[/b]

I don't agree with that. Some of her statements are vapid, but I don't think that adjective applies to her as a person. It means dull, or lacking in liveliness.

P.S. That's another reason why Googling "she is vapid" is beside the point. The original reference to which that thread pointed was a reference to Alice Klein's "rhetorical vapidity", which is very different from saying Klein herself is "vapid".

That's why [url=http://www.rabble.ca/babble/ultimatebb.php?ubb=get_topic&f=15&t=001307#0... did a Google search on "rhetorical vapidity"[/url] (with the quotes, natch) and showed that, on the internet at least, that phrase was not an anti-female trope.

[ 27 October 2008: Message edited by: M. Spector ]

RosaL

quote:


Originally posted by M. Spector:
[b]I don't agree with that. Some of her statements are vapid, but I don't think that adjective applies to her as a person. It means dull, or lacking in liveliness.[/b]

All my life I'v been using it as a synonym for "superficial, empty". Incorrectly, however, as it now appears. [img]confused.gif" border="0[/img]

M. Spector M. Spector's picture

Heh, heh.

You might also want to check out the difference between "immanent" and "imminent". [img]wink.gif" border="0[/img]

janfromthebruce

quote:


Originally posted by oldgoat:
[b]Golly. And at first you seemed so nice.[/b]

I still am oldgoat and so are you. [img]cool.gif" border="0[/img]

janfromthebruce

quote:


Originally posted by M. Spector:
[b]I don't agree with that. Some of her statements are vapid, but I don't think that adjective applies to her as a person. It means dull, or lacking in liveliness.

P.S. That's another reason why Googling "she is vapid" is beside the point. The original reference to which that thread pointed was a reference to Alice Klein's "rhetorical vapidity", which is very different from saying Klein herself is "vapid".

That's why [url=http://www.rabble.ca/babble/ultimatebb.php?ubb=get_topic&f=15&t=001307#0... did a Google search on "rhetorical vapidity"[/url] (with the quotes, natch) and showed that, on the internet at least, that phrase was not an anti-female trope.

[ 27 October 2008: Message edited by: M. Spector ][/b]


And I googled [i]rhetorical vapidity[/i]without quotes to see if it had [i]internal consistency[/i], and guess what it did!

RosaL

quote:


Originally posted by M. Spector:
[b]Heh, heh.

You might also want to check out the difference between "immanent" and "imminent". [img]wink.gif" border="0[/img] [/b]


Well, I [i]do[/i] know [i]that[/i] - it was just a spelling error. Really [img]wink.gif" border="0[/img]

oldgoat

quote:


Originally posted by janfromthebruce:
[b]

I still am oldgoat and so are you. [img]cool.gif" border="0[/img] [/b]


Well yes, very true. Just a brief note of explanation and I'll no longer bother the mathematically competent. In grade school, I was lousy at math, and in high school it became a real full blown phobia. Combined with other things going on in my life, it caused me to never actually pass grade 10. When I finally made it to university years later as a mature student I found I needed to take stats if I wanted to get an MSW, but I might as well have tried to learn to fly. Oddly enough I managed research methodology ok, which uses math differently.

I ended up doing the same sort of stuff anyway, but through a more roundabout route and with a history degree.

So carry on with your serious discussion.
[img]smile.gif" border="0[/img]

[ 27 October 2008: Message edited by: oldgoat ]

M. Spector M. Spector's picture

quote:


Originally posted by janfromthebruce:
[b]And I googled [i]rhetorical vapidity[/i]without quotes to see if it had [i]internal consistency[/i], and guess what it did![/b]

My guess would be that it returned a whole bunch of web pages that had nothing whatsoever to do with the phrase "rhetorical vapidity" and therefore produced no relevant statistics at all.

And I just did that search to confirm that my guess was spot on.

Unionist

quote:


Originally posted by janfromthebruce:
[b]Unionist you were trying to assert that only this reliability measure was valid, but [b]Test/Retest[/b], [i]is the more conservative method to estimate reliability[/i]. Essentially what Remind showed was that your "measurement" lacked "internal consistency." She used a different grouping (no quote marks) to measure the same concepts of he is vapid and she is vapid. The difference in numbers of responses in the Google search for both genders did not generate a high correlation and thus lacks reliability.

I hope that made sense, and just wanted to move from a personal debate.[/b]


Hi jftb, just noticed your post for the first time.

You're mistaken. Remind's search was meaningless and measured nothing. Why is this not obvious??

kropotkin1951 kropotkin1951's picture

I think that the BNL hypothesis that a K car was reliable has been tested and retested. [img]wink.gif" border="0[/img]

janfromthebruce

quote:


Originally posted by oldgoat:
[b]

Well yes, very true. Just a brief note of explanation and I'll no longer bother the mathematically competent. In grade school, I was lousy at math, and in high school it became a real full blown phobia. Combined with other things going on in my life, it caused me to never actually pass grade 10. When I finally made it to university years later as a mature student I found I needed to take stats if I wanted to get an MSW, but I might as well have tried to learn to fly. Oddly enough I managed research methodology ok, which uses math differently.

I ended up doing the same sort of stuff anyway, but through a more roundabout route and with a history degree.

So carry on with your serious discussion.
[img]smile.gif" border="0[/img]

[ 27 October 2008: Message edited by: oldgoat ][/b]


Hey, oldgoat I know about that request to get into an MSW program. Classical stats is well classical, and that math stuff is trying. In my own MSW I completely avoided quantitative and used qualitative design and avoided that stuff. Good that things changed.

Unionist

I've always maintained, if you can't spend quality time with your family, settle for quantity time.

Michelle

oldgoat, same here re: math. I dropped math after grade 10 math (which I barely squeaked through).

Then I took a first year economics course in university which required some calculus. I got 92%.

I think my math phobia was just that - a phobia. I'd probably be good at it if I did it now, but I've never gotten around to trying.

[ 27 October 2008: Message edited by: Michelle ]

Boom Boom Boom Boom's picture

I have quite a serious hearing loss, not quite made up for with a hearing aid, so listening to theory in math class in high school was a struggle, and I didn't do all that well. Then, I got accepted into college as a mature student a few years after finishing the four year tech program in high school, and things improved dramatically - smaller classes and more of a serious attitude from fellow students. Along with the required courses in Communication Arts, I did Micro and Macro Economics, and that was an eye opener. After I finished college, I worked as a program officer for the Local Initiatives Program in Ottawa (early 1970s) for a few years, saved big bux, and went to university to do a bachelor's in Experimental Psychology, which was mostly advanced statistical analysis. I did quite well, so having had difficulty with math in high school weasn't a harbinger of anything at all. It wasn't until I got out of high school that I was able to really get anywhere. Later on I did a Master's in a completely unrelated field. I have two older brothers who had careers in management and accounting, and I think I could have done the same.

Fidel

quote:


Originally posted by Michelle:
[b]

Then I took a first year economics course in university which required some calculus. I got 92%. [/b]


Holy crap! That's a pretty high mark for an upper level math course like calculus. That's not easy stuff.

Michelle

No no. I mean, I got 92% in my economics course. I've never taken calculus in university or high school. But once I was doing a bit of it practically, in my economics course, it came easily to me.

In fact, I found that economics course so interesting that I wanted to major in it. But at Queen's, where I was going at the time, you have to take first year calculus in order to complete the Economics degree - it was a required course. Terribly disappointing to me, but I knew there was no way I could pass a calculus course when I had absolutely no math background to speak of - I can barely do basic algebra.

So I majored in Philosophy instead. [img]biggrin.gif" border="0[/img]

[ 27 October 2008: Message edited by: Michelle ]

Fidel

quote:


Originally posted by Boom Boom:
[b] and went to university to do a bachelor's in Experimental Psychology, which was mostly advanced statistical analysis. I did quite well,[/b]

Holy frijole! I knew babblers weren't slow, but I'm beginning to realize I'm keepin some pretty fast company here... [img]smile.gif" border="0[/img]

Fidel

quote:


Originally posted by Michelle:
[b] I can barely do basic algebra.

So I majored in Philosophy instead. [img]biggrin.gif" border="0[/img]

[ 27 October 2008: Message edited by: Michelle ][/b]


I think if you can master what few rules of highschool algebra there are, that's a big part of calculus nailed down right there. I flunked grade eleven general level math when I was young, and so that, im my mind, reinforced the idea that upper level math was impossible for me. Like you and Boom Boom, I was wrong about that, too.

Boom Boom Boom Boom's picture

I gravitated to experimental psych partly because I could control my surroundings - I'm very hard of hearing - and doing experiments and then statistical analysis of these was an environment that allowed me to work at my own speed. [img]smile.gif" border="0[/img]

Boom Boom Boom Boom's picture

I feel a 'Nature versus Nurture' debate coming on... [img]biggrin.gif" border="0[/img]

M. Spector M. Spector's picture

quote:


Originally posted by Michelle:
[b]Since this seems to be more about research methods than a discussion about rabble content, should we move this to humanities and science?[/b]

How about babble banter?

Tommy_Paine

Well, move it if you must but not while I compose, please. If you do, I will become uncomposed. Which is better than decomposed, I suppose.

I've done Reliability and Repeatability studies on measuring equipment at work. We used precision instruments made to measure steel parts. And I found numerous ways to manipuate results.

And don't get me started on Statistical Process Control.

While it's valuable to attempt to quantify something in order to problem solve, it's tricky enough when we are dealing with solid "things". I think it goes irrevocably awry when we try to quantify an abstract concept.

But, we will continue to do this, if only to keep the social "sciences" alive and well.

[img]biggrin.gif" border="0[/img]

[ 28 October 2008: Message edited by: Tommy_Paine ]

janfromthebruce

So now that we are all doing true confessions, I have another one. I took my 3rd year univers stats course by distant ed, and the prof told me that as long as the 8 assignments were completed before the final exam in April, he didn't care about the interm dates.
I spent the month of March doing stats 7/12, and wrote the only final exam with absolutely no idea how I did. I needed a 65% or above to get into social work - yep, I barely made it.

So when I did my Masters in Social Work I knew that if anybody hired me as a researcher that in fact I really knew nothing. So I took a Masters quantitative and qualitative MS level course at Laurie which I didn't have to. My friends thought I was nuts and said I would bring my marks down.
This was personal though cause I really did need this info in the field of social policy.
I worked hard and guess what I ended up with A+ as my final grade point average. It is the mark that I am the proudest of all though university, and the fact that I actually gained knowledge that I was completely deficit in.
True confessions!

Caissa

I tried to avoid taking grade 12 math but the high school principal said i wouldn't be one of the 8 students in my grad class of 350 who wasn't taking it. Never really liked math.

Twelve years later (93-94) when I did my B.Ed. there was a mandatory course in educational stats. Spent the night before the exam teaching an older buddy (he was 40, I was 30)the course material while we drank scotch. We both got through the exam the next day. So much for state dependent learning.

Our oldest son is not enjoying math in grade 6. C on his first test while having scored 119 out of 120 on spelling so far this year. My challenge at the moment is to prepare him for a test on geometry tomorrow. He's been developing some good mnemonics but we'll see how his coverage of the material is.

On the upside, he's learning how to calculate odds for Hold 'em hands. [img]smile.gif" border="0[/img] [img]wink.gif" border="0[/img]

Michelle

My son loves math, gets perfect or near perfect on every test. He's not a bad speller, but finds writing composition challenging.

I'm pretty sure I wasn't handed someone else's baby at the hospital... [img]wink.gif" border="0[/img]

Boom Boom Boom Boom's picture

Speaking of spelling, Caissa, my worse school experience ever probably was a spelling competition in grade five - that would have been in 1960. The teacher divided the class into rows of two, and alternated giving words to spell. When it came my turn, because I'm very hard of hearing, of course I didn't hear the word I was asked to spell. So, using my ingrained 'coping mechanism', I asked the teacher to spell the word for me, because I couldn't hear it... forgetting I was in a spelling competition, and of course the whole class burst out laughing, and I was throughly humiliated. [img]frown.gif" border="0[/img] [img]frown.gif" border="0[/img] [img]frown.gif" border="0[/img] [img]frown.gif" border="0[/img] [img]frown.gif" border="0[/img]

Unionist

[img]biggrin.gif" border="0[/img] [img]biggrin.gif" border="0[/img] [img]biggrin.gif" border="0[/img]

Which parent hasn't had that thought?

Tommy_Paine

I get compliments about my daughters, and when I do I say, yeah, well, Darwin took a break....

So, I'm dissapointed no one took the bait about my dig at social sciences. I once considered myself babbles slyest master baiter, but I will have to reconsider.

Anywho, I googled "Cervantes literature" and "Daniel Steel literature".

Apparently, there are 800,000 more people who consider Daniel Steel a bigger part of literature than Cervantes. (1,450,000 Cervantes hits to 2,250,000 Daniel Steel hits)

Happy reading.

janfromthebruce

quote:


Originally posted by Tommy_Paine:
[b]I get compliments about my daughters, and when I do I say, yeah, well, Darwin took a break....

So, I'm dissapointed no one took the bait about my dig at social sciences. I once considered myself babbles slyest master baiter, but I will have to reconsider.

Anywho, I googled "Cervantes literature" and "Daniel Steel literature".

Apparently, there are 800,000 more people who consider Daniel Steel a bigger part of literature than Cervantes. (1,450,000 Cervantes hits to 2,250,000 Daniel Steel hits)

Happy reading.[/b]


book baiting is not allowed. [img]wink.gif" border="0[/img]

Tommy_Paine

Janfromthebruce/smartass, seven hits.

Janfromthebruce/locquacious interesting vibrant, zero hits.

Don't you love imperical data?

[img]biggrin.gif" border="0[/img]

M. Spector M. Spector's picture

quote:


Originally posted by Tommy_Paine:
[b]Apparently, there are 800,000 more people who consider Daniel Steel a bigger part of literature than Cervantes. (1,450,000 Cervantes hits to 2,250,000 Daniel Steel hits).[/b]

Daniel Steel? who'da thunk?

And I've never even heard of him!

Maysie Maysie's picture

He's the lesser-known brother of course.

Me, I love Jack Collins. Now, there's some steamy manly novels.

Tommy_Paine

See, I'm [i]proud[/i] I didn't know how to spell Steel's first name.

Big fan, Spector?

[img]wink.gif" border="0[/img]

janfromthebruce

quote:


Originally posted by Tommy_Paine:
[b]Janfromthebruce/smartass, seven hits.

Janfromthebruce/locquacious interesting vibrant, zero hits.

Don't you love imperical data?

[img]biggrin.gif" border="0[/img] [/b]


that is war (does anybody know what button I need to push on my keyboard to get it to quit typing in French. eg. Ииииии