> y{x[ bbjbj 4hΐΐ2q""e%e%e%e%e%y%y%y%8%D%\y%NQ&'1'1'1'((((((CMEMEMEMEMEMEM$ORiM]e%((((((((((iMe%e%1'1'MJ+J+J+((e%1'e%1'CMJ+((CMJ+J+CI1'y%(XH~/MM0N#IXS )BXSIIFXSe%/J((((J+((((((((((iMiMb*((((((N((((((((XS((((((((((((((((((" +$: Ordinal Logistic HLM
Timothy Olsen
HLM II Dr. Gagnes Class
ABSTRACT
Ordinal Logistic Hierarchical Linear Modeling (HLM) is a method that has seen limited use due to its complex nature, and lack of software to properly model it. Recently, modeling capabilities have been added to the HLM software ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"3T6TGG4C"}]} (Raudenbush, Bryk, & Congdon, 2004) which allow easy modeling of ordinal dependent variables in a logistic fashion. I describe the method, convey an interpretation of coefficients from a sample study, and discuss other research which has employed the method.
Keywords: ordinal logistic HLM, research methods, hierarchical linear modeling
I. Description of Method
In much of research our outcome variables are considered to be continuous with the range of observed sample. Examples of such outcomes are miles per gallon, weight, height, test scores, etc. Sometimes the outcome is a mutually exclusive or binary (0 or 1) measure, such as cheating or not cheating, graduating or not graduating, approving of Pinochet or not approving of Pinochet. This analysis is easily done with Logistic regression.
Many interesting dependent variables do not involve an either or proposition, but rather some meaningful ordinal response. A primary characteristic of ordinal data is that a number assigned to successive categories can represent varying orders of magnitude, or a greater than, less than quality (Stevens 1946). These scales can be found in many places in education and social science research. Examples in education might include grades (0 = F, 1 = D, 2 = C, 3 = B, 4 = A), or levels of quality work (0 = un-satisfactory, 1 = satisfactory, 2 = excellent). Examples in other fields might include injury sustained in car accidents (0 = mild, 1 = moderate, 2 = severe) or alcohol problems of parents (0 = none, 1 = some, 2 = several).
The level of difficulty in analysis of comparing two levels is much more complex for ordinal than binary dependent variables. In a binary scale, the probability of graduating on time is the inverse of the probability of not graduating on time. When this is extended to an ordinal scale of graduating on time, graduating late, and not graduating at all the same inverse proportionality rule does not apply.
Because this analysis is more complex, there is little published research which use it. A Google search for ordinal logistic HLM yields three results, which are publications from the researchers at UCLA who did a study on test assessments which favor English learners (cite). I will discuss this paper in Section 4.
The methodologist who has written the most in this area is Hedeker, however his writings are at the multilevel statistic level, and not specific to the HLM software. Hedeker has a seminal paper in this area ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"QTQVMKXK"}]} (Hedeker & Gibbons, 1994), and a more recent book chapter which explains the concept of thresholds quite well ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"K3TGUIH6"}]} (Hedeker, 2007). His homepage has a freely-available copy of this book chapter, and other resources ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"9DBFWGMQ"}]} (Multilevel Analysis - Don Hedeker, 2009).
II. Example
I will illustrate this using a serious parenting and social issue, child punishment. The design of this hypothetical study consists of a survey sent to parents in each state of the United States. The survey asked each parent what their level of acceptability of physical punishment for children was ( 3 = high, 2 = medium, 1 = low). It also asked them gender (0 = female, 1 = male), and their educational level (0 = no BS degree, 1 = BS degree or higher). In addition, state government statistics of number of deaths resulting from child punishment were recorded in the level-2 state data.
We would like to see what effect gender and educational level, and have on the personal level of acceptability of physical punishment for children. We also would like to see what affect the state level has.
Putting this into HLM for Windows ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"3T6TGG4C","position":1}]} (Raudenbush et al., 2004), gives us these Level 1 and Level 2 equations:
Level 1 Equation
Prob[R = 1|B] = P'(1) = P(1)
Prob[R <= 2|B] = P'(2) = P(1) + P(2)
Prob[R <= 3|B] = 1.0
where
P(1) = Prob[ChildPunishmentAcceptability(1) = 1|B]
P(2) = Prob[ChildPunishmentAcceptability (2) = 1|B]
log[P'(1)/(1 - P'(1))] = B0 + B1*(GENDER) +B2*(EducationLevel)
log[P'(2)/(1 - P'(2))] = B0 + B1*(GENDER) +B2*(EducationLevel) + d(2)
Note: Differing from Logistic regression, cumulative probabilities are at play here, this explains the three different probability functions. The first probability function is the probability of being in category 1. The second probability function is the probability of being in category 2 or 1. The third probability function is the probability in being in any category, and thus equal to 1. The d(2) at the end of the last equation signifies the difference necessary to adjust from the first probability function, to the second.
Level 2 Equations
b0j=g00+g01Num_deathsj+ + u0j
b1j=g10
b2j=g20
d(2)
Table 1. Acceptability of Physical Punishment of ChildrenCoefficients (log-odds)Interceptg00-1.78Femaleg10-0.32Educationg20-0.24Num_Deathsg010.3Threshold d(2)0.86
Interpreting coefficients in Ordinal Logistic HLM, is the same as interpreting coefficients in logistic HLM. However, in Ordinal Logistic HLM there is one new thing the Threshold level. There will always be n-2 threshold for every ordinal category. Therefore we only one threshold in this example.
Negative Coefficients can be interpreted decreasing likelihood of being accepting of physical punishment of children. Positive coefficients can be interpreted as increasing likelihood of being accepting. We can interpret the coefficient of e(00 by transforming it from log-odds to probability form: p = e(00 / (1 + e(00 ). Doing so yields .14, which we can interpret as %14 of all people sampled have a very high acceptability of child punishment.
Threshold levels give you the probability of n-1 categories. In our case this is high (=1) and medium (=2) categories, the bottom two categories. By adding this coefficient to the transformation of the log-odds scale we can receive the probability the bottom two levels: p = e(00+(2) / (1 + e(00+(2)) = p = e-1.78+.86 / (1 + e-1.78+.86 ) = 28%. By derivation the probability that respondents had a low level of acceptability is 72%.
The other coefficients can be easily interpreted as probabilities in like manner. As a rule of thumb, negative coefficients are indicative of a decreasing likelihood of viewing physical child punishment as acceptable. Thus, as the number of deaths (per million) due to child maltreatment goes up by one, the likelihood of the states inhabitants to have higher acceptability levels rises.
III. Methodological Issues
Ordinal Logistic HLM has similar sample size and other issues which effect logistic HLM, and HLM. The mathematics behind ordinal logistic HLM are quite a bit more complex than logistic HLM.
At this point, I am unaware of other issues which are specific to this method.
IV. examples from the literature
There are not many examples from the literature which use ordinal logistic HLM methods. However, there are a few great examples that are simple to understand and well written, I will discuss these papers in the Exemplars section below, other research which uses multilevel methods and ordinal dependent variables I will discuss in the Others section below.
EXEMPLARS
Two researchers from Universities in Spain have written two papers which are well written and good exemplars of the method. Their first work ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"HSD7KTSR"}]} (Gracia & Herrero, 2008) is a multilevel study of 10,812 people nested within 208 cities in 14 countries of the European Union. The dependent or outcome variable which they measure is three category ordinal measure of acceptability of physical punishment for children. They collect several individual level measures such as gender, education level and age. They also use country data about the number of deaths from child maltreatment (per million people), and a binary measure of if physical punishment was banned in that country by 1999. Their results section is well written and is very helpful to the amateur methodologist. As their ordinal scale consists of three items they have report one threshold level. Their use of the method is cleanly done, and simple to understand.
The other work ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"ZVDJQEJ3"}]} (Gracia & Herrero, 2006) is similar, but deals with acceptability of domestic violence against women (DVAW). Their data comes from 13,475 people nested in 212 cities within 15 countries of the European Union. The dependent variable consists of a four level ordinal measure of acceptability of domestic violence against women (1 = always unacceptable, 4 = always acceptable). Their notable results include significant effects of victim blaming, and perceived lack of DVAW as a problem both of which lead to an increased likelihood in acceptability of DVAW.
Another great exemplar includes the work of the UCLA researchers who are interested the factors involved in the achievement gap between English Learners and non-English Learners in elementary schools ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"CP829C7T"}]} (Boscardin, Aguirre-Munoz, Chinen, Leon, & Shin, 2004).
Their sample consists of 1,038 6th grade students taught by 27 different teachers who are a part of 7 elementary schools in Los Angeles, California. Each student was administed a Language Arts Performance Assignment in which they were asked to choose a literary work which contains a heroic character and describe the character in writing, citing the text. These assignments were then rated by their teachers who assigned four point grade to their work (1 = sub-standard, 4 = best). This was used as the dependent or outcome variable. At the teacher level data was collected on subject matter expertise and classroom time spent on literary analysis and writing. This paper is unique in that it used raters to determine the ordinal values, rather than perceptual measures.
OTHERS
Other papers do not include threshold values and are less clear about their use of HLM. One example is a public health study from Taiwan. This study is motivated by the fact that although women live longer than men on average, they report poorer health. ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"BUXWRGV7"}]} (Chen, Chang, & Yang, 2008) investigated how gender roles derived from personal and household factors may affect perceived health measures. Their sample consists of 50,354 respondents in 13,000 households in 867 villages. The ordinal dependent variable consists of a perceived health measure ( 1 = good, 4 = poor and not capable of self-care ). This study is interesting, however as they do not list threshold levels it is difficult to see the probabilities groups. Another similar study breaks a binary measure of prior consumption of alcohol and no prior consumption of alcohol ADDIN ZOTERO_ITEM {"sort":true,"citationItems":[{"key":"BHITQHXV"}]} (Med, 2004).
REFERENCES
ADDIN ZOTERO_BIBL Boscardin, C. K., Aguirre-Munoz, Z., Chinen, M., Leon, S., & Shin, H. S. (2004). Consequences and Validity of Performance Assessment for English Learners: Assessing Opportunity to Learn (OTL) in Grade 6 Language Arts. CSE Report 635. Center for Research on Evaluation Standards and Student Testing CRESST, 78.
Chen, D. R., Chang, L. Y., & Yang, M. L. (2008). Gender-specific responses to social determinants associated with self-perceived health in Taiwan: A multilevel approach. Social Science & Medicine.
Gracia, E., & Herrero, J. (2006). Acceptability of domestic violence against women in the European Union: a multilevel analysis. Journal of Epidemiology & Community Health, 60(2), 123.
Gracia, E., & Herrero, J. (2008). Is It Considered Violence? The Acceptability of Physical Punishment of Children in Europe. Journal of Marriage and Family, 70#5=>GXd+ , r s v
w
!++7HT
Est ekIJ`az{'ǿǻǻǿǿǻäÜٜÔjht+Ujh]EUhZh%h~
h xh{CJaJhXh{ht+hv
hYV0JhYVOJQJhYVhYVjhYVUh.hYVhR.hbh hqw):#>Gw
tJMd`gd8=dgd8=dgdqw)$d5$7$8$9DH$a$gdqw)dgdYVgdYV d1$gdqw);dgdqw)9dgdqw)'(*}~/:LM->?!"@tܭ}vhhSLzh3(hSLzh8=h8=!hh8=56B*H*]phhh8=56B*]phjhYVUh8=hFh.hh xh=[hhqw)h h0,ht+hYVhYVhYVjhZUhZht+jht+U(1GN@,@L$d$5$7$8$9DH$Ifa$gd<=dgddgd8=
$&,.28:>@HJL
ִssb!h<=h<=B*OJQJaJphh<=h<=CJOJQJ%h<=h<=B*CJOJQJaJphhh xB*phhh8=B*phhquhquB*phhquB*phh3(hquB*phh3(h8=B*OJQJphh3(h8=B*H*phh3(h8=B*ph!hh8=56B*\]ph$'gkd$$IflI019
G@G
tH
644
lBa]pyt<=$d$5$7$8$9DH$Ifa$gd<=Xkd$$Iflv9
GH
t
H
644
lBa]p
yt<=
.6BNkd0$$IflF1
9
g@gag
tH
644
lBa]pyt<=$d$5$7$8$9DH$Ifa$gd<=$d$5$7$8$9DH$Ifa$gd<= .68BDX`bln !!!λλޞ{njfbfbW jghNphH*hhh<=h<=hCJOJQJhB*CJOJQJaJph%h<=hB*CJOJQJaJphh8?B*CJOJQJaJphh<=h<=CJOJQJ%h<=h<=B*CJOJQJaJphhB*CJOJQJaJph!h<=h<=B*OJQJaJph h<=h<=B*H*^JaJph BDX`lggN$d$5$7$8$9DH$Ifa$gd<=$d$5$7$8$9DH$Ifa$gd<=kd$$Ifl;F1
9
g@gag
tH
644
lBa]pyt<=lnggN$d$5$7$8$9DH$Ifa$gd<=$d$5$7$8$9DH$Ifa$gd<=kd$$Ifl;F1
9
g@gag
tH
644
lBa]pyt<=ggN$d$5$7$8$9DH$Ifa$gd<=$d$5$7$8$9DH$Ifa$gd<=kd$$IflF1
9
g@gag
tH
644
lBa]pyt<=ggN$d$5$7$8$9DH$Ifa$gd<=$d$5$7$8$9DH$Ifa$gd<=kdD$$Ifl,F1
9
g@gag
tH
644
lBa]pyt<= T":%"'=''K(l(xpphh_xx_ d1$gdqw)dgdL.dgddgdqw)kd $$Ifl,F1
9
g@gag
tH
644
lBa]pyt<=
!!!!!!!!!!!!!!!T"h#i#k#$$$$"$$$,$.$>$J$P$d$p$v$$$8%:%%,&v&&&!'"'''<'='''K(O(k(l(())))[*ȽȽhh5hhZh3(hqw)h hFh8?h.hL.H*\mH sH hL.H*hL.hhH*\mH sH hH* jghNphH*hhhNphH*:l())-J0144889:V;`x``a+b{b}b~bbbbbb0d^`0gd. d1$gdqw)dgdqw)[*l*m*****,, ,----..3.4..../a/I0J00111Y1Z111111144444T5555566Q6W6?777*8+8Ѿ锪hAjh~Uh. hqw)5 h.5h~hFhFH*hFh,jh9>tUhZhZ$|hih9>tjhZ$|UhZ$|h3(hihYVhYVhYVjhUh7+8,8r8s8~8888888899::;H;J;L;;;;;`W`v`````aaaa[bjbybzb{b|b~bbbbbbbbbbbbbb
hH0J=jhH0J=UhH
ht+CJ
hHCJh?gjh?gUh9h9Uh.h.6hZ$|jhZ$|Uh h~hAhAh.h.h.jh.U6(1), 210-217.
Hedeker, D. (2007). Multilevel models for ordinal and nominal variables. Handbook of Multilevel Analysis.
Hedeker, D., & Gibbons, R. D. (1994). A random-effects ordinal regression model for multilevel analysis. Biometrics, 50(4), 933-44.
Med, K. J. P. (2004). A Study of the Relationship between Parental Alcohol Problems and Alcohol Use among Adolescent Females in Republic of Korea. Korean J Prev Med, 37(3), 232-237.
Multilevel Analysis - Don Hedeker. (2009). . Retrieved February 21, 2009, from http://tigger.uic.edu/%7Ehedeker/ml.html.
Raudenbush, S., Bryk, A., & Congdon, R. (2004). HLM for Windows. Version 6.06.
Olsen Ordinal Logistic HLM Primer
PAGE 6
Hierarchical Linear Modeling II Phil Gagne - GSU
bbbbbbbbbb0d^`0gd.??h]h?&`D bbbbbbbbbh9h9h?ghH
hH6CJ
h%6CJhH0J=OJQJjhH0J=UhF0J=mHnHu21F:pH/ =!"#$%$$If]!vh5H
#vH
:Vlv
t
H
6,5H
/Ba]p
yt<=$$If]!vh5@5#v@#v:VlI
tH
6,5@5/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl
tH
6,5@5a59/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl;
tH
6,5@5a59/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl;
tH
6,5@5a59/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl
tH
6,5@5a59/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl,
tH
6,5@5a59/Ba]pyt<=$$If]!vh5@5a5#v@#va#v:Vl,
tH
6,5@5a59/Ba]pyt<=^O 666666666666666 666666666666666666666666666 6666666666 666666666666 6666666666666666666666666666666666666666666666666666666666666666662 0@P`p2( 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p8XV~_HmH nH sH tH d`dzNormal"$dhx5$7$8$9DH$a$CJOJQJ_HmH sH tH Z@Zz0 Heading 1$ 1$@&a$5;B*CJphDDz0 Heading 2$1$@&a$5;FFz0 Heading 3$x<1$@&a$5>>z0 Heading 4$1$@&a$6<<z0 Heading 5$1$@&5JJz0 Heading 6
<@&6CJOJQJ<<z0 Heading 7$@&5CJBBz0 Heading 8$d@&a$5D Dz0 Heading 9
<@&
56CJDA`DDefault Paragraph FontRi@R0Table Normal4
l4a(k (
0No ListRRz0Heading 1 Char5CJ KH OJQJ\^JaJ TTz0Heading 2 Char 56CJOJQJ\]^JaJPOPz0Heading 3 Char5CJOJQJ^JmH sH N!Nz0Heading 4 Char5CJOJQJ\^JaJT1Tz0Heading 5 Char 56CJOJQJ\]^JaJFAFz0Heading 6 Char5OJQJ\^JHQHz0Heading 7 CharCJOJQJ^JaJNaNz0Heading 8 Char6CJOJQJ]^JaJ@q@ z0Heading 9 CharOJQJ^JTTz0Heading 2 Char15;CJOJQJ^JmH sH JJz0
Bulleted List&^&`>>z0
Table Textda$CJ66z0Table Titlea$88z0Style Italic688z0Comment TextCJNNz0Comment Text CharCJOJQJ^JaJ8+8 z0Endnote TextCJNNz0Endnote Text CharCJOJQJ^JaJ2
2z0Index 1
!^`22z0Index 2
"^`22z0Index 3
#^`2
2z0Index 4
$^`22z0Index 5
%^`22z0Index 6
&^`22z0Index 7
'^`22z0Index 8
(^`22z0Index 9
)^p`:!:z0
Index Heading*5- ,z0
Macro TextF+$
` @
dh5$7$8$9DH$`a$OJQJ_HmH sH tH N/N+z0Macro Text CharOJQJ_HmH sH tH L,Lz0Table of Authorities
-^`D#Dz0Table of Figures
.^` :.:z0TOA Heading/x5&&z0TOC 10**z0TOC 21^**z0TOC 32^**z0TOC 43^**z0TOC 54^**z0TOC 65^**z0TOC 76^**z0TOC 87^**z0TOC 98^>>@>:z0Title91$a$5;CJ KHJJ9z0
Title Char5CJ KHOJQJ\^JaJ 6O6zAuthor;da$>U>z0 Hyperlink>*B*CJ^Jph>)@>z0Page NumberCJOJQJ^J8"8z0Caption><1$a$H @H@z0Footer?d!1$a$CJBB?z0Footer CharCJOJQJ^JaJ<<zE-mail addressACJB"Bz010 pointBd]^CJH2Hz0NNormal BulletC^`4@B4Ez0Header
D!BQBDz0Header CharCJOJQJ^JaJ@b@(0List Paragraph F^m$HrHH@l0Balloon TextGCJOJQJ^JaJBBGz0Balloon Text CharCJ^J<<P&0apple-style-span^JFFP&0apple-converted-space^J((P&0nfakpe^JF'FM0Comment ReferenceCJ^JaJ@j@NM0Comment SubjectM5\FFMd0Comment Subject Char5\PK![Content_Types].xmlj0Eжr(Iw},-j4 wP-t#bΙ{UTU^hd}㨫)*1P' ^W0)T9<l#$yi};~@(Hu*Dנz/0ǰ$X3aZ,D0j~3߶b~i>3\`?/[G\!-Rk.sԻ..a濭?PK!֧6_rels/.relsj0}Q%v/C/}(h"O
= C?hv=Ʌ%[xp{۵_Pѣ<1H0ORBdJE4b$q_6LR7`0̞O,En7Lib/SeеPK!kytheme/theme/themeManager.xmlM
@}w7c(EbˮCAǠҟ7՛K
Y,
e.|,H,lxɴIsQ}#Ր ֵ+!,^$j=GW)E+&
8PK!Ptheme/theme/theme1.xmlYOo6w toc'vuر-MniP@I}úama[إ4:lЯGRX^6؊>$!)O^rC$y@/yH*)UDb`}"qۋJחX^)I`nEp)liV[]1M<OP6r=zgbIguSebORD۫qu gZo~ٺlAplxpT0+[}`jzAV2Fi@qv֬5\|ʜ̭NleXdsjcs7f
W+Ն7`gȘJj|h(KD-
dXiJ؇(x$(:;˹!I_TS1?E??ZBΪmU/?~xY'y5g&/ɋ>GMGeD3Vq%'#q$8K)fw9:ĵ
x}rxwr:\TZaG*y8IjbRc|XŻǿI
u3KGnD1NIBs
RuK>V.EL+M2#'fi~Vvl{u8zH
*:(W☕
~JTe\O*tHGHY}KNP*ݾ˦TѼ9/#A7qZ$*c?qUnwN%Oi4=3ڗP
1Pm\\9Mؓ2aD];Yt\[x]}Wr|]g-
eW
)6-rCSj
id DЇAΜIqbJ#x꺃6k#ASh&ʌt(Q%p%m&]caSl=X\P1Mh9MVdDAaVB[݈fJíP|8քAV^f
Hn-"d>znǊ ة>b&2vKyϼD:,AGm\nziÙ.uχYC6OMf3or$5NHT[XF64T,ќM0E)`#5XY`פ;%1U٥m;R>QDDcpU'&LE/pm%]8firS4d7y\`JnίIR3U~7+#mqBiDi*L69mY&iHE=(K&N!V.KeLDĕ{D vEꦚdeNƟe(MN9ߜR6&3(a/DUz<{ˊYȳV)9Z[4^n5!J?Q3eBoCMm<.vpIYfZY_p[=al-Y}Nc͙ŋ4vfavl'SA8|*u{-ߟ0%M07%<ҍPK!
ѐ'theme/theme/_rels/themeManager.xml.relsM
0wooӺ&݈Э5
6?$Q
,.aic21h:qm@RN;d`o7gK(M&$R(.1r'JЊT8V"AȻHu}|$b{P8g/]QAsم(#L[PK-![Content_Types].xmlPK-!֧6+_rels/.relsPK-!kytheme/theme/themeManager.xmlPK-!Ptheme/theme/theme1.xmlPK-!
ѐ' theme/theme/_rels/themeManager.xml.relsPK]
N3"h 00ppps'![*+8bb "%+-.3Bll(bb!#$&'()*,2+r`z'}>8?"""~%%%
*Q*m*,,,,
-2N3QQQQQQQQQQQ079s!T #AA@0(
B
S ?89""m*n*22222222222222L3O3222222222222L3O35=Xde k I
J
01FF[[%&((,%,,,,,,,-2222222222222
3L3O35=Xde k I
J
01FF[[%&((,%,,,,,,,-22O3<++,2W,x/r,@S
,(4
,}#,@#`,1,,A^,3`7,,/,Y,s,(,,x!",L#,?@
$,#J&&,$,&,/',*',^',?w*,R1,,jR-,g/,2,s,6,pu9,p=,c?,@{A,qB,eaL,|N,VR.#
\T,4dX,v\,C"P],R],4]_,+c,\5d,WWd,f,:$j,4Rk,d7p,gp,0r,r,8s,'s,T(u,:,@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(^`^Jo()^`^J.pL^p`L^J.@^@`^J.^`^J.L^`L^J.^`^J.^`^J.PL^P`L^J.@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(^`^J.^`^J.pL^p`L^J.@^@`^J.^`^J.L^`L^J.^`^J.^`^J.PL^P`L^J.@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh&^&`OJQJ^Jo(@xh8^8`OJQJ^Jo(@xh8^8`OJQJ^Jo(<2WeaLs,6pu9$,&0r4dX4]_g/(4
r's*'8sR]++#J&&L#f3`7T(uc?p=/\TWWd?@
$+c^'qBs@S
/'A^Y}#(2:C"P]:$j?w*|N\5d4Rkgp#`v\R1,d7p1x/rx!"@{AjR-VR@<< xJpM3R`7jDLck}I8n m ,{)&A22L];B!2$P&(c0(Y)qw).A.L.Q1V3\78;<8=<=
?8?:CoD]EBGb:I9K`*P{PNQWRhS U9:UYVgtqu x]pxSLz@^z{'{Z$|lY|~Dd~|j"~
X@lueMF5S]64jtny:F-A5lE$vv$,$Zz@9cl$i3(R.0F6H`#t+^dz2/mDB^b:0 3zBI#N;-4?{O><aV;"NKJS9i'6@&fT2\Y "X}%;|~X.| ::(0,22@8
Xlla0N3@@@$L@@UnknownG* Times New Roman5Symbol3.* Arial7.{ @Calibri7K@Cambria?= * Courier New5.*aTahomaACambria Math"1h:fZf9fB+ EB+ E!xx0222QKP ?c0(2:!xx- AIS SIG-ED Conference Submission GuidelinessOriginal author: Paul Gray Tim Olsen<
!"#$%&'()*+,-./0123456789:;Oh+'0(
0 AIS SIG-ED Conference Submission GuidelinessOriginal author: Paul GrayhThis paper is based on the CAIS format guidelines available from: http://cais.aisnet.org/format.aspNormal.dotmTim Olsen3Microsoft Office Word@40@
@ds@ B+՜.+,D՜.+,t0hp
Claremont Graduate UniversityE2. AIS SIG-ED Conference Submission GuidelinessTitle :BZOTERO_PREF_1@UoIfgkRn:http:://www.zotero.org/styles/apa:in-text:1:0:False
!"#$%&'()*+,-./012346789:;<>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgijklmnoqrstuvwzRoot Entry F`<|Data
51Table=TTWordDocument4hSummaryInformation(hDocumentSummaryInformation8pCompObjy
F'Microsoft Office Word 97-2003 Document
MSWordDocWord.Document.89q