Illusory intuitive inferences: Matching heuristics explain logical intuitions (Supplementary Materials)

1 Authors’ Notes

This document contains supplementary materials of the following study:

Ghasemi, O., Handley, S., Howarth, S. (2022). Illusory intuitive inferences: Matching heuristics explain logical intuitions 

All materials, data, and analysis scripts can be retrieved at the Open Science framework (https://osf.io/we27a/). 

All correspondence related to this project should be addressed to: 

Omid Ghasemi (Macquarie University, Sydney, NSW, 2109, AUSTRALIA) 

Email: omidreza.ghasemi@mq.edu.au 

2 Materials

• Materials including reasoning problems can be retrieved at the Open Science framework (https://osf.io/3cvxe/). 

Tables S1, S2, and S3 show examples of conflict and non-conflict arguments of Experiments 2, 3, and 4

Table S1: Examples of argument types across conflict and polarity conditions of Experiment 2

Affirmative			Negative
	Non-conflict	Conflict		Non-conflict	Conflict
If p then q p or q, not both not-p	If whales can walk then they can climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they can travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No	If not-p then q not-p or else q p	If Whales cannot swim then they can travel the oceans Either whales cannot swim or they can travel the oceans, but not both Therefore, whales can swim L – Yes B - Yes	If whales cannot walk then they can climb mountains Either whales cannot walk or they can climb mountains, but not both Therefore, whales can walk L – Yes B - no
If p then q p or q, not both p	If whales can walk then they can climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they can travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales can swim L – No B - Yes	If not-p then q not-p or else q not-p	If Whales cannot swim then they can travel the oceans Either whales cannot swim or they can travel the oceans, but not both Therefore, whales cannot swim L – No B - No	If whales cannot walk then they can climb mountains Either whales cannot walk or they can climb mountains, but not both Therefore, whales cannot walk L - No B - Yes
If p then not-q p or not-q, not both not-p	If whales can walk then they cannot climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they cannot travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No	If not-p then not-q not-p or not-q, not both p	If Whales cannot swim then they cannot travel the oceans Either whales cannot swim or they cannot travel the oceans, but not both Therefore, whales can swim L – Yes B - Yes	If whales cannot walk then they cannot climb mountains Either whales cannot walk or they cannot climb mountains, but not both Therefore, whales can walk L – Yes B - no
If p then not-q p or not-q, not both p	If whales can walk then they cannot climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they cannot travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales can swim L – No B - Yes	If not-p then not-q not-p or not-q, not both not-p	If Whales cannot swim then they cannot travel the oceans Either whales cannot swim or they cannot travel the oceans, but not both Therefore, whales cannot swim L – No B - No	If whales cannot walk then they cannot climb mountains Either whales cannot walk or they cannot climb mountains, but not both Therefore, whales cannot walk L - No B - Yes

Table S2: Examples of argument types across conflict and alignment conditions of Experiment 3

Aligned			Non-Aligned
	Non-conflict	Conflict		Non-conflict	Conflict
If and only if p then q p or q, or both p	If and only if whales can swim then they can travel the oceans Whales can swim or they can travel the oceans, or both Therefore, whales can swim L – Yes B - Yes	If and only if whales can walk then they can climb mountains Whales can walk or they can climb mountains, or both Therefore, whales can walk L – Yes B - No	If p then q p or else q not-p	If whales can walk then they can climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they can travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No
If and only if p then q p or q, or both Not-p	If and only if whales can swim then they can travel the oceans Whales can swim or they can travel the oceans, or both Therefore, whales cannot swim L – No B - No	If and only if whales can walk then they can climb mountains Whales can walk or they can climb mountains, or both Therefore, whales cannot walk L - No B - Yes	If p then q p or else q p	If whales can walk then they can climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they can travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales can swim L – No B - Yes
If and only if p then not-q p or not-q, or both p	If and only if whales can swim then they cannot travel the oceans Whales can swim or they cannot travel the oceans, or both Therefore, whales can swim L – Yes B - Yes	If and only if whales can walk then they cannot climb mountains Whales can walk or they cannot climb mountains, or both Therefore, whales can walk L – Yes B - No	If p then not-q p or else not-q not-p	If whales can walk then they cannot climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they cannot travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No
If and only if p then not-q p or not-q, or both Not-p	If and only if whales can swim then they cannot travel the oceans Whales can swim or they cannot travel the oceans, or both Therefore, whales cannot swim L – No B - No	If and only if whales can walk then they cannot climb mountains Whales can walk or they cannot climb mountains, or both Therefore, whales cannot walk L - No B - Yes	If p then not-q p or else not-q p	If whales can walk then they cannot climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they cannot travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales can swim L – No B - Yes

Table S3: Examples of argument types across conflict and matching conditions of Experiment 4

Matching absent			Matching Present
	Non-conflict	Conflict		Non-conflict	Conflict
If p then q Not-p or else not-q not-p	If whales can walk then they can climb mountains Either whales cannot walk or they cannot climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they can travel the oceans Either whales cannot swim or they cannot travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No	If and only if p then q p or q, or both p	If and only if whales can swim then they can travel the oceans Whales can swim or they can travel the oceans, or both Therefore, whales can swim L – Yes B - Yes	If and only if whales can walk then they can climb mountains Whales can walk or they can climb mountains, or both Therefore, whales can walk L – Yes B - No
If p then q Not-p or else not-q p	If whales can walk then they can climb mountains Either whales cannot walk or they cannot climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they can travel the oceans Either whales cannot swim or they cannot travel the oceans, but not both Therefore, whales can swim L – No B - Yes	If and only if p then q p or q, or both Not-p	If and only if whales can swim then they can travel the oceans Whales can swim or they can travel the oceans, or both Therefore, whales cannot swim L – No B - No	If and only if whales can walk then they can climb mountains Whales can walk or they can climb mountains, or both Therefore, whales cannot walk L - No B - Yes
If p then not-q Not-p or else q not-p	If whales can walk then they cannot climb mountains Either whales cannot walk or they can climb mountains, but not both Therefore, whales cannot walk L – Yes B - Yes	If Whales can swim then they cannot travel the oceans Either whales cannot swim or they can travel the oceans, but not both Therefore, whales cannot swim L – Yes B - No	If and only if p then not-q p or not-q, or both p	If and only if whales can swim then they cannot travel the oceans Whales can swim or they cannot travel the oceans, or both Therefore, whales can swim L – Yes B - Yes	If and only if whales can walk then they cannot climb mountains Whales can walk or they cannot climb mountains, or both Therefore, whales can walk L – Yes B - No
If p then not-q Not-p or else q p	If whales can walk then they cannot climb mountains Either whales cannot walk or they can climb mountains, but not both Therefore, whales can walk L - No B - No	If Whales can swim then they cannot travel the oceans Either whales cannot swim or they can travel the oceans, but not both Therefore, whales can swim L – No B - Yes	If and only if p then not-q p or not-q, or both Not-p	If and only if whales can swim then they cannot travel the oceans Whales can swim or they cannot travel the oceans, or both Therefore, whales cannot swim L – No B - No	If and only if whales can walk then they cannot climb mountains Whales can walk or they cannot climb mountains, or both Therefore, whales cannot walk L - No B - Yes
If not-p then q p or else not-q p	If Whales cannot swim then they can travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales can swim L – Yes B - Yes	If whales cannot walk then they can climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales can walk L – Yes B - No	If and only if not-p then q Not-p or q, or both Not-p	If and only if whales cannot walk then they can climb mountains Whales cannot walk or they can climb mountains, or both Therefore, whales cannot walk L – Yes B - Yes	If and only if whales cannot swim then they can travel the oceans Whales cannot swim or they can travel the oceans, or both Therefore, whales cannot swim L – Yes B - No
If not-p then q p or else not-q not-p	If Whales cannot swim then they can travel the oceans Either whales can swim or they cannot travel the oceans, but not both Therefore, whales cannot swim L - No B - No	If whales cannot walk then they can climb mountains Either whales can walk or they cannot climb mountains, but not both Therefore, whales cannot walk L – No B - Yes	If and only if not-p then q Not-p or q, or both p	If and only if whales cannot walk then they can climb mountains Whales cannot walk or they can climb mountains, or both Therefore, whales can walk L – No B - No	If and only if whales cannot swim then they can travel the oceans Whales cannot swim or they can travel the oceans, or both Therefore, whales can swim L - No B - Yes
If not-p then not-q p or else q p	If Whales cannot swim then they cannot travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales can swim L – Yes B - Yes	If whales cannot walk then they cannot climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales can walk L – Yes B - No	If and only if not-p then not-q Not-p or not-q, or both Not-p	If and only if whales cannot walk then they cannot climb mountains Whales cannot walk or they cannot climb mountains, or both Therefore, whales cannot walk L – Yes B - Yes	If and only if whales cannot swim then they cannot travel the oceans Whales cannot swim or they cannot travel the oceans, or both Therefore, whales cannot swim L – Yes B - No
If not-p then not-q p or else q not-p	If Whales cannot swim then they cannot travel the oceans Either whales can swim or they can travel the oceans, but not both Therefore, whales cannot swim L - No B - No	If whales cannot walk then they cannot climb mountains Either whales can walk or they can climb mountains, but not both Therefore, whales cannot walk L – No B - Yes	If and only if not-p then not-q Not-p or not-q, or both p	If and only if whales cannot walk then they cannot climb mountains Whales cannot walk or they cannot climb mountains, or both Therefore, whales can walk L – No B - No	If and only if whales cannot swim then they cannot travel the oceans Whales cannot swim or they cannot travel the oceans, or both Therefore, whales can swim L - No B - Yes

3 Frequentist Analysis

3.1 Model Specifications

To see the model structures, click on the “Code” button below.

#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 1   -------------------------####
#-------------------------------------------------------------------------------#

exp1_m1 <- mixed(accuracy ~ conflict * instruction  +
                   (1 |subject)+
                   (1 |content),
                 exp1_df, method = "LRT",
                 control = glmerControl(optimizer="bobyqa"),
                 expand_re = TRUE, family=binomial(link="logit"))


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 2   -------------------------####
#-------------------------------------------------------------------------------#

exp2_m1 <- mixed(accuracy ~ conflict * instruction * matching  +
                   (1 |subject)+
                   (1 |content),
                 exp2_df, method = "LRT",
                 control = glmerControl(optimizer="bobyqa"),
                 expand_re = TRUE, family=binomial(link="logit"))


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 3   -------------------------####
#-------------------------------------------------------------------------------#

exp3_m1 <- mixed(accuracy ~ conflict * instruction * alignment  +
                   (1 |subject)+
                   (1 |content),
                 exp3_df, method = "LRT",
                 control = glmerControl(optimizer="bobyqa"),
                 expand_re = TRUE, family=binomial(link="logit"))


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 4   -------------------------####
#-------------------------------------------------------------------------------#

exp4_m1 <- mixed(accuracy ~ conflict * instruction * matching  +
                   (1 |subject)+
                   (1 |content),
                 exp4_df, method = "LRT",
                 control = glmerControl(optimizer="bobyqa"),
                 expand_re = TRUE, family=binomial(link="logit"))

3.2 Model Outputs

Table S4: The outputs of the frequentist mixed model of Experiment 1

Effect	df	Chisq	p.value
conflict	1	61.26 ***	<.001
instruction	1	931.46 ***	<.001
conflict:instruction	1	271.81 ***	<.001

Table S5: The outputs of the frequentist mixed model of Experiment 2

Effect	df	Chisq	p.value
conflict	1	28.52 ***	<.001
instruction	1	492.58 ***	<.001
matching	1	5.62 *	.018
conflict:instruction	1	123.39 ***	<.001
conflict:matching	1	16.05 ***	<.001
instruction:matching	1	4.65 *	.031
conflict:instruction:matching	1	16.12 ***	<.001

Table S6: The outputs of the frequentist mixed model of Experiment 3

Effect	df	Chisq	p.value
conflict	1	113.10 ***	<.001
instruction	1	140.07 ***	<.001
alignment	1	81.57 ***	<.001
conflict:instruction	1	59.42 ***	<.001
conflict:alignment	1	47.57 ***	<.001
instruction:alignment	1	93.79 ***	<.001
conflict:instruction:alignment	1	32.53 ***	<.001

Table S7: The outputs of the frequentist mixed model of Experiment 4

Effect	df	Chisq	p.value
conflict	1	51.47 ***	<.001
instruction	1	217.04 ***	<.001
matching	1	44.19 ***	<.001
conflict:instruction	1	16.56 ***	<.001
conflict:matching	1	8.00 **	.005
instruction:matching	1	58.62 ***	<.001
conflict:instruction:matching	1	5.31 *	.021

4 Baysian Analysis

4.1 Model Specifications

To see the model structures and prior specifications, click on the “Code” button below.

#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 1   -------------------------####
#-------------------------------------------------------------------------------#

m1_exp1 <- brm(accuracy ~ conflict*instruction + 
                 (conflict*instruction|subject) + 
                 (conflict*instruction|content),
               data=exp1_df,
               family=bernoulli("logit"),
               prior=c(prior(normal(0, 2), class = Intercept),
                       prior(normal(0, 2), class = b),
                       prior(cauchy(0, 1), class = sd),
                       prior(lkj(2), class = cor)),
               sample_prior = TRUE, 
               cores = 4, warmup = 1000, iter = 5000, chains = 4, seed = 123,
               save_pars = save_pars(all=TRUE),
               file = here("Output","brms_fits","m1_exp1_fit")
) 


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 2   -------------------------####
#-------------------------------------------------------------------------------#

m1_exp2 <- brm(accuracy ~ conflict*instruction*matching + 
                 (conflict*instruction*matching|subject) + 
                 (conflict*instruction*matching|content),
               data=exp2_df,
               family=bernoulli("logit"),
               prior=c(prior(normal(0, 2), class = Intercept),
                       prior(normal(0, 2), class = b),
                       prior(cauchy(0, 1), class = sd),
                       prior(lkj(2), class = cor)),
               sample_prior = TRUE, 
               cores = 4, warmup = 1000, iter = 5000, chains = 4, seed = 123,
               save_pars = save_pars(all=TRUE),
               file = here("Output","brms_fits","m1_exp2_fit")
)


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 3   -------------------------####
#-------------------------------------------------------------------------------#

m1_exp3 <- brm(accuracy ~ conflict*instruction*alignment + 
                 (conflict*instruction*alignment|subject) + 
                 (conflict*instruction*alignment|content),
               data=exp3_df,
               family=bernoulli("logit"),
               prior=c(prior(normal(0, 2), class = Intercept),
                       prior(normal(0, 2), class = b),
                       prior(cauchy(0, 1), class = sd),
                       prior(lkj(2), class = cor)),
               sample_prior = TRUE, 
               cores = 4, warmup = 1000, iter = 5000, chains = 4, seed = 123,
               save_pars = save_pars(all=TRUE),
               file = here("Output","brms_fits","m1_exp3_fit")
)


#-------------------------------------------------------------------------------#
####-----------------------------    Experiment 4   -------------------------####
#-------------------------------------------------------------------------------#

m1_exp4 <- brm(accuracy ~ conflict*instruction*matching + 
                 (conflict*instruction*matching|subject) + 
                 (conflict*instruction*matching|content),
               data=exp4_df,
               family=bernoulli("logit"),
               prior=c(prior(normal(0, 2), class = Intercept),
                       prior(normal(0, 2), class = b),
                       prior(cauchy(0, 1), class = sd),
                       prior(lkj(2), class = cor)),
               sample_prior = TRUE, 
               cores = 4, warmup = 1000, iter = 5000, chains = 4, seed = 123,
               save_pars = save_pars(all=TRUE),
               file = here("Output","brms_fits","m1_exp4_fit")
) 

4.2 Model Outputs

Table S8: The outputs of the Bayesian model of Experiment 1

term	estimate	std.error	conf.low	conf.high
(Intercept)	0.7478715	0.1089979	0.5364442	0.9648377
conflict1	-0.3279396	0.0756902	-0.4784158	-0.1830718
instruction1	1.0682235	0.1023331	0.8697215	1.2725722
conflict1:instruction1	0.5775502	0.0871542	0.4062783	0.7539034

Table S9: The outputs of the Bayesian model of Experiment 2

term	estimate	std.error	conf.low	conf.high
(Intercept)	0.6026398	0.0910738	0.4247744	0.7830674
conflict1	0.1915229	0.0661993	0.0640880	0.3240719
instruction1	0.7293928	0.0823929	0.5717144	0.8945272
matching1	-0.0725328	0.0431414	-0.1571885	0.0122330
conflict1:instruction1	-0.3441337	0.0588354	-0.4592809	-0.2285232
conflict1:matching1	-0.1330674	0.0411452	-0.2140646	-0.0520653
instruction1:matching1	0.0673518	0.0474855	-0.0259206	0.1622134
conflict1:instruction1:matching1	-0.1322899	0.0457057	-0.2228168	-0.0417246

Table S10: The outputs of the Bayesian model of Experiment 3

term	estimate	std.error	conf.low	conf.high
(Intercept)	0.8218164	0.0875337	0.6539985	0.9989513
conflict1	0.3424426	0.0656099	0.2155711	0.4735768
instruction1	0.4188129	0.0691874	0.2822381	0.5569815
alignment1	0.2834383	0.0479068	0.1892087	0.3785210
conflict1:instruction1	-0.2469014	0.0536713	-0.3544331	-0.1449685
conflict1:alignment1	0.2491230	0.0464249	0.1583721	0.3421874
instruction1:alignment1	-0.3131266	0.0541080	-0.4198150	-0.2061604
conflict1:instruction1:alignment1	0.2162046	0.0457233	0.1276616	0.3076102

Table S11: The outputs of the Bayesian model of Experiment 4

term	estimate	std.error	conf.low	conf.high
(Intercept)	1.0426540	0.1405211	0.7663983	1.3167498
conflict1	0.2136212	0.0499052	0.1169086	0.3138859
instruction1	0.5316987	0.1076000	0.3201402	0.7502870
matching1	-0.2113385	0.0464515	-0.3022402	-0.1213219
conflict1:instruction1	-0.1106335	0.0499203	-0.2090020	-0.0128317
conflict1:matching1	-0.0754793	0.0465011	-0.1672775	0.0155511
instruction1:matching1	0.2530217	0.0501773	0.1552709	0.3515141
conflict1:instruction1:matching1	-0.0692438	0.0462164	-0.1599525	0.0212917

4.3 Postrior Predictive Checks

As figure S12 shows, the simulated data from our Bayesian models mirrored the original data very well.

Figure S12: Posterior predictive checks of our Bayesian mixed models. For each plot, we used 1000 simulations.

4.4 Trace Plots

Figure S13 shows the trace plots for the theoretically significant effects for each model. As the figure shows, the chains have successfully converged. Moreover, none of population-level effects had \(\hat R\) values larger than 1 which is evidence of convergence.

Figure S13: trace plots of the theoretically important terms in each model