Consider the case of two A and B examiners, who evaluate the response sheets of 20 students in a class and mark each student as “passport” or “fail,” with each examiner reaching half of the students. Table 1 presents three different situations that can occur. In situation 1 in this table, eight students receive a pass score from the two examiners, eight from the examiners a “bad grade” and four from one examiner the pass mark, but the “fail” score of the other (two from A and the other two from B). Thus, the results of the two examiners are the same for 16/20 students (agreement – 16/20 – 0.80, disagreement – 4/20 – 0.20). It looks good. However, it is not taken into account that some notes could have been presumptions and that the agreement could have been reached by chance. The analysis of the agreement with more than two ratchets is a complex and controversial subject, see Fleiss (1981, p. 225). Once your company has a more agile contract analysis and the results are displayed in your KPI, you may feel more confident about taking on new business obligations. Setting up new offers will be much easier if you have your entire contract library at your fingertips during negotiation. You can then work more efficiently and take bolder steps in growing your business.
Maxwell`s statistically significant test statistics mentioned above show that advisors disagree in at least one category. McNemar`s general statistics indicate that differences of opinion are not evenly distributed. N.b. You cannot reliably compare Kappa values from different studies, as kappa is sensitive to the prevalence of different categories. In other words, if one category is observed more frequently in one study than another, Kappa may indicate a difference in the inter-rater agreement that is not due to advisors. Basically and above, a kappa less than 0.2 for poor match, and a kappa above 0.8 indicates a very good adhesion beyond chance. Weighted Kappa partly compensates for a problem with unweighted kappa, namely that it is not suited to the degree of disagreement.