Researching the bleeding obvious

ACMA has today released a research report titled Australia in the Digital Economy: Trust and confidence.

In their media release and the executive summary we are informed that

Australians value the internet and see it as critical to their daily lives...However, while Australians overwhelmingly see the internet as having affected their lives positively, they still have concerns about the potential for the internet to negatively affect their privacy and security. These concerns currently do not form a barrier to participation in the online environment, as increasing numbers of people use the internet for a wide range of activities including e-commerce and social networking.

One could posit that original primary research wasn't required to reach these conclusions. Perhaps the research is justified if it gives a clear lead on policy implications.*

The interesting part of the research is the analysis of users self-evaluation of their Internet skills where 36% od respondants rated themselves as above average skill levels (12% very, 24% somewhat), while 45% rated their skill level as average. (Of the 19% who rated themselves below average, it was 7% in very much below average and 12% somewhat). Is this possible? Shouldn't an equal number be below and above average? That depends on what meaning of "average" is used - if it was a median then that is what you'd expect, if it was a mean the kind of outcome above is possible. If we assume "average" means the mean plus or minus one standard deviation the outcome is possible on a hypothesised "score" of skills that would have a distribution that was clumped very much above the mean but had a very long distribution below it.

But that assumes that one can meaningfully "score" skills. It does pose the question of whether relying on user's self-assessment is appropriate versus actually testing those skills. A really useful study would involve testing the skills as well as seeking the self-assessment.

The report almost does this in its correlation of self assessed competency against the measures taken to protect against on-line risks. This is made somewhat difficult because of the reports measure of action taken. The Very much above average skills group had a relatively low level of having installed anti-virus programs, but they had a much higher level of "computer had filters/software installed when bought" which would typically include anti-virus. If we make that assumption there is overall the expected correlation between assessed skill level and level of protection.

The report itself errs by asserting that "eighty-one per cent of internet users are confidant in their skills". The error is that a user's belief in whether their skills are average or better does not equate to whether their skills are adequate or not.

The report's conclusion that high levels of self-assesed skills do not translate into taking proactive protection measures seems to be based on not including reliance on pre-installed software. Even were it true it is not sufficient evidence to support the reports major conclusion that

In order for consumers to continue to trust the internet and to ensure the growth of the digital economy, consumers need to be informed about online risks and ways to protect their computers and themselves from the more negative and harmful aspects associated with internet usage.

It is hard to determine how that conclusion can be reached on the basis of the evidence presented. This is compounded in the report's conclusion in stating "there is a critical role for industry and governments in the continuing improvement of consumer awareness".

In general the report is composed of relatively inconsequential statistics leading to disconnected conclusions that otherwise appeal to the interests of a regulator. Perhaps these weaknesses can explain why research conducted in June 2008 is being released in March 2009.

* Note however that this is one area where the current structuring of ACMA under an establising Act with three separate principle Acts to be administered creates some confusion. The question of how much policy analysis is meant to be done by ACMA as opposed to DBCDE is an interesting question.