Report Interpretation

How it works

Every time a user accesses something on the WebCT server, the server logs where they came from, the date and time, what they requested, and, if they are logged in, their WebCT ID. It also logs a few other things and can be configured to log user agents and referrers.

Tool Use reads through the server log file and for each request extracts the date and WebCT ID from the log file and maps the requested URL to a course and tool. The report is then generated from this data.

The analysed requests are actually further filtered in that they must have been accessible to WebCT students. Note that some designer and teaching assistant activity, such as reading discussions involves requests to student accessible URLs. The only way to generate a student only report is to identify students by their WebCT ID and filter on this. Tool use has run time options to filter by WebCT ID format. To filter by other criteria, the logs could be filtered first, or a custom routine added to tool use.

Thus Tool Use reports what is used or accessed in a course, rather than what has been added to a course. Some implications of this are discussed below.

What is presented

For each tool, use is expressed by three measures:

• the number of requests the server received for the tool
• the number of courses in which the tool was used
• the number of unique users who used the tool in any course

The measures are calculated for each week during the analysed period and for the period as a whole. They are then presented by the following:

• Summary:
  1. the measures versus week, summed over all tools
  2. the measures for each tool, summed over all weeks
  3. requests, courses and users by category/term, summed over all tools and weeks
  4. courses ordered by requests, users and requests per user, summed over all tools and weeks
• Weekly, all tools:
  5. for all tools, all measures versus week
  6. for all tools, one measure versus week
     (this is a subset of c)
• Weekly, one tool:
  7. for a given tool, each measure versus week
     (this is a subset of c)
• Week:
  
  8. for a given week, each measure for all tools

The numbers are given as both absolute values and as percentages. With the exception of the tables for each individual week (f), the percentages are expressed relative to the total for the period.

For example, you might find from (b) that 51% of courses used the discussion tool (as in the sample report). This means that over the whole period analysed, the discussion tool was accessed in 51% of courses that had any tool accessed over the whole analysis period. For an individual week, you might find that the discussion tool was accessed in 47% of courses (6-12 September in the sample report). This means that the discussion tool was accessed in 47% of the courses that had any tool accessed in that week. The percentage of the number of courses over the whole period would be given by multiplying by the % in the column header, i.e., 47% x 96% = 45% or by taking the absolute value and dividing by the total courses over the whole period 215 / 477 = 45%.

Things to watch

Tool identification

The identification of tools from the logs is refined with every release. Running future versions of tool use on the same logs as this version is likely to generate a slightly different report. However, general conclusions drawn from the reports are likely to be the same.

Interpreting requests

Care needs to be taken if using requests to make comparisons between tools. For example, a user going to discussions from the homepage, composing a message, previewing the message and posting it will generate four requests, and a user going to discussions, selecting a topic, opening a thread, reading a message and then closing the thread will generate five requests. Compared to this, a user going to a particular day on the calendar in the current month will generate two requests, and a user viewing the syllabus one request.

Two tools to be particularly careful about interpreting are chat and whiteboard. Activity within these is not logged in the web server log. All that is logged for chat is the access to the page from where the users access the rooms. To gauge the activity within chat, some sort of monitoring of the chat logs would be necessary. Also, Tool Use does not filter out designer activity in these tools (e.g., customising chat rooms and viewing logs). Similar caveats apply to interpreting use of the equation editor and Wimba.

Implications of "what is used"

Because the report is based on what is used or accessed in a course, rather than what has been added to a course, tools that have been added to a course might not be counted, and, conversely, tools that a designer has not added might be counted.

For example, if a course designer has added the glossary tool but students never access it, then the course will not be counted in the courses using the glossary tool. Similarly, if a designer has not added the mail tool to their course but a student accesses that course's mail tool directly via its URL, then that will be counted in the courses that are using the mail tool.

Miscellaneous

• Single URL
  This tool simply adds a link to a course. Therefore its use cannot be reported.
• Upload attachment
  This is not included in any of the measures.
• Index and Search
  Reported together.