WikiDo builds a database of computer task automations based on crowd-sourced help from a community of Internet users. Users contribute by simply performing a set of instructions which are translated from a natural language (English) to a sequence of GUI actions. For example, the sentence “click on OK” will be translated to a GUI command LEFT_CLICK and an object BUTTON: OK on which to perform that command. As the commands are performed, the system aggregates traces from multiple users into a canonical sequence of GUI actions that, when executed on a user machine, can automatically perform the corresponding task. When a user searches the WikiDo database for a task they do not know how to perform, they can either read a text version of the GUI actions, use the solution as a tutorial that will walk them through how to perform the task step by step, or allow the solution to run automatically on their computer to perform the desired task.
WikiDo achieves high-accuracy annotations for documents by merging multiple action sequences to filter out idiosyncrasies and mistakes of individual solutions. Additionally, a classifier is used to predict which steps are likely to be misinterpreted and requests human intervention to properly perform them. This process can be done iteratively until the translation is believed to be correct.