Version 1.0 29/01/2009 Nilo Menezes - Multitel ASBL (nilo.menezes@multitel.be)
Previous versions:
Version 0.9 06/08/2008 Nilo Menezes - Multitel ASBL (nilo.menezes@multitel.be)
The Multimodal Middleware Protocol is an easy way to interconnect components, abstracting network complexity and enabling fast experimentation. It was developed to solve a common problem when working with Multimodality: how to integrate components written in different programming languages and doesn’t know about each other?
The Multimodal Middleware Protocol was developed in OpenInterface Project, more details on “About OpenInterface” below. The Multimodal Hub was developed at Multitel ASBL Belgium. During the OpenInterface project, we adapted the Multimodal Hub to use the new protocol and to use a graphical interface called Multimodal Browser. The Python Multimodal Hub project aims to provide an open source implementation of the concepts developed during the project and to foster the creation of new components and implementations.
This document introduces component writing techniques in Python.
This document was written for programmers willing to use the Python Multimodal Hub as a foundation for their new Multimodal Devices.
Multimodal interaction is challenging to develop especially in mobile/ubiquitous settings with novel interaction devices.
Addressing this challenge, the FP6-35182 OpenInterface project (http://www.oi-project.org) focuses on the design and development of an open source platform for the rapid development of multimodal interactive systems as a central tool for an iterative user-centered design process.
The OpenInterface Interaction Development Environment (OIDE) includes development tools for assembling, combining and testing modalities at multiple levels of abstraction (from the raw data to the semantic of users’ events).
The OIDE is built on top of a runtime kernel which enables the communication between heterogeneous components. The OpenInterface project considers two interaction contexts, namely a mobile and an ubiquitous setting, and two testbeds, namely multimodal interaction for large information spaces (e.g. maps), and multimodal interaction for games. The two testbeds are used to explore the functionality of the platform.
The OpenInterface project includes 10 European partners, five academic research institutes and five industrial partners.
This work was done with the support of the Walloon Region and of the European Fund for the Regional Development.
In this tutorial, we will create two simple components. These components will use the UDP protocol to communicate with the Multimodal Hub.
The first component will be called NumberGenerator. The NumberGenerator will be responsible for sending messages to the Multimodal Hub with a new number in each of them.
The second component will be the NumberPrinter. The NumberPrinter will simply print the messages received from the Hub on the screen.
+=================+ +================+ +===============+
| NumberGenerator |==>| Multimodal Hub |==>| NumberPrinter |
+=================+ +================+ +===============+
The goal will be to generate a event with a new number using the NumberGenerator, send it to the Multimodal Hub and then receive this message using the NumberPrinter.
We will learn how to create new components, associate a network server to them, connect to the Multimodal Hub, generate and consume events.
You need to have a running version of Python 2.5 and the Multimodal Middleware Protocol package installed. You can download Python in http://www.python.org/download/.
The installation packages and the source code of the Multimodal Middleware Protocol can be found at http://sourceforge.net/project/showfiles.php?group_id=216180.
Let’s create our first component!
First create a new file, NumberGenerator.py. We will need to import some packages, so copy the lines below to the new file:
from MultimodalMiddlewareProtocol.Device import *
import MultimodalMiddlewareProtocol.Fields
import MultimodalMiddlewareProtocol.FieldFactory
from MultimodalMiddlewareProtocol.servers.UDPServer import *
import time
Then, we should add some code to start our component from the command line:
if __name__ == '__main__':
startGenerator()
We need to define the function startGenerator. Let’s do this step by step:
def startGenerator():
client = UDPServer("", 0)
The UDPServer class creates a network server, using the UDP protocol. We pass an empty string as its first parameter, because we want the operating system to bind all IP addresses of this computer to the newly created server. The 0 means we don’t have a specific network port for this component, letting the operating system allocate a free port for us.
client.setMultimodalHubAddress(("127.0.0.1",6000))
We set the MultimodalHubAddress to the same computer we are running this component, but we can use any other IP, since we have a hub running there. The port 6000 is the default port number for the Multimodal Hub.
client.start()
With this line, we activate the UDPServer. It will bind to a port and start waiting for network messages. But until this point, we don’t have a MultimodalDevice, not yet. Let’s create ours:
device = MultimodalDevice("NumberGenerator", client, 1L)
We create a new MultimodalDevice object, assigning NumberGenerator as its name and passing client as our previously created network server. We called it client because if we look to the Hub as a network server, our components will be its clients. The last parameter is the number 1L. The L is just a type hint to help the Python interpreter to understand this as a number of type long. This number is the DeviceID of the new device. It will be used when registering this device with the hub. You need to visit the component repository to get an unique number.
device.produces.append(0xE0000001L)
This line tells our device to announce it can produce events with id 0xE0000001L. This is very important because the Multimodal Hub uses this information to know what events your device can provide. Remember to visit the component repository to register unique numbers for your custom events. In this tutorial we will use 0xE0000001L that is already registered as an event generated by the NumberGenerator sample component.
device.register()
With this call, we ask the device to register itself with the Multimodal Hub. You don’t need to care about how the Multimodal Device does this, but it can only do this with your help. That’s why we need to create a network server first, set the Multimodal Hub address and start our server to get ready for the component registration.
This new device needs to generate its own events. Let’s do a simple program to generate a new event every 0.020 seconds.
i = 0
record = FieldFactory.fieldFromCodeList("LLLL")
The FieldFactory module contains utility functions to help us to create our messages. In this case, we created a variable named record with the help of fieldFromCodeList function. It takes “LLLL” as a sequence of four Long fields to create and returns a list with these fields already instantieted. You can find all Field codes like “L” for Long in the Consts module.
We will use record to fill in the values of our message:
while(1):
record[0].set(device.UCID)
record[1].set(device.deviceID)
record[2].set(0xE0000001L) # EventID
record[3].set(i) # generated number
print i
i+=1
As record is a list of Field objects, we can call the method set to assign the right value to each field.
As specified in the protocol, the first field, at position zero (record[0]), contains the UCID. We will use device.UCID because it contains the UCID already assigned by the Multimodal Hub during the component registration.
The second position contains the deviceID, the third the event ID and the fourth the number being generated. Remember indices start at 0, so the fourth field is at position 3 (record[3]).
Than we printed the number on the screen just before incrementing it.
Now that we have filled in the right values to each field, we need to send this message to the Multimodal Hub. We do this in the following line:
device.server.sendFields(None, record)
We pass None in the first parameter because we want to send the fields in the record variable to the Multimodal Hub. The sendFields method will write each field in the binary format requested by the protocol, so you don’t need to care about this step. Everything you need to supply is a list of Field objects with the correct values.
The complete source code should look like this:
from MultimodalMiddlewareProtocol.Device import *
import MultimodalMiddlewareProtocol.Fields
import MultimodalMiddlewareProtocol.FieldFactory
from MultimodalMiddlewareProtocol.servers.UDPServer import *
import time
def startGenerator():
client = UDPServer("", 0)
client.setMultimodalHubAddress(("127.0.0.1",6000))
client.start()
device = MultimodalDevice("NumberGenerator", client, 1L)
device.produces.append(0xE0000001L)
device.register()
i = 0
record = FieldFactory.fieldFromCodeList("LLLL")
while(1):
record[0].set(device.UCID)
record[1].set(device.deviceID)
record[2].set(0xE0000001L) # EventID
record[3].set(i) # generated number
print i
i+=1
device.server.sendFields(None, record)
time.sleep(0.020)
if __name__ == '__main__':
startGenerator()
With this code, we have what is called a producer component in the Multimodal Middleware Protocol Specification. We say that the NumberGenerator produces messages to the Multimodal Hub.
The number printer component is very similar to the NumberGenerator. It also needs to have an unique DeviceID, to create his own server and to register with the Multimodal Hub.
The main difference is that we need to register our message processing routines in a different way. In the NumberGenerator, we send messages in the main loop. In the NumberPrinter, the main loop needs to wait for new messages and we need to inform our new device how to handle these messages. The NumberPrinter will then consume the message events generated by the Number Generator.
Create a new file with the code below:
from MultimodalMiddlewareProtocol.Device import MultimodalDevice
from MultimodalMiddlewareProtocol import Fields
from MultimodalMiddlewareProtocol import FieldFactory
from MultimodalMiddlewareProtocol.servers.UDPServer import UDPServer
if __name__ == '__main__':
startPrinter()
raw_input()
We need to define our startPrinter function:
def startPrinter():
client = UDPServer("", 0)
client.setMultimodalHubAddress(("127.0.0.1",6000))
client.start()
device = MultimodalDevice("NumberPrinter", client, 2L)
It really looks like the beginning of the startGenerator function of our previous component. But we can see we are using a different name for our new device and a new DeviceID. You can see this difference in the first(“NumberPrinter”) and last(2L) value of the MultimodalDevice call.
Now, let’s see something really different:
device.eventProcessors[0xE0000001L] = printNumber
With this line, we add event ID 0xE0000001L to our processed messages list, but we do even more than that: we assign printNumber, another function, to be called every time we receive a message with this event ID. We will define this function later.
device.consumes.append(0xE0000001L)
We add the event ID 0xE0000001 to the list of events our device consumes. It means we are telling the MultimodalHub that we know how to process messages with this ID and that messages can be forwarded to our component.
To complete our startPrinter function, we need to ask the device to register itself with the MultimodalHub, as we did before:
device.register()
But, we still need to define the printNumber function. Let’s do it now:
def printNumber(command):
record = FieldFactory.fieldFromCodeList("LLLL")
FieldFactory.readFields(record, command.dataIn)
print record[3].get()
The printNumber function is defined with a single parameter: command. The command is always an CommandHeader object. It contains all needed fields to process messages, like the message event ID, device ID and the originating component UCID. It also includes the raw data of the message and the network server that received it.
In the first line of printNumber, we created a list of fields (four long fields) to decode our message. Than we used the readFields function of FieldFactory to read all fields from the stream command.dataIn. After this decoding period, we have all fields easily accessible in the record array. In our printNumber function, we just print the fourth field.
Here you have the complete source code:
from MultimodalMiddlewareProtocol.Device import MultimodalDevice
from MultimodalMiddlewareProtocol import Fields
from MultimodalMiddlewareProtocol import FieldFactory
from MultimodalMiddlewareProtocol.servers.UDPServer import UDPServer
def printNumber(command):
record = FieldFactory.fieldFromCodeList("LLLL")
FieldFactory.readFields(record, command.dataIn)
print record[3].get()
def startPrinter():
client = UDPServer("", 0)
client.setMultimodalHubAddress(("127.0.0.1",6000))
print "Starting"
client.start()
device = MultimodalDevice("NumberPrinter", client, 2L)
device.eventProcessors[0xE0000001L] = printNumber
device.consumes.append(0xE0000001L)
print "Registering"
device.register()
print "Started"
if __name__ == '__main__':
startPrinter()
raw_input()
The raw_input in the last line was used to keep the NumberPrinter running until we press a key. In the NumberGenerator this wasn’t need because the main loop of the program was busy generating messages.
We can test our new components following a simple sequence:
Although the order we start our components doesn’t matter to the Multimodal Hub, we must keep in mind to always start the Multimodal Hub first. If you forget to start the Multimodal Hub as the first component, you will have to close all other components and then restart them again.
If everything was ok, you should start receiving the numbers generated in the NumberPrinter component. The TkBrowser should print two connected devices and his message counters should go crazy.