[Raphael,Adrien,Benjamin]
18/05/2015

We finally succeed to test our code on the Nao robot, unfortunattely some errors appeared. We asked Titouan to help us in order to reprogram an « if » box. We didn’t find out what the real problem was.

19/05/2015

We found the solution to our problem, we reconfigured the « random int » box which returns a random number between 1 and 10. The robot will asked us 4 questions and then will stop. When we answer « NO » to the question « Would you like to play ? » , the robot will sit down and be quiet. If we wait 10s and say « Ok let’s play » to it, it will answer « Cool ! », get up and start to ask the questions.

[Yani]

Today, we tried to find out how to send movements request to the Nao robot through the computer terminal. As last week we could send “say” request but didn’t succeed in sending “motion” request. We are not sure if we can send direct request for motion through terminal. We found online some code in which the developers seem to create more or less their own keywords and syntax that implements the Nao movements. We’ve spend a lot of time trying to understand the code and building on top it.

Firstly we could succeed in sending Say requests using the code found online. Just before the end of the work session we understood a little bit more about how are implemented the boxes found online. But we are not completely sure that we use it correctly and we are trying many different manipulations with. Here is the Boxes we found online:

The next step is to use the motions keywords.

[2] The last time, we developed a way to analyze movement on our computer. Thus the next logical step is to put that code on the NAO in order to process image from its own camera.

After struggling a bit on how to use the NAO camera service, we were soon able to fetch a frame from the camera and display its characteristics (width, height). The working code in the form of a Chrregraph box is available here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

#
# Cette box est une box basique permettant de récupérer une image
# de la caméra de NAO (grâve au service ALVideoDevice)
#

class MyClass(GeneratedClass):
    def __init__(self):
        GeneratedClass.__init__(self)

        self.avd = None
        self.strMyClientName = None

    def onLoad(self):
        self.connectToCamera()
        self.getImageFromCamera()
        self.disconnectFromCamera()

    def onUnload(self):
        self.avd = None
        self.strMyClientName = None

    def onInput_onStart(self):
        #self.onStopped() #activate the output of the box
        pass

    def onInput_onStop(self):
        self.onUnload() #it is recommended to reuse the clean-up as the box is stopped
        self.onStopped() #activate the output of the box


    def connectToCamera(self):
        self.log("STARTNAO: connecting to camera...")

        try:
            self.avd = ALProxy("ALVideoDevice")

            self.strMyClientName = self.avd.subscribeCamera(
                # Name              # CameraNum     # Resolution    # Colorspace        # FPS
                self.getName(),     0,              1,              11,                 5
            )

            self.log("STARTNAO: connected to camera")
        except BaseException, err:
            self.log("STARTNAO: error connecting to camera: %s" % err)


    def disconnectFromCamera(self):
        self.log("STARTNAO: disconnecting from camera...")

        try:
            self.avd.unsubscribe( self.strMyClientName )
        except BaseException, err:
            self.log("STARTNAO: error disconnecting from camera: %s" % err)

        self.log("STARTNAO: disconnected from camera")


    def getImageFromCamera(self):
        self.log("STARTNAO: getting camera image...")

        try:
            dataImage = self.avd.getImageRemote(self.strMyClientName)

            self.log("STARTNAO: camera image is null: %s" % dataImage is None)

            if dataImage is not None:
                self.log("STARTNAO: camera image property [0]: %s" % dataImage[0])
                self.log("STARTNAO: camera image property [1]: %s" % dataImage[1])
                self.log("STARTNAO: camera image property [2]: %s" % dataImage[2])

        except BaseException, err:
            self.log("STARTNAO: error getting camera image: %s" % err)

        self.log("STARTNAO: camera image got")

        return None

This box is the basis we are currently using to implement the algorithm we tested on our computer. It uses the log system of Choregraph to display frame properties to test the camera.

[4] We are trying to detect red balls on the Nao. First we did it on our computer, here is the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#!/usr/bin/env python 
import cv2
import numpy as np

#cap = cv2.VideoCapture(0)
c = cv2.VideoCapture(0)
detected = False
while(detected is not True):

    # Take each frame
    _, img = c.read()
    img = cv2.flip(img,5)

    # Convert BGR to HSV
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

    #cv2.imshow("b", hsv)
    # define range of blue color in HSV
    lower_color = np.array([110,50,50], dtype=np.uint8)
    upper_color = np.array([130,255,255], dtype=np.uint8)

    # Threshold the HSV image to get only blue colors
    mask = cv2.inRange(hsv, lower_color, upper_color)
    #cv2.imshow('mask',mask)  
    # Bitwise-AND mask and original image
    res = cv2.bitwise_and(img,img, mask= mask)
    #cv2.imshow("b", res)

    imgray = cv2.cvtColor(res,cv2.COLOR_BGR2GRAY)
    ret,thresh = cv2.threshold(imgray,127,255,0)
    contours, hierarchy = cv2.findContours(imgray,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
    

    #erode = cv2.erode(res,None,iterations = 3)
    #dilate = cv2.dilate(erode,None,iterations = 10)
    #contours,hierarchy = cv2.findContours(res,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
    for cnt in contours:

        x,y,w,h = cv2.boundingRect(cnt)
        print w,h
        if(w < 60 or h < 60):
            continue

        cx,cy = x+w/2, y+h/2

        if 20 < res.item(cy,cx,0) < 30:
             #cv2.rectangle(f,(x,y),(x+w,y+h),[0,255,255],2)
            print "yellow :", x,y,w,h
        elif 100 < res.item(cy,cx,0) < 120:
             cv2.rectangle(img,(x,y),(x+w,y+h),[255,0,0],2)
             print "blue :", x,y,w,h
             detected = True
        cv2.imshow('img',img)  

    k = cv2.waitKey(5) & 0xFF
    if k == 27:
        break

cv2.destroyAllWindows()

On the computer, this code work fine, but on the Nao, it seems that the image processing is too slow. Our project manager contacted Aldebaran to know more about this issue.
They recommended us to use Nao library to import the camera image insted of using cv2 library.

We are know trying to adapt our code to these guidelines to make all that work.

[2] In the group II, we are currently working with OpenCV in order to detect motion, quantify it and execute actions under certain conditions. For the moment, the Python code works well on our computers (it detects the motion accordingly to what we wanted). The next step is to put that code on the NAO to make it detect what we want.

This article aims to explain the process we followed to understand how OpenCV work and to detect an amout of motion in an image.

I. Discover OpenCV

We discovered the basics of OpenCV by speaking with another group who use it to detect colors. In a few minutes, we were able to find a code on Internet that would read the input of the computer camera and display it in a window.

This code let us understand how OpenCV work a little bit:

• It works with frames: each frame is an image we can manipulate and compare with others ;
• It brings natively a lot of image processing tools that we can use to achieve what we want ;
• It can detect by itself the movement between two images ;

By searching a bit on Internet, we were able to build a movement tracker that would create red rectangles around what move in real time.

II. Quantify the amount of movement

Once we were able to detect movement, the next issue was to quantify that movement in order to trigger or not our action.

a. First try: percentage of the image in movement

Our first implemetation used a percentage of the image covered by red rectangles: if more than 30% of the image was in movement, we trigger the actions. However, this method wasn’t great for two main reasons:

• We didn’t have a notion of time, so a single big movement could be enough to trigger the system ;
• Every single movement, even the smallest ones like a face movement, were tracked, so the percentage was pretty bad ;

b. Improve the method by adding time and ignoring some rectangles

In a second and third implementations, we worked on these problems and solved them by adding a time notion and by ignoring small rectangles. Now, the motion had to be detected for a certain number of frame to trigger the event.

But we faced a last problem when we tested the code: if someone moved his or her hand very closely to the camera, the system would be triggered even if the motion is small. That is not what we want, so we tried to improve our algorithm.

III. The motion distance

The best thing we could do at this step was to test and experiment. After a while, we discovered something interesting: if the motion is far enough, a single big red rectangle is created instead of a lot of small ones. Great!

Thus we chose to change our algorithm: instead of ignoring too small rectangles, we will consider only the biggest one for the percentage. After some tests, we were proud: our system works pretty well!

IV. The final code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

#!/usr/bin/env python

import cv

class Target:

    def __init__(self):
        self.capture = cv.CaptureFromCAM(0)

    def run(self):
        maxSize = cv.GetCaptureProperty(self.capture, cv.CV_CAP_PROP_FRAME_WIDTH) * cv.GetCaptureProperty(self.capture, cv.CV_CAP_PROP_FRAME_HEIGHT)

        # Capture first frame to get size
        frame = cv.QueryFrame(self.capture)
        grey_image = cv.CreateImage(cv.GetSize(frame), cv.IPL_DEPTH_8U, 1)
        moving_average = cv.CreateImage(cv.GetSize(frame), cv.IPL_DEPTH_32F, 3)

        first = True

        counting = False
        movement_count = 0

        while True:
            color_image = cv.QueryFrame(self.capture)

            # Smooth to get rid of false positives
            cv.Smooth(color_image, color_image, cv.CV_GAUSSIAN, 3, 0)

            if first:
                difference = cv.CloneImage(color_image)
                temp = cv.CloneImage(color_image)
                cv.ConvertScale(color_image, moving_average, 1.0, 0.0)
                first = False
            else:
                cv.RunningAvg(color_image, moving_average, 0.020, None)

            # Convert the scale of the moving average.
            cv.ConvertScale(moving_average, temp, 1.0, 0.0)

            # Minus the current frame from the moving average.
            cv.AbsDiff(color_image, temp, difference)

            # Convert the image to grayscale.
            cv.CvtColor(difference, grey_image, cv.CV_RGB2GRAY)

            # Convert the image to black and white.
            cv.Threshold(grey_image, grey_image, 70, 255, cv.CV_THRESH_BINARY)

            # Dilate and erode to get people blobs
            cv.Dilate(grey_image, grey_image, None, 18)
            cv.Erode(grey_image, grey_image, None, 10)

            storage = cv.CreateMemStorage(0)
            contour = cv.FindContours(grey_image, storage, cv.CV_RETR_CCOMP, cv.CV_CHAIN_APPROX_SIMPLE)
            points = []

            max_percentage = 0

            while contour:
                bound_rect = cv.BoundingRect(list(contour))
                contour = contour.h_next()

                pt1 = (bound_rect[0], bound_rect[1])
                pt2 = (bound_rect[0] + bound_rect[2], bound_rect[1] + bound_rect[3])

                area = bound_rect[3] * bound_rect[2]
                percentage = (area / maxSize) * 100

                if percentage > max_percentage:
                    max_percentage = percentage

                points.append(pt1)
                points.append(pt2)
                cv.Rectangle(color_image, pt1, pt2, cv.CV_RGB(255,0,0), 1)

            if not counting and max_percentage > 75:
                movement_count = 0
                counting = True
                print "Demarrage comptage"
            elif counting and max_percentage > 50:
                movement_count += 1
                print "Incrementation"
            elif max_percentage < 20:
                movement_count = 0
                counting = False
                print "Fin comptage"

            if movement_count > 50:
                print "OK"

if __name__=="__main__":
    t = Target()
    t.run()

[1] This week, we have found a script opening a communication socket on Nao. With this one, we can send commands, having a particular syntax, to Nao, witch will analyse and execute the action. It uses the TCP protocol, on port 8080, and so can be easily used by a PHP script.

Moreover, the usage is very simple : Choregraphe, which has to be install on the web server machine, executes the script once. There, it constantly listen to the port 8080, and treat every message it recieved.

We tried to implement a way for the Nao to remember the riddles it has already asked but without an actual Nao on which to try, we don’t know if it work. For that we have written a small box that combine the “if box” and the “get data” box. We’ll try on a robot next week !

Today we started by trying to see more clearly through the functions proposed at this link : http://www.lucubratory.eu/independent-socket-server-on-nao/. We are trying to see where does the error came from and how to correct it if we can. So we changed the “port” argument in the function into a chosen number (8080), we could get send packets from computer to the Nao with no loss. This means that the Nao and the computer are actually connected, but we can’t get any authentication from the Nao.
Thanks to this http://dhrc.snu.ac.kr/nao-robot-programming-guide-creating-a-tcpip-module-using-motion-capture-part-1/, we succeeded in creating a communication socket on Nao in order to communicate with him though a web server machine.
Here is a picture of the Choregraphe’s boxes setting:

The socket is created and the Nao will listen on port 8080 for any incoming request. We will show the code of the box later. Basically now we can communicate with Nao from the computer, we can send him some sentences to say. So the connection is done but we still have some issues:

• For now we still need to manually run Choregraphe in order to run the server on the Nao, but we will try to write a PHP script that would automatically run it.
• We are still learning the syntax of the request to send to Nao, we could make him speak but for now we couldn’t make him move.
• Also we’ve set the Nao’s right foot button as an interrupter to cut the connection.

[1] During the last project session we finally decided after a lot debates to give up on trin to implement a whistle detector in the Nao as it really feels difficult and we are scared to remain stuck on this until the end and get no results.

Instead we are going to build the web interface for Nao, the concept is to great an interface from which we could perform simple action with the Nao, like getting up, sitting, tai-chi … So we first started to search for information on how to implement a server with the Nao, we found some tutorials that unfortunatelly required some administrator access to the Nao that we don’t have, but we’re seeing with the CCRI if we could get it or not.

We tried to communicate with Nao by using TCP protocol. Actually, we managed to connect to his port, but there, no action was possible. So we tried to start a web server on the port 8080, using the same library that it uses in order to create the admin interface (port 80). However, we hadn’t the permissions in order to perform this action.

Group 1

Firstly, we’ve move forward on creating customized boxes using Nao’s implemented functions which are described in the thechnical documentation. For example we’ve made a box that allow the Nao robots to record a sound and play it.

Furthermore we’ve find out that we can use something called “La transformation de Fourier” and implement it as a function so the Nao can isolate specific sound frequency in a record. That’s probably how we can detect whistles but research is still on going. The part of making the Nao detect someone yelling is considered as done. Indeed a box call “soundpeak” allow us to do that. Still the Nao won’t make the difference between a normal sound coming from a close source and a loud sound coming from a distance source but this not really a problem for what we are doing.

Group 2 :

Nao need to perform some actions/movement when we are doing some large movements in front of the camera. If he cannot recognize these movements, he does a movement with his arms like protecting himself(he puts his arms in front of his face).. Which consist of not doing the good movement. And After downloading opencv, We have successfully managed to use the motion detector.

Group 3 :

Nao starts asking if the person wants to play. In case of a negative answer, he sits down, makes his head going down, turns his eyes to red and stays like that. When the player replies positively, he asks a riddle.
A random question is taken in a list of 10, and NAO detects if the given answer is the correct one. He continues and takes another question, different from the old one.

Group 4:

In our team, we are trying to make Nao recognize red and blue balls. For this, we use the open source library OpenCV for the Nao.
We managed to analyze pictures with OpenCV and locate blue and red.
Today, we did not manage to use well OpenCV with the Nao.

Welcome to our class project blog !

We are currently working on a project involving Nao. The goal is to find new ways to interact with the robot.

We invite you to look at the project definition page to learn more about the project.