Hello, I am Masakazu Imamura from Osaka Prefecture University, Japan.
The title of the talk is "Suitable camera and rotation navigation
for people with visual impairment
on looking for something using object detection technique."
This is a joint work with Yoshihiko Inoue, Kazunori Minatani and Koichi Kise.
One of the difficulties for people with visual impairment is
that they have a limited access to visual information.
For example, this person wants to drink a tea, but doesn't know which one is.
In such a case, the person can use smartphone apps which can tell what it is
like this.
Apps that tell visual information can be categorized into two.
Those in the first category can ask remote sighted people for help,
which include VisWiz and be my eyes.
Those in the second category use computer vision techniques,
which include EnVision AI, TapTapSee, and Seeing AI.
In this paper, we focus on the latter one.
While the apps are powerful and useful, they have a limitation.
That is, they assume the object is in front of the person.
For example, in this case, the person wants to recognize tea,
but actually the bottle of the tea is not in front of the person.
In such a case. The person needs to find the bottle.
However, the smartphone apps do not tell where it is.
Let us summarize the situation. If what is unknown and where is known,
which is named category (i),
a representive task is obtaining the visual information
on the object that the user photographs,
which can be realized by current smartphone apps.
And, if what is known and where is unknown, which is named Category (ii),
a representative task is looking for something.
And, a camera with a wide field of view,
such as omnidirectional camera would be useful.
For a person who is not familiar with omnidirectional camera,
this is a device, which usually have multiple cameras,
and this is an image taken by the omnidirectional camera.
By extending the idea, we can also consider Category (iii)
in which what is unknown and where is also unknown.
A representative task is finding something valuable and unexpected to the user.
In this case, we expect more visual information is required.
But, the information provided to the user should be selected.
In this paper, we focus on Category (ii).
Specifically, the task of looking for something.
Here is an overview of the research.
In the task of looking for something,
we implement a computer-vision-based prototype system
that guides the user to reach the target object.
We investigate whether an omnidirectional camera is
more suitable than the normal camera,
and we compare 5 rotation navigation methods,
including 3 novel voice navigation methods.
Here is the agenda.
We completed introduction and are going into method.
Let us present the prototype system
that consists of a laptop computer and a camera system.
The procedure of the prototype system is threefold.
When the person wants to look for his backpack,
in the first step, the prototype system finds a backpack.
In the second step, on the spot, the system navigates the user to rotate
so that the user is heading to the backpack.
In the third step, the system let the user advance toward the target object,
and stop in front of the object.
We mainly focus on Step 2.
There are some existing rotation navigation methods.
Ahmetovic et al. examine 1 + 3 sound navigation methods
in the context of turn-by-turn navigation.
Here is the list of some navigation methods they examined.
These three methods are also combined with the first one, Ping.
Let me briefly introduce the sound navigation methods.
First of all, Ping just let the user know when the target object is in front of the user.
For example,
like this.
The second one is intermittent sound (IS).
In this one,
something like this.
So, the interval is getting smaller, and in front of the object, the frequency is 15 Hz.
That's what I couldn't reproduce.
If it is 15 Hz, it should be something like pi pi pi pi pi pi.
The third one is amplitude modulation (AM).
In this one, a 440 Hz sinusoid is moderated by a sub-audio sinusoid
from 1 to 15 Hz.
It was also not easy to reproduce for me.
The fourth one is musical scale (MS).
In this one, the angle is subdivided into eight circular sectors,
and every time the user enters a sector, it plays the piano sound.
According to their experiment,
they concluded IS and MS combined with Ping were the best.
Here is the list of rotation navigation methods we examined,
which include 3 voice navigation and 2 sound navigation methods.
Voice navigation has not been used so far.
Let me present the rotation navigation methods one by one.
The first one is Left or right (LR),
which is very simple. It just tells left or right.
For example, if the person wants to look for his backpack,
"right, right, right," "in front of you."
Something like this.
The second one is Angle (AG).
This method tells the angle toward the target object, like this.
"60 degrees, right,"
"15 degrees right," "in front of you."
The third one is Clock position (CP).
It is similar to AG, but tells by using the clock position.
For example,
"2 o'clock,"
"in front of you."
The fourth one is intermittent beep (IB),
which is close to IS.
The difference from IS is that within 15 degrees towards the target object,
it plays sound like pi pi pi pi pi pi pi.
The fifth one is pitch (PT).
This is:
something like this.
Let's move on to user study.
The purposes of the user study is twofold.
One is comparison of 5 rotation navigation methods.
The other is selection of camera. We use 2 camera systems.
One is omnidirectional camera,
Which uses an omnidirectional camera.
The other is pseudo smartphone.
We assume that this is a smartphone,
but for the fair comparison, we use the web camera.
In addition, we use the same electronic compass for the rotation navigation,
and the same depth camera for the forward navigation, for fair comparison.
In the user study, 7 people with visual impairment participated.
We carried out 2 experiments which are about the rotation navigation and camera.
A survey, which is actually interview, for looking for something
was performed before the experiments.
But, because of time constraints of the presentation, I will omit this.
On the right hand side, this is a snapshot in the experiment.
He is the experimenter who has a laptop computer.
And, he is a participant who has the camera system.
This is the target object.
This is the result of the experiment 1, about rotation navigation.
In this experiment, a participant tried the 5 rotation navigation method one by one,
and told us their preference in a 5-point scale.
1 means the worst, and 5 means the best.
As you can see,
no one preferred LR because it does not tell anything about the rotation angle.
However, regarding the remaining 4, everybody has a different preference.
Every method was liked by at least one person.
Regarding the phenomenon, related results were reported.
In PerCom 2019,
Ahmetovic et al. reported that musical experiments affect the users' behavior.
In Web4All in 2019,
Ahmetovic et al. reported expertise affects interaction preferences
in navigation assistance.
In the interview after the experiment,
we found some interesting comments from the participants.
Let me introduce a representative one.
A participant told us his idea about the clock position navigation method.
In our method, the front of the user is always 12 o'clock,
and navigates the position of the target object.
However, this person wants to put the target object at 12 o'clock,
and wants to navigate the direction of the person.
This indicates that everybody has different ideas about the world and navigation.
This is the result of experiment 2, about the camera system.
As you can see, 6 participants out of 7 preferred omnidirectional camera.
Let me conclude the talk.
In this talk, we categorize the tasks of obtaining visual information into three.
We proposed a prototype system for looking for something
that used an omnidirectional camera and the use of voice in rotation navigation.
A user study comprised of 7 people with visual impairment confirmed that
(1) an omnidirectional camera is preferred.
(2) users have different preferences in rotation navigation.
Future work.
We found that the personalization of the prototype system is very important.
For example, a participant wanted to find his or her own train pass,
not the general one.
This means that we need to personalize the object detector.
Also we need to personalize the navigation method.
From the interview,
we also found that the existence of the object is
far more important than navigation of the direction,
and also navigation of the distance.
If it takes time to detect the object or failed in detection,
the user thinks the object does not exist in the room.
So, we need to clarify the existence of the object.
That's it. Thank you very much for your attention.