Initial Planning and Coordination

Project Description

• We are aiming to create a system by which a drone can autonomously fly in an indoor environment and deliver packages between specified locations.
• Though this has essentially already been done by others, this project could potentially help implement this system in ElRo and further the practical knowledge and experience of our group members.

Communication

• Team: Owen G., Theo C. We communicate by texting and google docs.
• Group: Our group is fairly independent. Though the drone take-down teams are related to ours in that we share the same equipment, we shouldn’t really need to communicate very much (if needed, email should suffice).

Tech Analysis

• Navigate Linux
• Run AR Drone SDK examples
• Edit AR Drone SDK examples
• Navigate AR Drone SDK
• Use OpenCV
• Incorporate OpenCV in AR Drone SDK

Competence

• What we already know:
• Minor experience in C
• What we don’t already know:
• Experience in Linux
• Experience with the AR Drone
• Experience with the SDK
• Running examples
• Editing examples
• Experience with OpenCV
• Further knowledge of C programming

Safety

• Flying the drone offers a safety hazard; we need open, unoccupied spaces to test the flight of the drone.

Equipment, Materials, Budget

• Parrot AR Drone (already purchased)
• AR Drone SDK (free)
• OpenCV (free)
• Computer with Linux OS (free)

Schedule

• This week we want to focus on having everything working as it was when the previous group finished. We want to run the SDK examples sdk_demo and navigation, as well as run the OpenCV application created by last year’s group on both of our computers. If we run into any problems, we want to resolve them by the end of this week.
• We also, at the same time, want to work on our gantt chart. However, if it was not intended that we jump into the project yet, we can just try to finish the gantt chart before starting.
• The first task will be done by both of us, and the second (gantt chart) will be edited by both of us, but I will share the gantt chart.

Resources:

— On simultaneous localization and mapping:
http://www.nickd.nl/dl/thesis_Nick_Dijkshoorn.pdf 
— On using openCV within the SDK to stream video from the drone to a computer: 
http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux
— On using the special ROS to code the drone:
http://robohub.org/up-and-flying-with-the-ar-drone-and-ros-getting-started/
— A tutorial for programming the AR.Drone (communication syntax):
http://www.robotappstore.com/Knowledge-Base/Programming-ARDrone/101.html
— More info on the drone-computer communication
https://www.objc.io/issues/8-quadcopter/communicating-with-the-quadcopter/
https://abstract.cs.washington.edu/~shwetak/classes/ee472/assignments/lab4/drone_api.pdf
— Processing to communicate using UDP in oscP5 library
http://www.sojamo.de/libraries/oscp5/
— Processing to communicate using AR Drone library
http://kougaku-navi.net/ARDroneForP5/index_en.html
— Float to binary conversion
http://kipirvine.com/asm/workbook/floating_tut.htm
— SDK Understanding
http://gauth.fr/2011/09/introduction-to-the-ar-drone-sdk/
http://www.warp1337.com/content/ardrone-20-install-sdk-and-fly-2-minutes-ubuntu-linux-1210
http://www.ardrone-flyers.com/forum/viewtopic.php?t=83

Resource from Last Year

Team Progress Report Blog: http://advstem1.blogspot.com/
Project Resource including Gantt Chart: http://stem14-15.blogspot.com/2015/02/project-resource-drone-delivery-system.html

Missing Initial Planning and Coordination

Your team has missed the deadline of posting "Initial Planning and Coordination". Please make it up ASAP.

More Notes: Process Ideas

Method

• The client gives a compiled list of commands to follow. The commands could be directions.
• directions are of two types (each corresponding to yaw changes):
• Left   (Ø°)
• Right (Ø°)
• at each corner is a tag (preliminarily a color, later a unique tag to match)
• the direction indicates which tag to seek next
• TEST: wall tags vs. floor tags?
• wall tags are simpler to find (only frontal camera)
• floor tags are more adaptable (not every corner ends with a wall)

AR.Drone Programming Protocols

• In case the packet contains more than one command, the new line byte \r 0x0A is used to separate the commands.
• Strings are encoded as 8-bit ASCII characters
• The maximum length of the command is 1024 characters
• 30 MS delay must be passed between the commands
• Commands should be no more than 2 seconds apart (continuous commands is better)
• The client must use the host AT command port to communicate 
• See developer guide, sec.6 for more on AT command syntax
• The IP address of the drone is 192.168.1.1 and there are three ports we can use to connect over UDP:
• Navigation Data Port = 5554
• On-Board Video Port = 5555
• AT Command Port = 5556

• AT*FTRIM = #<LF>                                                                
• Command for horizontal calibration (flat trim)
• #=1 
• LF = return character
• AT*REF = #,0001000101010100000000†§00000000<LF>   
• Argument 1 is sent as an integer (written here is its derivative binary form).
• §(1=change)(0=stay|stopEmergency)
• †(1=upUntilLifted)(0=downUntilLanded) 
• AT*PCMD = #,€,R,P,G,Y<LF>
• €(1=readCommands)(0=hover)
• R(-1,1 = roll)
• P(-1,1 = pitch)
• G(-1,1 = gaz)
• Y(-1,1 = yaw)
• -1,1 = float corresponds to %age of preset power range, in the format of its equivalent integer.
• AT*LED = #,@,F,D<LF>
• @ = predefined animation tag number (int 0-20)
• F = frequency (delay between commands)
• D = duration (number of times)

Resources

— A tutorial for programming the AR.Drone (communication syntax):

http://www.robotappstore.com/Knowledge-Base/Programming-ARDrone/101.html

— More info on the drone-computer communication

https://www.objc.io/issues/8-quadcopter/communicating-with-the-quadcopter/

https://abstract.cs.washington.edu/~shwetak/classes/ee472/assignments/lab4/drone_api.pdf

— Processing to communicate using UDP in oscP5 library (I would prefer something like
     this, as I am familiar with Processing)

http://www.sojamo.de/libraries/oscp5/

— Processing to communicate using created AR Drone library (if previous resource's option
    doesn't work)

http://kougaku-navi.net/ARDroneForP5/index_en.html

— Float to binary conversion (for commands, float > binary > integer is a necessary conversion)

http://kipirvine.com/asm/workbook/floating_tut.htm

To Research Next

• Receiving drone's navdata
• Receiving drone's video
• Image matching
• Color-searching

Research Task 2 (08/03/15 - 08/16/15)

Feedback on Two Methods

      Using third-party tool to create applications

1. You can refer to the "Third-Party Tools" in the project resource page, and find several tools you can use. Some of them are no longer actively maintained.
2. Last year students have used AutoFlight (running on Windows PC) form LBPC Labs to fly drones and write autonomous flight routines using its Python-like script language, called AutoScript. Students also use its function to acquire ultrasound sensor data to perform autonomous landing. (You can find the example at 3:21 of City of STEM video.) However, based on students' feedback, several flight control functions are not reliable/working. Also, the documentation of using OpenCV with AutoFlight is missing.
3. Last year students have also tried to use Nodecopter to program the motion. It's a newly developed tool and started getting popular. However, they encountered some problems and didn't get through it.

      Create Software from Scratch

1.  There are many existing AR Drone-based research projects/capstone projects, so, practically speaking, we don't need to start everything from scratch. Since Linux is probably the most popular platform in academia, many of the  projects are Linux-based.
2. Last year students followed a dissertation A.R Drone Vision-Guided Searching (2013) form Derek Long, and coded the computer vision part of the autonomous drone navigation. (Some results can be found at 4:12 of City of STEM video.) The goal is to use a series of simple circular visual marks to guide the drone. They have acquired the drone video, adapted it to OpenCV format, transformed it to HSV color space, converted it to binary by thresholding. Then, based on the binary video, they calculated the size and center of mass of the white pixels, and use them to estimate the distance and direction of the visual marks. What they haven't got time to finish is marry the computer vision part with the motion control part to make the drone navigate autonomously by following the marks.
3. As far as the path planning, the drone should be able to navigate from any room to any other room on the second floor autonomously (whether starting from the middle of the room or just at the door are both ok). In other word, some UI for users to assign the navigation tasks is required.

Comments on What Can You Do

      Four Stages of the Project

1. Stage I: As you suggested, after testing and calibrating all the relevant sensors, program the drone motion based on the sensor data (Linux preferred). 
2. Stage II: If general sensors are not good enough, you can follow up and continue the work left from last year to do vision-based navigation (more precisely, visual mark-based navigation). 
3. Stage III: After having some experience on vision-based navigation, we can let go the visual marks. Instead, use pre-stored visual features to guide the drone navigation. Those features can be acquired and created in the training phase (we will hold the drone, walk through the hallway, and capture the crucial scenes for navigation). When drone fly by itself, the program will process the real-time video and find matches to the stored features. Based on the matching information, the drone knows where it is and navigate toward the destination accordingly.     
4.  Stage IV: If you can reach stage III, and still have time, you can try to implement the whole SLAM (much more advanced math involved). We don't even need to manually train the drone. The drone will map the unknown environment and locate itself accordingly. Based on the map created by the done, we can ask drone to go wherever we want.

Tasks

      If you agree with the picture I just painted, you can start studying/coding/experimenting:

1. Sensors: Study the information about all the relevant sensors including accelerometers, gyroscopes, magnetometers, ultrasound sensors, pressure sensor, etc. in AR Drone Developer Guide.
2. AR Drone Linux SDK
   1) Read and take notes on Chapter 10 of AR Drone Developer Guide.
   2) Download and install Ubuntu 12.04 (LTS).
   3) Install AR Drone SDK 2.0.1
   4) Run examples "navigation" & "sdk_demo", and study their code.
3. Start coding based on those examples.

Some Notes

Choice between two methods:

• Create application (more desirable, but maybe has less capabilities in terms of creating automatic control)

• Create software from scratch (pretty much anything is possible, but it’s much more difficult)

What am I doing?

• Altering interaction between drone and client controller so that the client gives a pre-compiled list of markers/commands to follow. Previously, the client controller gave instructions for maneuvers and changes in orientation.
• Perhaps the user selects a path consisting of a number of markers, which is then sent to the drone as a list of commands.
• The commands could contain a string of goals.
• Markers have attributed locations (x,y,z).
• Locations are created in relation to an origin point (i.e. the center of the music room).
• We would compile a list of locations in relation to the origin, which is used to write the commands list.
• The drone would keep track of its position relative to the origin point.
• This "mental image” could be aided by sensory input of the external environment (i.e. the drone pauses while obstructions block the way).
• However, there is no distance sensor that faces forward.
• Mr. Lin says that the self-derived location will probably worsen over time.
• TEST: accuracy of the altimeter (the sonar sensor).
• TEST: accuracy of the accelerometer.
• TEST: orientation of the sonar distance sensor.
• ANSWER: pointed downwards, but perhaps it can be reoriented.

• The commands could be IDs which respond to physical markers.
• In this case, the markers are placed throughout the school to be followed.
• The markers may have to have recognizable ID tags, which probably would require a more sophisticated seeking function than simply finding the concentration of a certain color.
• The ID could be a series of shapes and/or colors…?

Resources

- On simultaneous localization and mapping:

  http://www.nickd.nl/dl/thesis_Nick_Dijkshoorn.pdf 

- On using openCV within the SDK to stream video from the drone to a computer: 

http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/

Re: Method Question

Sorry to miss your post. There are a few thoughts about your proposal:

1. Since the drone will be navigating in a known indoor environment, you should take advantage of that knowledge and integrate the geometry (or map) of the building into your navigation algorithm.
2. The Inertial Measurement Unit (IMU) of the drone can definitely gives you valuable real-time information of navigation. Use it to control the drone motion should be part of the learning curve of drone programming and worth trying to solve our problem. We have used it indirectly (through third-party tools) before. Somehow it seems not very reliable, and the drone will drift gradually. However, it shouldn't prevent you from testing the idea under your direct control (i.e., your own program).
3. The ultrasound sensor on the drone is currently used to determine the height, not the distance of obstacles in front of the drone. It is possible to hack the drone and add a ultrasound sensor (and other circuits such as Arduino board) pointing toward the front direction to detect objects.
4. The camera on the drone is actually another powerful sensor which can provide a lot information. Just like human navigating around the indoor environment, we rely heavily on the vision. The color marks guided navigation is just the first step of our project. By putting color marks around the strategic locations of the building will ease the task of visual detection. It is a simplified, intermediate step to test the image processing and navigation. Once that have been done, we can explore more sophisticated algorithms for navigation without any marks, and even handling the moving obstacle avoidance.