Presentation 6 Link

Final Presentation

Project Link

To Project on GitHub

Presentation 5 Link

Presentation 5

Presentation 4 Link

Presentation 4

Presentation 3 Link

Presentation 3

Plan to Fix Call Stack Error

• Array blobs[]           
• Array goodPixels[]    //the white pixels that were marked during thresholding
• for (goodPixel in goodPixels) {
• if (there aren’t any blobs with goodPixel) {
• Array links[]
• Array news[]
• news.append(goodPixel)
• while (news.length > 0) {
• int num = news.length
• for (int index = num-1; index > -1; index--) {
• checkForNewLinks(news[index])      
• links.append(news[index])  
• news.remove(news[index])  
• }
• }
• blobs.append(new Blob(links)) 
• }
• }

Issues as of 4/5/16

Though I edited the program to find a line on the floor and hover over it, and I’m fairly certain everything I wrote makes logical sense, the drone still behaves in weird ways. There are two main problems we are having with controlling the drone:

1. It drifts off course even when its image processing seems to be precise.
2. It has been returning an error that reads: “maximum call stack size exceeded” almost every run now.

What I’ve learned is that, to address the second issue, node only allows a certain number of function calls in a single step. This is to prevent a system from locking into an endless function and crashing.

My current image processing method uses recursion to group image pixels into different blobs after thresholding, and this method is the one that has been returning the call stack error. Therefore, I believe I have to change this function to avoid using recursion (the blob grouping function currently, when creating a blob, calls itself hundreds of times before allowing the program to continue).

My current goal, in conclusion, is to create a new findBlobs() function that avoids the use of recursion.

As to the first problem, I still don’t know how we’re going to fix it, so I’m open to suggestions.

Progress Report: 3/20/16

Progress:

• Figured out how to access the ar drone’s navdata (cardinal orientation and angles, velocity, altitude, etc.)
• Can read in this information and store it in local variables on the connected computer

Problems:

• Since I still haven’t tested out this new sensor data, I’m not sure how effective or accurate it will be
• My program, which should find the most likely path and align the drone along it, doesn’t return the correct result when multiple lines are detected on the floor

Plans:

• Test out the navdata and store all relevant information (for now, that’s mostly just the drone’s velocity and altitude)
• Use the velocity data to counter drifting
• Fix the program to return the correct path results when given multiple possibilities
• Incorporate a second color-thresholding method to find junctions, where the drone will stop and switch paths (a turn or fork in the path)

Progress Report: 3/13/16

News:

- Fixed up path detection on ground to find the best path within the blob that looks most like a line

Plans:
- Still need to fix the algorithm so I find the direction of that path (pick one of the two possible directions)
- Then have the drone align itself to the path direction
- Then test drone alignment by flying it over a tape line and having the drone align to the path

Progress Report: 3/6/16

We have decided to switch our strategy of autonomous flight by using the bottom camera and following a line on the floor. This was for the following reasons:

• The main camera couldn't detect markers that was further from a certain distance, which would eliminate the possibility of a marker at the end of hallways
• Following markers on the floor could possibly increase the accuracy of the flight
• Image processing could possibly be more accurate due to the fact it is just finding a line rather than a shape
• The drone will not fly as high, hopefully to increase the accuracy of the flight and line detection

We are now focused on switching the existing image processing to detect lines rather than a circle, and also thinking of different ways to deal with intersections and turning. So far, we are thinking that different colored circles will be used at the intersection and help to determine the direction the drone will turn to. This will be helpful once we implement flight paths to different rooms. Multi-color detection will also have to be utilized in order for this to be successful.

One problem that we have is that when detecting a line, the algorithm will highlight the border of the image for unknown reasons. Hopefully this isn't a significant issue and will be resolved soon. 

Progress Report: 2/28/16

This week I focused on trying to fix some of the unexpected behavior in the drone's flight path, given a marker in frame to follow. It still doesn't work 100% accurately, but I did have some ideas while tweaking the image processing programs:

• The environment in which I tested did result in a lot of unwanted noise. To help counter this inevitability, I want to start introducing the tracking of two different colors, and use a marker that contains more than a single identifying color. I have tested this possibility and linked a picture as a proof-of-concept below, which is a screen shot of the output I got while trying 2-color detection at home.
• I removed erosion from the algorithm, which was slower than it was worth.
• I added in a function which marks all blobs that the drone finds within the frame. I noticed that the drone actually seems to skip over certain candidates for the marker sometimes. I think this may be an important part of the problem.
• The distance from the drone to the marker cannot be too large, or the resulting representation in the video appears insignificant. Therefore, it will probably be necessary to use floor markers in between junctions, perhaps using the bottom camera and a separate color + color detection algorithm.
• Rethinking our whole approach, it may be better to keep all markers on the floor, with, for example, blue circles connected by red lines. It seems kind of rediculous to rethink the approach now, but I think this new method has potential to be better...

Detecting 2 Colors

Notes: 2/23/16

• Problem:
• The drone, though it seems to locate the target accurately enough, does not fly in an expected manner, given a target placed in front of it.

• Possible Improvements:
• Remove erosion (it may be that slowness is more of a hindrance than noise reduction is an improvement) and detection at edges of the image
• Introduce multiple-color detection (make target-identification more accurate), and make a target of concentric, differently-colored circles
• Account for the tilt of the drone, by rotating the X and Y axes of the incoming video images to fix what is “horizontal” and “vertical” 
• Find better, more distinct color (green-blue?)

• Possible Diagnoses:
• The drone is not given enough time to respond to commands, and is thus responding late
• The drone is reacting to noise (unwanted areas that are registered as within the color range)
• The algorithm for finding the marker does not consider every blob, or does not create a blob around every cluster of pixels

• Plans for the Near Future: 
• Move to an area which has no possibility of noise (maybe put white posters up on hallway lockers while testing)
• Mark all blobs, to make sure every candidate is considered
• Look over the program thoroughly, to see what commands are paired to which conditions
• Try removing erosion and check for improvement
• Try putting a lag on sending new commands, and check for improvement
• Try using target’s radius to control Z-axis movement (forward, backward)

Progress Report 2/8/16

• Progress:
• Created a more accurate image processing program, which detects "blobs" of pixels. Blobs have to be a certain size or they will be disregarded. This helps to get rid of noise and unwanted detection.
• Added a prompt script into the code, asking the user for a room before the drone proceeds with the movement. 
• The marker resembles a light skin color, which often gets the drone confused when a face is present within the photo. Changing the color or creating a marker with two different colors could probably fix this issue. 
• Some future plans:
• Blob detection through recursion
• Creating pre-determined flight plans for rooms
• Problems:
• The image processing program is not constantly accurate
• We can't get horizontal movement to work alongside with vertical movement 
• Pre-determined flight plans, while successful, could take a long period of time to accomplish

Notes: 1/19/16

I had an idea for improving my image processing algorithm. Up until now, my program, after it erodes the edge pixels that fell within the color range, has been averaging the white pixels' locations to return an estimate of the target's location within an image. There are a number of weaknesses to this approach:

1. The method assumes that erosion will eliminate all significant noise, but that cannot always be the case. Therefore, if there are 2 or 3 or 4 significant patches of white pixels in the image, then the averaging will probably return an estimated target location in the blank region between them, where the target cannot be.
2. It is difficult to estimate the radius of a given marker, because noise is not separable from what it thinks to be the bounds of the marker.
3. Since it is hard to estimate the radius of the marker, then it is also difficult to estimate its circularity, which I would find by averaging the absolute difference between each radius along the 4 cardinal lines and the average of these 4 radii (average difference in radius length around the circle).

My new method would replace the location averaging following erosion, and instead identify "blobs" (connected groups) of white pixels, and then compare them. This would be done through recursion, with the function looking for white pixels, then looking for pixels connected to each white pixel found until a blob is finished, then moving on to find the next blob. By the end, the program would have an array of blobs stored, each containing an array of pixels within it. Here are the improvements with this new approach:

1. This new function does not rely on erosion to eliminate all contending noise. Instead, it accepts that there may be more than one significant blob of white pixels remaining, and can choose which among them is most likely to be the marker. So the marker location will always be surrounded by white pixels
2. Now that the given marker location will assuredly be within a blob of white pixels, it is easy to estimate the radius of the marker.
3. Like #2, it is now also possible to estimate:
1. Circularity (average difference in radius length around the blob's edge)
2. Mass (# of pixels in the blob)
3. Density (how well each blob fits into the color range)

The only significant drawback to my new method is its decreased speed. This new function is MUCH slower than the old one. However, I tested the method myself in Processing with video input from my laptop's built-in camera and was able to bypass this problem. I simply checked every 5 pixels, and then displayed the final image with each pixel actually being made up of 25 pixels. Here is a video of my testing, holding a printed yellow-orange target:

I thought it worked really well, so I'll try incorporating the new algorithm into my processing function. I did it in processing and not in node.js, where my project is written, because I am much more familiar with processing, and because processing has a much easier way to display outputs visually (like drawing a circle over the target).

NOTE: In the program I used here, I thresholded pixels based on proportions 

            — for example, keep pixel if:  (red/blue) > 2.2   AND   (green/blue) > 1.8 —

            instead of a number range to account for lighting.

I will put the code for the program in the google doc for future reference.

Progress Report - 12/20/15

• Progress:
• I downloaded the jimp module, an alternative/complement to opencv whose library seems to be documented and explained more thoroughly.
• The file I required in the node test program is weird, called index.js or something. I couldn’t find a file named jimp.js, which would usually signal the file with all the others required. However, it seems to work; I’ve tried a few methods w/o error-"not a function".
• I created a rudimentary tracking function given an single image (the image type [.png] matches the images which the drone should feed to the computer):
• 1) I use opencv to threshold the image for red
• 2) Then I feed that output to jimp
• 3) I use jimp to analyze the pixels and their neighbors (for now, horizontally) at a set distance
• 4) I use jimp to find average X and Y of retained pixels
• 5) I now have the marker’s location in the image, along with its horizontal and vertical limits (I draw a point at that location).
• 6) I can now approximate the radius of the marker by finding the average difference between each limit and the center.

• Problems: (Crossed out, because I resolved the problem)
• I tried testing jimp, but the png image I read which was the output of an opencv method came up undefined in pixel color when I tried to read pixel(0,0).
• I think it’s because png images can be formatted in different ways, with different numbers of bytes per pixel.
• I also tried jpg, but that also came up as undefined. I haven’t researched jpg formatting, but perhaps there’s a similar problem.
• Perhaps there’s a way to convert an image to rgba before reading it with jimp.
• Perhaps there’s a way to output an image with a specific data format.
• Solution:
• the color RGBA values returned are not stored in a matrix; they are stored as properties of the color object. 
• So color[0] = undefined, but color.r = the amount of red in the image 
• :)

• Plans:
• Test
• stream = new opencv.ImageStream()
• stream.on(‘data’, function(matrix) {
• //image conversion
• //image analysis (retrieve marker location)
• }
• ardrone.createPngStream().pipe(stream)
• //Perhaps learn how to output the imageStream to a window display OR send the imageStream to a web browser to display. I know for a fact that the latter is possible; I’ve seen it done.
• Improve Tracking Function
• Incorporate vertical neighbors
• Check neighboring pixels at intermediate distances to improve erosion function’s accuracy, so I can lower the distance
• Incorporate ardrone.createPngStream() to input video
• Display video output in browser?
• This isn't really necessary, but would be nice
• Drone
• Test command loops in the drone by putting land() and takeoff() in a repeating loop
• Use location and marker radius in command sequence for drone
• put location.x in center of image by setting yaw value
• move until marker.radius is greater than a certain value
• I assume marker.radius graphed per distance from marker would be a “1/x” kind of function...
• Fix leaning in drone flight
• HOW???
• Try another drone?
• Perhaps leaning is OK, once the drone starts following a marker path to correct is drifting
• Pseudocode for Final Program
• Store command string specified by user
• Take off
• variable: Image = empty
• variable: radius = 0
• variable: Y = constant //used to spin un a direction until marker is found
• for (command in command string) {
• while (no marker is detected in the following image) {
• Get image from drone
• Detect marker location, using the tracking function
• Send image to user //Maybe; not necessary
• Yaw += Y
• }
• while (radius < R) {
• if (location.y is too far from equator) change height
• if (location.x is too far from meridian) change yaw
• if (neither of the above) move forward
• while no marker is detected in the following image {
• Get image from drone
• Detect marker location
• Send image to client
• Yaw += Y
• }
• }
• Stop
• if (command == end) {
• Land
• }
• else if (command == left/right) {
• Change yaw left/right
• }

RE: Patent Assignment

An excellent patent list! I like your idea of inducing positive buoyancy. Does any one patent it? Is it feasible while people are looking ways to miniaturize the drones? How much volume is required to reduce the net weight of the drone? I remember that the German company Festo has done something similar to float their robots. Here are the links of AirJelly and AirPenguin.

Patent Assignment

1.      Methods for adaptive control in a UAV (drone) carrying a hanging load

a.      1000000800044

b.      with a feedback linearization controller (FLC)

2.      A micro-aerial vehicle with built-in methods for running on land, landing, takeoff, and flying

a.      US 20140319266 A1

3.      Method for automatically changing the altitude of a drone in anticipation of doing a flip, and then doing the flip

a.      US 20130006448 A1

4.      Method to send drone information to users connected via iOS devices

a.      US 20120299751 A1

5.      A device with a touchscreen used to communicate with the drone, and it’s appropriate methods

a.      US 20110221692 A1

6.      An application that allows user to receive real-time aerial imagery from drones

a.      US 20120299751 A1

7.      A modular drone

a.      US 20140277854 A1

8.      Method for deploying a parachute on a drone

a.      7643887

9.      A type of drone made for investigating closed environments

a.      51752823

10.   A device for launching and recovering a drone, and a drone designed to complement the device

a.      US 20120292430 A1

11.   Generic drone control system

a.      25449387

12.   System for sending virtual shots between drones fitted with ID beacons

a.      US 20110299732 A1

b.      the victim recognizes itself in the image of the attacker

13.   A device shaped like a ring which absorbs vibrations of the drone and can hold a sensory device within it

a.      US 20120234969 A1

14.   A drone designed for railway maintenance

a.      US 20130220162 A1

15.   Control system for drones using a pair of stick controls

a.      25021972

16.   Controlling drone’s acceleration through feedback from another accelerating object

a.      22577543

17.   Target Seeking Simulator

a.      25503929

18.   Fire Detection System Using AI

a.      24206901

19.   Precision parachute recovery system

a.      24961033

20.   Methods and apparatus for vision-based object-tracking

a.      US 20130272570 A1

b.      library of images

c.      optical flow module

d.      decision forest module

e.      color tracking module

21.   Systems and methods for vision-based target tracking, designed for use by a UAV

a.      54290321

22.   Methods and systems for swapping the battery in a drone, using an energy station

a.      54106948

23.   Methods for having a UAV extract energy from a transmission line (power line?)

a.      39596616

24.   Abstract idea for resources to be passed from a self-sacrificing drone to a main drone (i.e. a power supply)

a.      US 20130132317 A1

25.   Methods, systems, and apparatus for simulating swarm behaviour in drones

a.      US patent: 9,104,201

b.      also supports controlling the drones to move along a path while maintaining the formation through a minimum-distance-to-other-drones requirement

AREAS OF FUTURE INNOVATION

1.   Reference #23 and #22 remind me of the recent ideas relating to charging a battery remotely (probably something with a strong magnetic field around it). I’m not sure exactly how it works, but I think this method of wireless charging could be used to help drones fly without having to land

2.   References #20,  #21,  #12, and #25 suggest that UAVs could collectively send data to a single control center, which could create maps, etc. with the total data and send individualized controls to each of the drones based upon what the center knows about the environment. Basically, the drones could use each other's video feeds to know what they cannot see.

3.   In general, a problem with drones seems to be the low amount of power they can store. Future innovation could offer energy-reducing methods for flying the drone, like

a.      making it lighter

i.       lighter materials

ii.      perhaps incorporate a form of induced positive buoyancy, like attaching a small zeppelin-like component filled with helium

b.      lessening motor friction

c.      shrinking components

d.      creating a more efficient battery

Progress Report - 12/01/15

Progress:

• We successfully downloaded OpenCV to work with the corresponding node.js module
• We successfully downloaded the opencv module for node.js, which can be combined with the ardrone module
• We’ve gone through some of the examples for image processing
• conversion to grayscale
• copying images
• saving and importing images
• creating a binary image highlighting pixels within a certain RGB range
• finding faces
• drawing circles

Problems:

• The functions that enabled us to complete the image processing examples are very self-contained, and I have found it hard to learn to do things at a basic level:
• How to get the color of a pixel at a given location in the image matrix
• How to set the value of a pixel at a given location
• The format in which the pixel values are stored
• The coordinate system used to create the matrices

Plans:

• Figure out the answers to the questions posed in the “Problems” section
• Feed in the drone’s video into a method using the opencv module, and display the video
• Run the image processing examples on the drone video pipeline
• Create a command loop that runs forever
• necessary to analyze images and send commands indefinitely
• Perhaps this would be in a form as simple as:
• for var i=0; i<N; i++ {
•      delay 100
•      turn 10˚ clock-wise
• }

Progress Report - 11/1/15

Progress:

• Learned some more terminology for the ar-drone module of node.js
• Downloaded OpenCV on Mac computers successfully
• We were able to write complete programs using node.js and run them, so that the drone could do a sequence of actions simply by running the script.
• Got further in the install process of opencv after realizing opencv 3.0 wasn’t going to work out

Problems:

• The opencv module is proving difficult to download, much less integrate into a program with the ar-drone module
• Downloading opencv 3.0 was impossible, coming across errors before the install even started. Once downloading opencv 2.4.1, the install seemed to be going well until the end where Owen came across some warnings, and Theo came across errors.
• Still unaware how to fix the drifting problem of the drone, but we plan to look further into that as the programming is developed a little more.

Plans:

• Check if Linux computer already has the opencv module installed — if so, try writing a program (as described in previous report) with it.
• Hopefully fixing the opencv problems on the mac laptops, and if that is fixed we will integrate it with node.js on OSX instead of Linux.
• Start to learn and write programs using both opencv and node.js
• Unsure of which platform this will ultimately take place on, Linux or OSX
• Once we become experienced with the basic examples of opencv we can start to move towards the more complex programs that will allow autonomous flight.