Progress Report: 3/6/16

We have decided to switch our strategy of autonomous flight by using the bottom camera and following a line on the floor. This was for the following reasons:

• The main camera couldn't detect markers that was further from a certain distance, which would eliminate the possibility of a marker at the end of hallways
• Following markers on the floor could possibly increase the accuracy of the flight
• Image processing could possibly be more accurate due to the fact it is just finding a line rather than a shape
• The drone will not fly as high, hopefully to increase the accuracy of the flight and line detection

We are now focused on switching the existing image processing to detect lines rather than a circle, and also thinking of different ways to deal with intersections and turning. So far, we are thinking that different colored circles will be used at the intersection and help to determine the direction the drone will turn to. This will be helpful once we implement flight paths to different rooms. Multi-color detection will also have to be utilized in order for this to be successful.

One problem that we have is that when detecting a line, the algorithm will highlight the border of the image for unknown reasons. Hopefully this isn't a significant issue and will be resolved soon. 

Progress Report: 2/28/16

This week I focused on trying to fix some of the unexpected behavior in the drone's flight path, given a marker in frame to follow. It still doesn't work 100% accurately, but I did have some ideas while tweaking the image processing programs:

• The environment in which I tested did result in a lot of unwanted noise. To help counter this inevitability, I want to start introducing the tracking of two different colors, and use a marker that contains more than a single identifying color. I have tested this possibility and linked a picture as a proof-of-concept below, which is a screen shot of the output I got while trying 2-color detection at home.
• I removed erosion from the algorithm, which was slower than it was worth.
• I added in a function which marks all blobs that the drone finds within the frame. I noticed that the drone actually seems to skip over certain candidates for the marker sometimes. I think this may be an important part of the problem.
• The distance from the drone to the marker cannot be too large, or the resulting representation in the video appears insignificant. Therefore, it will probably be necessary to use floor markers in between junctions, perhaps using the bottom camera and a separate color + color detection algorithm.
• Rethinking our whole approach, it may be better to keep all markers on the floor, with, for example, blue circles connected by red lines. It seems kind of rediculous to rethink the approach now, but I think this new method has potential to be better...

Detecting 2 Colors

Notes: 2/23/16

• Problem:
• The drone, though it seems to locate the target accurately enough, does not fly in an expected manner, given a target placed in front of it.

• Possible Improvements:
• Remove erosion (it may be that slowness is more of a hindrance than noise reduction is an improvement) and detection at edges of the image
• Introduce multiple-color detection (make target-identification more accurate), and make a target of concentric, differently-colored circles
• Account for the tilt of the drone, by rotating the X and Y axes of the incoming video images to fix what is “horizontal” and “vertical” 
• Find better, more distinct color (green-blue?)

• Possible Diagnoses:
• The drone is not given enough time to respond to commands, and is thus responding late
• The drone is reacting to noise (unwanted areas that are registered as within the color range)
• The algorithm for finding the marker does not consider every blob, or does not create a blob around every cluster of pixels

• Plans for the Near Future: 
• Move to an area which has no possibility of noise (maybe put white posters up on hallway lockers while testing)
• Mark all blobs, to make sure every candidate is considered
• Look over the program thoroughly, to see what commands are paired to which conditions
• Try removing erosion and check for improvement
• Try putting a lag on sending new commands, and check for improvement
• Try using target’s radius to control Z-axis movement (forward, backward)

Progress Report 2/8/16

• Progress:
• Created a more accurate image processing program, which detects "blobs" of pixels. Blobs have to be a certain size or they will be disregarded. This helps to get rid of noise and unwanted detection.
• Added a prompt script into the code, asking the user for a room before the drone proceeds with the movement. 
• The marker resembles a light skin color, which often gets the drone confused when a face is present within the photo. Changing the color or creating a marker with two different colors could probably fix this issue. 
• Some future plans:
• Blob detection through recursion
• Creating pre-determined flight plans for rooms
• Problems:
• The image processing program is not constantly accurate
• We can't get horizontal movement to work alongside with vertical movement 
• Pre-determined flight plans, while successful, could take a long period of time to accomplish

Notes: 1/19/16

I had an idea for improving my image processing algorithm. Up until now, my program, after it erodes the edge pixels that fell within the color range, has been averaging the white pixels' locations to return an estimate of the target's location within an image. There are a number of weaknesses to this approach:

1. The method assumes that erosion will eliminate all significant noise, but that cannot always be the case. Therefore, if there are 2 or 3 or 4 significant patches of white pixels in the image, then the averaging will probably return an estimated target location in the blank region between them, where the target cannot be.
2. It is difficult to estimate the radius of a given marker, because noise is not separable from what it thinks to be the bounds of the marker.
3. Since it is hard to estimate the radius of the marker, then it is also difficult to estimate its circularity, which I would find by averaging the absolute difference between each radius along the 4 cardinal lines and the average of these 4 radii (average difference in radius length around the circle).

My new method would replace the location averaging following erosion, and instead identify "blobs" (connected groups) of white pixels, and then compare them. This would be done through recursion, with the function looking for white pixels, then looking for pixels connected to each white pixel found until a blob is finished, then moving on to find the next blob. By the end, the program would have an array of blobs stored, each containing an array of pixels within it. Here are the improvements with this new approach:

1. This new function does not rely on erosion to eliminate all contending noise. Instead, it accepts that there may be more than one significant blob of white pixels remaining, and can choose which among them is most likely to be the marker. So the marker location will always be surrounded by white pixels
2. Now that the given marker location will assuredly be within a blob of white pixels, it is easy to estimate the radius of the marker.
3. Like #2, it is now also possible to estimate:
1. Circularity (average difference in radius length around the blob's edge)
2. Mass (# of pixels in the blob)
3. Density (how well each blob fits into the color range)

The only significant drawback to my new method is its decreased speed. This new function is MUCH slower than the old one. However, I tested the method myself in Processing with video input from my laptop's built-in camera and was able to bypass this problem. I simply checked every 5 pixels, and then displayed the final image with each pixel actually being made up of 25 pixels. Here is a video of my testing, holding a printed yellow-orange target:

I thought it worked really well, so I'll try incorporating the new algorithm into my processing function. I did it in processing and not in node.js, where my project is written, because I am much more familiar with processing, and because processing has a much easier way to display outputs visually (like drawing a circle over the target).

NOTE: In the program I used here, I thresholded pixels based on proportions 

            — for example, keep pixel if:  (red/blue) > 2.2   AND   (green/blue) > 1.8 —

            instead of a number range to account for lighting.

I will put the code for the program in the google doc for future reference.

Progress Report - 12/20/15

• Progress:
• I downloaded the jimp module, an alternative/complement to opencv whose library seems to be documented and explained more thoroughly.
• The file I required in the node test program is weird, called index.js or something. I couldn’t find a file named jimp.js, which would usually signal the file with all the others required. However, it seems to work; I’ve tried a few methods w/o error-"not a function".
• I created a rudimentary tracking function given an single image (the image type [.png] matches the images which the drone should feed to the computer):
• 1) I use opencv to threshold the image for red
• 2) Then I feed that output to jimp
• 3) I use jimp to analyze the pixels and their neighbors (for now, horizontally) at a set distance
• 4) I use jimp to find average X and Y of retained pixels
• 5) I now have the marker’s location in the image, along with its horizontal and vertical limits (I draw a point at that location).
• 6) I can now approximate the radius of the marker by finding the average difference between each limit and the center.

• Problems: (Crossed out, because I resolved the problem)
• I tried testing jimp, but the png image I read which was the output of an opencv method came up undefined in pixel color when I tried to read pixel(0,0).
• I think it’s because png images can be formatted in different ways, with different numbers of bytes per pixel.
• I also tried jpg, but that also came up as undefined. I haven’t researched jpg formatting, but perhaps there’s a similar problem.
• Perhaps there’s a way to convert an image to rgba before reading it with jimp.
• Perhaps there’s a way to output an image with a specific data format.
• Solution:
• the color RGBA values returned are not stored in a matrix; they are stored as properties of the color object. 
• So color[0] = undefined, but color.r = the amount of red in the image 
• :)

• Plans:
• Test
• stream = new opencv.ImageStream()
• stream.on(‘data’, function(matrix) {
• //image conversion
• //image analysis (retrieve marker location)
• }
• ardrone.createPngStream().pipe(stream)
• //Perhaps learn how to output the imageStream to a window display OR send the imageStream to a web browser to display. I know for a fact that the latter is possible; I’ve seen it done.
• Improve Tracking Function
• Incorporate vertical neighbors
• Check neighboring pixels at intermediate distances to improve erosion function’s accuracy, so I can lower the distance
• Incorporate ardrone.createPngStream() to input video
• Display video output in browser?
• This isn't really necessary, but would be nice
• Drone
• Test command loops in the drone by putting land() and takeoff() in a repeating loop
• Use location and marker radius in command sequence for drone
• put location.x in center of image by setting yaw value
• move until marker.radius is greater than a certain value
• I assume marker.radius graphed per distance from marker would be a “1/x” kind of function...
• Fix leaning in drone flight
• HOW???
• Try another drone?
• Perhaps leaning is OK, once the drone starts following a marker path to correct is drifting
• Pseudocode for Final Program
• Store command string specified by user
• Take off
• variable: Image = empty
• variable: radius = 0
• variable: Y = constant //used to spin un a direction until marker is found
• for (command in command string) {
• while (no marker is detected in the following image) {
• Get image from drone
• Detect marker location, using the tracking function
• Send image to user //Maybe; not necessary
• Yaw += Y
• }
• while (radius < R) {
• if (location.y is too far from equator) change height
• if (location.x is too far from meridian) change yaw
• if (neither of the above) move forward
• while no marker is detected in the following image {
• Get image from drone
• Detect marker location
• Send image to client
• Yaw += Y
• }
• }
• Stop
• if (command == end) {
• Land
• }
• else if (command == left/right) {
• Change yaw left/right
• }

RE: Patent Assignment

An excellent patent list! I like your idea of inducing positive buoyancy. Does any one patent it? Is it feasible while people are looking ways to miniaturize the drones? How much volume is required to reduce the net weight of the drone? I remember that the German company Festo has done something similar to float their robots. Here are the links of AirJelly and AirPenguin.