I have a mostly-complete draft of the training and detection phases of the instance model. This includes color patch extraction, color histogram construction, and consolidation for the training phase, and patch matching and mean-shift mode estimation for detection. Remaining implentation work includes integrating the base model and graph cut segmentation with the instance model.
I made a lot of progress on implementing the instance model. I finished or nearly finished interest point selection, model patch generation, consolidation, color histogram creation, and color correlation algorithms. Also, I also made more progress on chapter 2 of the thesis report.
Because the desktop computer was too old and not powerful enough, I switched to the Mac, and things are running much better now. Matlab on the Mac benchmarks at four times the performance of the old Windows machine.
This week, I plan to finish implementation of the instance model, hopefully finish chapter two, and begin planning and paperwork for the defense.
I completed the design for the instance model. The instance model will be built based on an initial detection and segmentation using the boundary-fragment base model (which is already implemented). Color image patches will be extracted from the segmented image and clustered in the manner of Leibe and Lebo’s implicit shape models. A color correlation function will be used for the clustering distance function, and patches will be selected using the Harris corner detector. In addition, the instance model will include a global appearance factor, represented in a color histogram of the entire segmentation mask. This information will be joined with the base model.
The detection procedure will begin with the base model and instance model performing detection in parallel, in their typical boundary-fragment and implicit-shape manners. The unclustered Hough space votes will be combined (with appropriate weights) and clustered using mean-shift to find a detection. The graph cut algorithm will be used for segmentation. Its “boundary” term will be calculated from the backprojected boundary fragment parts and the image’s edge transform. The “region” term will be generated from a combination of the backprojected instance model parts and the correlation of each pixel with the color histogram.
Diagrams of the design: training and detection/segmentation.
Most of the instance model’s components are already implemented, including the color correlation, color histogram, interest point and patch extraction, clustering, voting, and mean-shift algorithms. The work remaining is to assemble these components into the actual model and modify the graph cut segmentation code.
Chapter 1 of the thesis report is finished, with figures added and “to-do” notes completed. I also began work on chapter 2 and updated existing text to reflect the final design decisions.
I also laid out a detailed plan for all of the remaining work.
This week, I plan to finish chapter two and implement the training portion of the instance model. After that, the detection/segmentation code, testing, and remaining chapters of the report are all that remain.
Downloaded the TU Darmstadt and TU Graz-02 databases (both available from the PASCAL object recognition database collection) and organized the images into training, validation, and test images. I am using Graz-02 for negative validation images, and Darmstadt for the other images. I ran the code against the databases to ensure things are still working properly.
Performance is quite poor on my home computer (AMD Duron 800 MHz with 384 MB memory). Matlab is many times faster on my Mac (1.25 GHz G4 and 1 GB memory). I haven’t yet decided whether to buy the toolkits necessary to finish implementation on the Mac.
I have focued mainly on designing the instance model, which is the majority of the work remaining for the thesis. Have considered several alternatives in part selection, representation, detection algorithm, and segmentation inputs. The instance model has two goals: to add appearance information to the shape information of the instance model, and to provide data for the region term of the graph-cut segmentation algorithm.
There are two main alternatives to part selection: pairing instance model parts with base model parts (boundary fragments), or selecting parts independently (such as with the Harris corner detector). The two main alternatives I am considering for representation are color histograms and color image patches (which would produce an instance model similar to Lebo and Leibe’s implicit shape models). Implementing model building and detection should not be difficult for any combination of these alternatives, and most of the components are already implemented in other areas of the system.
Generating data for the region term of graph cut segmentation is proving to be the biggest challenge. The difficulty is in finding a way to generate meaningful confidence values for all pixels of the image. It may be necessary to use a hybrid of instance model representation or classification methods, because a part-based model does not directly provide pixel-level confidence values at every point in the image. Some authors (such as Leibe) mention the use of uniform sampling over the test image before segmentation, so I will do some reading.
Color correlation and color histogram generation are two interesting algorithms I completed this week. Correlation is similar to normalized grayscale correlation, but uses the hue and saturation components of an HSV image. Because hue is a circular value (hue is represented as the angle component of a vector in the HSV color cone), conventional arithmetic, especially mean calculation, can produce inappropriate results. I will discuss the details of the algorithm later.
I also use hue and saturation for color histograms. The color histogram is basically a histogram with an extra independent axis. A histogram bin is identified as (h, s), where h and s are hue and saturation values.
Here are some results from current progress in my thesis at RIT. I have implemented the base boundary fragment model (Opelt 2006) and integrated it with graph cut segmentation (Boykov 2001). The graph cut segmentation’s energy function is based on:
Here are some images demonstrating the segmentations achieved. Note that the segmentation does use color information, even though these segmentation images are grayscale.
The image above shows two segmentations of a cow, using different parameterizations of the Canny edge detector. The left column shows the two segmentations. The upper right shows the image background, and the lower right shows the output of the Canny edge detector for the lower left segmentation.
We signed a lease on an apartment in Weehawken, NJ. This is across the Hudson from midtown Manhattan, about ten minutes by car or bus into the city. We took lots of pictures while viewing the apartment, but we won’t move in for a few weeks. Meanwhile, check out the pictures.
I ran the first full test of the base detection algorithm for my thesis. Against a validation set of 25 positive examples and 25 negative examples (of cows), it achieved 88% recall (12% false negatives) and 100% precision (no false positives). Also, the localization accuracy of the detections was very good: an average of 2% error in object centroid estimate and 90% bounding box overlap.
The validation images, used in training the model, do not make for a very “challenging” set for testing the model, but it is encouraging to see that the model handles these images well. Since this was the very first test, and changes were likely necessary to the thresholds and parameters used in the algorithm—assuming the algorithm itself was indeed written correctly—I was prepared for significantly worse performance.
Below is the proposed system diagram for my thesis work at RIT. The diagram is included in the proposal. It is displayed here as a low-resolution raster image for online viewing. In the proposal, it is a vector image and will thus print and zoom with much higher quality.
The system, as illustrated in the diagram, is separated into three conceptual (and chronological) parts: training, initial recognition of an object instance, and tracking the object by building a detailed instance model.
The next part of the diagram I will implement is the Graph Cut Segmentation during Instance Model Building.
The nerdcore artist MC Plus+ released his second album, Chip Hop. Its fresh beats, MC Max Flow, and PhD lyrics exceed all my expectations.
Here is an interesting couch at RIT. The back pieces are removable; they are anchored to the frame of the couch with vertical rods, like giant automobile headrests.
