What do color changes reveal about an outdoor scene?
Sunkavalli K, Romeiro F, Matusik W, Zickler T, Pfister H. What do color changes reveal about an outdoor scene?, in IEEE Conference on Computer Vision and Pattern Recognition. ; 2008.Abstract
In an extended image sequence of an outdoor scene, one observes changes in color induced by variations in the spectral composition of daylight. This paper proposes a model for these temporal color changes and explores its use for the analysis of outdoor scenes from time-lapse video data. We show that the time-varying changes in direct sunlight and ambient skylight can be recovered with this model, and that an image sequence can be decomposed into two corresponding components. The decomposition provides access to both radiometric and geometric information about a scene, and we demonstrate how this can be exploited for a variety of visual tasks, including color-constancy, background subtraction, shadow detection, scene reconstruction, and camera geo-location.
colorchanges_cvpr2008.pdf colorchanges_cvpr08.avi analyzing_time-lapse_data_gvi_group.pdf
A Perception-based Color Space for Illumination-invariant Image Processing
Chong H, Gortler S, Zickler; T. A Perception-based Color Space for Illumination-invariant Image Processing, in SIGGRAPH, (2008) . ; 2008.Abstract
Motivated by perceptual principles, we derive a new color space in which the associated metric approximates perceived distances and color displacements capture relationships that are robust to spectral changes in illumination. The resulting color space can be used with existing image processing algorithms with little or no change to the methods.
Passive Reflectometry
Romeiro F, Vasilyev Y, Zickler T. Passive Reflectometry, in European Conference on Computer Vision. ; 2008.Abstract
Different materials reflect light in different ways, so reflectance is a useful surface descriptor. Existing systems for measuring reflectance are cumbersome, however, and although the process can be streamlined using cameras, projectors and clever catadioptrics, it generally requires complex infrastructure. In this paper we propose a simpler method for inferring reflectance from images, one that eliminates the need for active lighting and exploits natural illumination instead. The method’s distinguishing property is its ability to handle a broad class of isotropic reflectance functions, including those that are neither radially-symmetric nor well-represented by low-parameter reflectance models. The key to the approach is a bi-variate representation of isotropic reflectance that enables a tractable inference algorithm while maintaining generality. The resulting method requires only a camera, a light probe, and as little as one HDR image of a known, curved, homogeneous surface.
Dense Specular Shape from Multiple Specular Flows
Vasilyev Y, Adato Y, Zickler T, Ben-Shahar O. Dense Specular Shape from Multiple Specular Flows, in Computer Vision and Pattern Recognition (CVPR). ; 2008.Abstract
Dense Specular Shape from Multiple Specular Flows Authors Vasilyev, Y; Adato, Y; Zickler, T; Ben-Shahar, O Abstract The inference of specular (mirror-like) shape is a particularly difficult problem because an image of a specular object is nothing but a distortion of the surrounding environment. Consequently, when the environment is unknown, such an image would seem to convey little information about the shape itself. It has recently been suggested (Adato et al., ICCV 2007) that observations of relative motion between a specular object and its environment can dramatically simplify the inference problem and allow one to recover shape without explicit knowledge of the environment content. However, this approach requires solving a non-linear PDE (the ‘shape from specular flow equation’) and analytic solutions are only known to exist for very constrained motions. In this paper, we consider the recovery of shape from specular flow under general motions. We show that while the ‘shape from specular flow’ PDE for a single motion is non-linear, we can combine observations of multiple specular flows from distinct relative motions to yield a linear set of equations. We derive necessary conditions for this procedure, discuss several numerical issues with their solution, and validate our results quantitatively using image data.
manyspecularflows_cvpr2008.pdf shapefrommultipleflows_poster_cvpr2008.pdf shape_from_specular_reflections_gvi_group.pdf
Autotagging Facebook: Social Network Context Improves Photo Annotation
Stone Z, Zickler T, Darrell T. Autotagging Facebook: Social Network Context Improves Photo Annotation, in First IEEE Workshop on Internet Vision (part of CVPR). Anchorage, AK ; 2008.Abstract
Most personal photos that are shared online are embedded in some form of social network, and these social networks are a potent sources of contextual information that can be leveraged for automatic image understanding. In this paper, we investigate the utility of social network context for the task of automatic face recognition in personal photographs. We combine face recognition scores with social context in a conditional random field (CRF) model and apply this model to label faces in photos from the popular online social network Facebook, which is now the top photo-sharing site on the Web with billions of photos in total. We demonstrate that our simple method of enhancing face recognition with social network context substantially increases recognition performance beyond that of a baseline face recognition system.
 Color Constancy Beyond Bags of Pixels
Chakrabarti A, Hirakawa K, Zickler T. Color Constancy Beyond Bags of Pixels, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Vol 2008. Anchorage, Ak ; 2008.Abstract
Estimating the color of a scene illuminant often plays a central role in computational color constancy. While this problem has received significant attention, the methods that exist do not maximally leverage spatial dependencies between pixels. Indeed, most methods treat the observed color (or its spatial derivative) at each pixel independently of its neighbors. We propose an alternative approach to illuminant estimation—one that employs an explicit statistical model to capture the spatial dependencies between pixels induced by the surfaces they observe. The parameters of this model are estimated from a training set of natural images captured under canonical illumination, and for a new image, an appropriate transform is found such that the corrected image best fits our model.
chakrabartihirakawazickler_cvpr08.pdf cvpr08poster.pdf
Effective separation of sparse and non-sparse image features for denoising
Chakrabarti A, Hirakawa K. Effective separation of sparse and non-sparse image features for denoising, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Las Vegas, NV ; 2008.Abstract
Over-complete representations of images such as undecimated wavelets have enjoyed immense popularity in recent years. Though they are efficient for modeling singularities and edges, natural images also consist of textures that are difficult to capture with any canonical transformation. In this work, we develop a new modeling strategy with a rigorous treatment of textured regions. Using principal components analysis as an approximate classifier for edges and textures, we partition an image into compressible and incompressible regions-with corresponding models matching their behaviors. A posterior median-based denoising method using these models is described with preliminary results that demonstrate the effectiveness of this approach.
icassp08.pdf ch_icassp_08.pdf
Photometric Stereo with Non-parametric and Spatially-varying Reflectance
Alldrin N, Zickler T, Kriegman D. Photometric Stereo with Non-parametric and Spatially-varying Reflectance. IEEE Conference on Computer Vision and Pattern Recognition. 2008.Abstract
We present a method for simultaneously recovering shape and spatially varying reflectance of a surface from photometric stereo images. The distinguishing feature of our approach is its generality; it does not rely on a specific parametric reflectance model and is therefore purely "data driven". This is achieved by employing novel bi-variate approximations of isotropic reflectance functions. By combining this new approximation with recent developments in photometric stereo, we are able to simultaneously estimate an independent surface normal at each point, a global set of non-parametric "basis material" BRDFs, and per-point material weights. Our experimental results validate the approach and demonstrate the utility of bi-variate reflectance functions for general non-parametric appearance capture. For more information, see Neil Alldrin's research page at UCSD.
generalphotometricstereo_cvpr2008.pdf photometric_stereo_with_non-parametric_and_spatially-varying_reflectance_gvi_g.pdf
Principles of Appearance Acquisition and Representation
Weyrich T, Lawrence J, Lensch HPA, Rusinkiewicz S, Zickler T. Principles of Appearance Acquisition and Representation. Foundations and Trends® in Computer Graphics and Vision. 2008;4(2) :75-191.Abstract
Algorithms for scene understanding and realistic image synthesis require accurate models of the way real-world materials scatter light. This study describes recent work in the graphics community to measure the spatially- and directionally-varying reflectance and subsurface scattering of complex materials, and to develop efficient representations and analysis tools for these datasets. We describe the design of acquisition devices and capture strategies for reflectance functions such as BRDFs and BSSRDFs, efficient factored representations, and a case study of capturing the appearance of human faces.
Color subspaces as photometric invariants
Zickler T, Mallick SP, Kriegman DJ, Belhumeur PN. Color subspaces as photometric invariants. International Journal of Computer Vision. 2008;79 :13-30.Abstract
Complex reflectance phenomena such as specular reflections confound many vision problems since they produce image "features" that do not correspond directly to intrinsic surface properties such as shape and spectral reflectance. A common approach to mitigate these effects is to explore functions of an image that are invariant to these photometric events. In this paper we describe a class of such invariants that result from exploiting color information in images of dichromatic surfaces. These invariants are derived from illuminant-dependent "subspaces" of RGB color space, and they enable the application of Lambertian-based vision techniques to a broad class of specular, non-Lambertian scenes. Using implementations of recent algorithms taken from the literature, we demonstrate the practical utility of these invariants for a wide variety of applications, including stereo, shape from shading, photometric stereo, material-based segmentation, and motion estimation.
The von Kries Hypothesis and a Basis for Color Constancy
Chong H, Gortler S, Zickler T. The von Kries Hypothesis and a Basis for Color Constancy, in IEEE International Conference on Computer Vision (ICCV). ; 2007.Abstract
Color constancy is almost exclusively modeled with diagonal transforms. However, the choice of basis under which diagonal transforms are taken is traditionally ad-hoc. Attempts to remedy the situation have been hindered by the fact that no joint characterization of the conditions for {sensors, illuminants, reflectances} to support diagonal color constancy has previously been achieved. In this work, we observe that the von Kries compatibility conditions are impositions only on the sensor measurements , not the physical spectra. This allows us to formulate the von Kries compatibility conditions succinctly as rank constraints on an order 3 measurement tensor. Given this, we propose an algorithm that computes a (locally) optimal choice of color basis for diagonal color constancy and compare the results against other proposed choices.
Isotropy, Reciprocity, and the GBR Ambiguity
Tan P, Mallick S, Kriegman D, Quan L, Zickler T. Isotropy, Reciprocity, and the GBR Ambiguity, in CVPR 2007. ; 2007.Abstract
A set of images of a Lambertian surface under varying lighting directions defines its shape up to a three parameter Generalized Bas-Relief (GBR) ambiguity. In this paper we examine this ambiguity in the context of surfaces having an additive non-Lambertian reflectance component, and we show that the GBR ambiguity is resolved by any non-Lambertian reflectance function that is isotropic and spatially invariant. The key observation is that each point on a curved surface under directional illumination is a member of a family of points that are in isotropic or reciprocal configurations. We show that the GBR can be resolved in closed form by identifying members of these families in two or more images. Based on this idea, we present an algorithm for recovering full Euclidean geometry from a set of uncalibrated photometric stereo images, and we evaluate it empirically on a number of examples.
 GPU based real-time instrument tracking with three-dimensional ultrasound
Novotny PM, Stoll JA, Vasilyev NV, del Nido PJ, Dupont PE, Zickler TE, Howe RD. GPU based real-time instrument tracking with three-dimensional ultrasound. Medical Image Analysis. 2007;11 :458-464.Abstract
Real-time three-dimensional ultrasound enables new intracardiac surgical procedures, but the distorted appearance of instruments in ultrasound poses a challenge to surgeons. This paper presents a detection technique that identifies the position of the instrument within the ultrasound volume. The algorithm uses a form of the generalized Radon transform to search for long straight objects in the ultrasound image, a feature characteristic of instruments and not found in cardiac tissue. When combined with passive markers placed on the instrument shaft, the full position and orientation of the instrument is found in 3D space. This detection technique is amenable to rapid execution on the current generation of personal computer graphics processor units (GPU). Our GPU implementation detected a surgical instrument in .31 ms, sufficient for real-time tracking at the 25-volumes per second rate of the ultrasound machine. A water tank experiment found instrument orientation errors of 1.1 and tip position errors of less than 1.8mm. Finally an in vivo study demonstrated successful instrument tracking inside a beating porcine heart.
Toward a Theory of Shape from Specular Flow
Adato Y, Vasilyev Y, Ben-Shahar O, Zickler T. Toward a Theory of Shape from Specular Flow, in International Conference on Computer Vision (ICCV). ; 2007.Abstract
The image of a curved, specular (mirror-like) surface is a distorted reflection of the environment. The goal of our work is to develop a framework for recovering general shape from such distortions when the environment is neither calibrated nor known. To achieve this goal we consider far-field illumination, where the object-environment distance is relatively large, and we examine the dense specular flow that is induced on the image plane through relative object-environment motion. We show that under these very practical conditions the observed specular flow can be related to surface shape through a pair of coupled nonlinear partial differential equations. Importantly, this relationship depends only on the environment’s relative motion and not its content. We examine the qualitative properties of these equations, present analytic methods for recovery of the shape in several special cases, and empirically validate our results using captured data. We also discuss the relevance to both computer vision and human perception.
shapefromspecularflow_iccv2007.pdf toward_a_theory_of_shape_from_specular_flow_gvi_group.pdf
 Model-based Stereo with Occlusions
Romeiro F, Zickler T. Model-based Stereo with Occlusions, in IEEE International Workshop on Analysis and Modeling of Faces and Gestures. ; 2007.Abstract
This paper addresses the recovery of face models from stereo pairs of images in the presence of foreign-body occlusions. In the proposed approach, a 3D morphable model (3DMM) for faces is augmented by an occlusion map defined on the model shape, and occlusion is detected with minimal computational overhead by incorporating robust estimators in the fitting process. Additionally, the method uses an explicit model for texture (or reflectance) in addition to shape, which is in contrast to most existing multi-view methods that use a shape model alone. We argue that both model components are required to handle certain classes of occluders, and we present empirical results to support this claim. In fact, the empirical results in this paper suggest that even in the absence of occlusions, stereo reconstruction using existing shape-only face models can perform poorly by some measures, and that the inclusion of an explicit texture model may be worth its computational expense.
modelstereowithocclusions_amfg2007.pdf amfg2007_pres.ppt