We present Interpolated SelectionConv, a method for transferring the weights of 2D CNNs to graph networks that operate specifically for surfaces, allowing for applications in spherical images and 3D meshes.
This paper builds off our previous work SelectionConv, which you can learn more about at our project page.
In the original SelectionConv work, edges were given the selection of the cardinal or ordinal direction most closely aligned with the spatial relationship between the respective pixels or nodes. With general sampling patterns, however, node relationships do not lie so cleanly along specific axes. Thus, it is beneficial to be able to interpolate between multiple selections. To do this, we first assign multiple edges to a single node with each edge having a different selection (i.e., we assign the same source and target node to multiple adjacency matrices). Then, we assign each of those edges a fractional interpolation value that determines the spatial relationship of the node relative to neighboring directions.
For example, if an angular interpolation is used, the edge between the given source and target node shown in the diagram would have selections of both 2 and 3. Specifically, it would have selection weights of \( w_2 = \frac{\theta_3}{\theta_2+\theta_3} \) and \( w_3 = 1 - w_2 \).
By using interpolated selections, our graph structures can be defined for generic point cloud structures, such as those generated from sampling a spherical surface or a 3D mesh. By defining the graph structure in 3D space, and then making selection assignments based on the local planar projection for each individual node, our interpolated SelectionConv can accurately represent oriented convolution on surfaces. This creates one cohesive framework that does not require designing a specific projection for each unique surface. Additionally, just like the original SelectionConv, we can transfer weights from networks trained on 2D images to our graph-based network.
Our approach is general enough to work with many 2D network architectures and tasks. Thus, we illustrate the results of transferring 2D weights for different tasks to our networks for spherical images and 3D meshes.
Our method can effectively stylize spherical images, without causing seams or distortion. It has fewer artifacts than the naive approach or the original SelectionConv method, especially near the poles of the image.
| Original | Naive |
| SelectionConv | Interpolated SelectionConv |
Additional examples are presented below.
| Original | Interpolated SelectionConv |
| Original | Interpolated SelectionConv |
| Original | Interpolated SelectionConv |
We evaluate our method in performing segmentation using the Stanford 2D-3D-S dataset. It consists of equirectangular images of various indoor scenes with 13 different classes. We compare to various methods designed specifically for spheres. Even after a simple direct transfer, our approach is competitive with the state-of-the-art methods.
| Method | Input | mIOU (Direct Transfer) | mIOU (Fine-Tuned) |
|---|---|---|---|
| SelectionConv Cubemap | RGB | 36.3% | - |
| UGSCNN ( Jiang et al. ) | RGB-D | - | 38.3% |
| GaugeNet ( Cohen et al. ) | RGB-D | - | 39.4% |
| Interpolated SelectionConv | RGB-D | 39.9% | 41.4% |
| HexRUNet ( Zhang et al. ) | RGB-D | - | 43.3% |
| TangentIms ( Eder et al. ) | RGB-D | 38.9% | 51.9% |
Lastly, we demonstrate the ability of our method to generalize to general surfaces by stylizing 3D meshes. In the original SelectionConv work, a UV edge-pairing process was required to determine edges across seams in the texture map. This process failed to remove all artifacts along seam boundaries and was time consuming for high-polygon-count meshes. Our new approach avoids this computationally-expensive step while achieving far better consistency along UV seams since the edges and selections are made in 3D space, rather than on the topologically complex 2D texture map.
| Original | Naive |
|
|
|
| SelectionConv | Interpolated SelectionConv |
|
|
|
Additionally, our method allows users to control the number of initial sampling points, making it time invariant to the complexity of the mesh. This also provides a simple way for controlling the level of detail in the stylization on the mesh while operating on high quality meshes and texture maps.
| Original | Style |
|
|
 |
| Naive (Low Resolution) | Interpolated SelectionConv (Low Resolution) |
|
|
|
| Naive (High Resolution) | Interpolated SelectionConv (High Resolution) |
|
|
|
Checkout our paper and supplemental material for additional results and to learn more.
@InProceedings{Hart_2023_WACV, author = {Hart, David and Whitney, Michael and Morse, Bryan}, title = {Interpolated SelectionConv for Spherical Images and Surfaces}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2023}, pages = {321-330} }