by David Martin, Jeffrey Martin, and the 360Cities Team
You are probably already familiar with medium of 360 degree panoramic images from either Google Street View or KML PhotoOverlays in Google Earth (from the Gigapxl, Gigapan, or 360Cities layers). Tools for authoring panoramic images are getting very good, but the process of preparing and presenting such images on the web is still difficult. That is part of the motivation behind the creation of 360cities.net: panoramic photographers use our site to replace the otherwise difficult and time-consuming process of publishing their geotagged panoramas to the web.
You can turn on the 360Cities gallery layer in Google Earth to view panoramas from our system. The screenshots below show how these panoramas appear in Google Earth. From top left to bottom right, these images show (1) the 360Cities icons marking the locations of panoramas; (2) the info window that opens when you click on one of those icons; (3) the floating sphere that appears when you get close to one of the images, and (4) the view from inside the spherical image. In this article we will demonstrate our technique of creating these spherical panoramic PhotoOverlays for Google Earth.
Google Earth Screenshots (click to view larger images)
A panoramic image is the ultimate wide angle image. A normal photo shows you what the world looks like in an instant in time from a single viewpoint over some limited field of view. The widest wide angle lens, a fisheye lens, can show you up to a 180 degree field of view, which covers half the viewing sphere. A panoramic image, on the other hand, covers the entire viewing sphere, so that you can look in any direction. A panoramic image is not a 3D image: It is still a single viewpoint image, but its field of view is not limited.
Because a panoramic image shows you what the world looks like from a single viewpoint, it is shaped like a sphere. Computers aren't so good at representing spheres, however, so panoramic images are stored as regular rectangular images. One could use any cartographic mapping to unwrap the sphere onto a rectangle. The most common representation used for panoramic images is the equirectangular projection, which looks like this:
The Birth of Benjamin Martin by Jeffrey Martin
Old Town Square Christmas Market by Jeffrey Martin
In an equirectangular projection, the equator and all lines of longitude are undistorted. You can picture the projection as follows: Prick a hole in the north and south poles of the sphere, and cut the date line. Now unwrap the sphere onto a flat surface by stretching at the poles without stretching the lines of longitude so that the image forms a rectangle. In an equirectangular projection, the lines of longitude and latitude form a grid of equal sized squares. The equator is the central horizontal line; the north pole is all along the top, and the south pole all along the bottom. Distortion is minimal at the equator, and infinite at the poles. The aspect ratio of an equirectangular image is 2:1, because the equator is twice the length of each line of longitude.
One does not typically view the equirectangular image, or "equirect", directly because of the severe distortions in the polar regions. The equirect is simply a means of storing the underlying spherical image. Special software can render an equirectangular image into an immersive panoramic experience. Click on the image above to view it in the 360Cities system. Make sure to click and drag on the image at 360Cities to pan; shift and control keys zoom in and out.
To create a high quality spherical panoramic image, you need some special photographic equipment and some special software. Typical photographic equipment includes a digital SLR, a fisheye lens such as the Sigma 8mm F3.5 EX DG or the Peleng 8mm f3.5, a tripod, and a panoramic tripod head. Because no lens can capture the entire image sphere in one shot, one needs to cover the sphere with several shots. The special tripod head helps to rotate the camera without changing the viewpoint. If the lens nodal point moves between shots, then parallax will make the next step very hard. With a fisheye lens, one covers the sphere with 2-6 shots, depending on the equipment, circumstances, and desired quality.
The next step is to combine the individual images into a spherical panoramic image in equirectangular format. The process of doing this is far beyond the scope of this article. There is a good free and open source tool called Hugin that many photographers use for this purpose. There are also commercial programs available. The process is currently semi-automatic, becoming more automatic as time goes on.
Once you have created a panoramic image in the 2:1 aspect equirectangular format, it is a simple matter to publish it on the 360Cities platform, which will make the image available not only on the web but on Google Earth as well.
Once you have signed up for 360 cities, you can optionally edit your profile and immediately upload your image. We recommend a minimum resolution of 5000x2500 pixels. You can upload any size, but the practical maximum resolution is currently 16384x8192 pixels. The image will then appear in your pending images list while our system processes it (see below). When processing is complete, the image will move to your unpublished images list. Before publishing your image, you should edit the image metadata to set at least the title, description, geo location, and heading.
Within a few minutes, your uploaded equirect will be available for viewing on the web and in Google Earth. During those couple minutes, our system converts the image into a variety of formats that enable efficient viewing at full resolution on various platforms. The next section discusses these formats.
Because the full resolution equirectangular images can be quite large (up to 50MB), downloading the equirect to a client for viewing is completely impractical. The solution to this problem is to use tiled image formats. You are probably familiar with the google maps view of the world. You can think of the google maps webpage as a window into a giant image covering the globe. How is that image represented? It is cut into small square tiles of a fixed size, typically 256x256 or 512x512 pixels, and only those tiles that you are looking at are downloaded. In this way, once can browse an arbitrarily large image, and the bandwidth requirement on the client is as small as possible -- proportional to the number of pixels in view. What about zooming in and out? Each zoom level has its own set of tiles. The entire set of tiles over all zoom levels is called a multiresolution tiled image pyramid.
We use this trick to efficiently deliver your high resolution panoramic image to a variety of clients. The equirectangular image that you upload is cut into hundreds or even thousands of tiles; those tiles are streamed to the client on demand, based on the client's current viewpoint and zoom level inside the spherical image. In fact, we cut the image into two separate multiresolution tile sets in order to support two different kinds of clients.
Google Earth, through its ImagePyramid mechanism (which is part of the PhotoOverlay mechanism), is able to render tiled equirectangular images. Because of distortion at the poles, however, a tiled equirect is not the optimal tiled format. We use it because we want to support Google Earth, but the amazing flash plugin that we use for rendering the panoramic images in web pages -- krpano -- supports multiresolution tiled cubic images. A cubic panoramic image is simply a different projection of the sphere into a rectangular image. In fact, it is a projection of the sphere into 6 square images. If you place a cube inside a sphere so that the corners of the cube touch the sphere, and you project the spherical image onto the faces of the cube, then you get the 6 images of the cubic panoramic format. For the images shown above, the cube face projections look like this:
| Front | Right | Back | Left | Up | Down |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| Front | Right | Back | Left | Up | Down |
You can see that this projection has far less distortion than the equirectangular projection. It is also more efficient in terms of storage space and far more efficient to render (requiring only two texture mapped triangles per tile!). Its only disadvantage is that it is somewhat less convenient to manipulate using image editing tools. As a final representation for rendering, however, it is the best choice. After reprojecting the uploaded equirectangular image onto full size cube faces, we build a multiresolution tiled pyramid for each cube face, yielding a multiresolution tiled cubic format for the panoramic image.
How many tiles are involved here? Consider a 10000x5000 pixel equirectangular image. We use 512x512 pixel tiles:
If one had to create these multiresolution tiled images by hand, one would go mad. The tiles also take up a lot of space, and are, at best, a hassle to deal with. The 360 cities platform hides this complexity. You simply upload a single equirect image, and we make sure it is converted and tiled as needed to support efficient rendering.
Where do we do the image processing and where do we store all these files? There are two amazing Amazon services that make it relatively painless to build a scalable processing pipeline and storage system. We use Amazon EC2 for the image processing computations, and we store all the images and tiles in Amazon S3. Our image processing pipeline uses a variety of free or inexpensive tools such as exiftool, nona, kmaketiles, and jpegtran. You might expect us to use the wonderful ImageMagick suite, but ImageMagick's memory requirements are far too high when processing large images to be of much use to us.
Once uploaded and converted, you may view the image both on the web and in Google Earth. For the images shown above:
In either format, as you pan around or zoom in and out, you'll see tiles loaded as needed, just as happens in Google Maps. Krpano and Google Earth load only the tiles from the multiresolution pyramid that are needed to render the current view. The tiled pyramids provide a representation that makes these resolution-independent viewers possible.
How can you get krpano and Google Earth to show images like this? If you use the 360Cities platform, then, of course, you don't have to worry yourself about the details. In the following subsections, we explain how we instruct these viewers to show these images.
The key sections of the KML for the first image above are shown below. These are all elements inside of the PhotoOverlay element:
<Icon>
<href>
http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/equirect/tile/512/$[level]/$[y]/$[x].jpg
</href>
</Icon>
<ViewVolume>
<leftFov>-180</leftFov>
<rightFov>180</rightFov>
<bottomFov>-90</bottomFov>
<topFov>90</topFov>
<near>10</near>
</ViewVolume>
<ImagePyramid>
<tileSize>512</tileSize>
<maxWidth>8192</maxWidth>
<maxHeight>4096</maxHeight>
<gridOrigin>upperLeft</gridOrigin>
</ImagePyramid>
<shape>sphere</shape>
The shape value of sphere alerts Google Earth that this PhotoOverlay is a spherical panoramic image. The href element specifies the URL for the tiles in the multiresolution equirectangular tiled pyramid. In the URL, the $[level] is replaced with the pyramid level number, starting at 0 for the apex of the pyramid; the $[x] and $[y] coordinates identify the tile within the level, starting at (0,0) in the upper left corner (as specified by gridOrigin). The other elements should be mostly self-explanatory. The ViewVolume element describes how much of the sphere is covered (all of it, in this case), and the ImagePyramid element describes the pyramid in terms of the tile size and the size of the base level of the pyramid (8192x4096 pixels). Google Earth calculates the number of pyramid levels from these values. For more information, see the KML Reference Documentation and the PhotoOverlay article in the KML Developer's Guide.
Krpano by has quite good documentation that describes the various ways to use the krpano flash plugin to view panoramic images in web pages. Note the short note on multiresolution panoramas. The canonical way to direct krpano is through an XML interface, but there is also a javascript interface. Here we'll show code for the javascript interface. The key sections of the javascript that communicate the structure of the multiresolution tiled pyramid are:
so.addVariable("image.type","cube");
so.addVariable("image.multires","true");
so.addVariable("image.tilesize","512");
so.addVariable("image.baseindex","0");
so.addVariable("image.level.count","3");
add_array_elem('image.level',{
'left.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/left/tile/512/0/%25r/%25c.jpg',
'right.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/right/tile/512/0/%25r/%25c.jpg',
'front.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/front/tile/512/0/%25r/%25c.jpg',
'back.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/back/tile/512/0/%25r/%25c.jpg',
'up.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/up/tile/512/0/%25r/%25c.jpg',
'down.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/down/tile/512/0/%25r/%25c.jpg',
'tiledimagewidth':'512', 'tiledimageheight':'512'
});
add_array_elem('image.level',{
'left.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/left/tile/512/1/%25r/%25c.jpg',
'right.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/right/tile/512/1/%25r/%25c.jpg',
'front.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/front/tile/512/1/%25r/%25c.jpg',
'back.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/back/tile/512/1/%25r/%25c.jpg',
'up.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/up/tile/512/1/%25r/%25c.jpg',
'down.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/down/tile/512/1/%25r/%25c.jpg',
'tiledimagewidth':'1024', 'tiledimageheight':'1024'
});
add_array_elem('image.level',{
'left.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/left/tile/512/2/%25r/%25c.jpg',
'right.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/right/tile/512/2/%25r/%25c.jpg',
'front.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/front/tile/512/2/%25r/%25c.jpg',
'back.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/back/tile/512/2/%25r/%25c.jpg',
'up.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/up/tile/512/2/%25r/%25c.jpg',
'down.url':'http://tiles.360cities.s3.amazonaws.com/pano/jeffrey-martin/00000038_pano.jpg/cube/down/tile/512/2/%25r/%25c.jpg',
'tiledimagewidth':'2048', 'tiledimageheight':'2048'
});
As you can see, we notify krpano that the image is a multires cube with a tilesize of 512 pixels. Each level is specified separately, giving the cube face sizes at that level as well as 6 URL patterns for the 6 cube faces. Each URL, as in the Google Earth KML example above, specifies the level number as well as row and column variables for the location of the tile within the layer. The coordinates of the tiles in this case start at 0 as specified by baseindex.
Panoramic images provide a compelling immersive experience that cannot be matched by traditional imagery. There exist good tools for creating and rendering these images, but managing and publishing them yourself is difficult and tedious. 360Cities is a platform that fills that crucial gap in the tool chain. Not only does 360Cities make embedding high resolution panoramic images on the web a snap, the image may be viewed in Google Earth as well. Now you have an excuse to buy that cool fisheye lens...