brought to you by Deparment of Optics & Acoustics

the library

Until Lytro comes with their own SDK, I am happy to share my own library:

DLLLytroCompatibleLibrary.dll (2.1.0.0)     XMLLytroCompatibleLibrary.xml

It was built as portable, which means it can be referenced by .NET Framework 4.0, Windows Store and Universal applications, Silverlight 4, Windows Phone 7, Xbox 360 and higher projects (Profile1).

The portable library assembly and documentation are released under MIT license. If you have access to the miloush.net library, you can download the unportable version here+xml.

Namespace UAM.InformatiX.Text.Json
Namespace UAM.Optics
Namespace UAM.Optics.LightField.Lytro
Namespace UAM.Optics.LightField.Lytro.Camera
Namespace UAM.Optics.LightField.Lytro.IO
Namespace UAM.Optics.LightField.Lytro.Metadata
Namespace UAM.Optics.LightField.Lytro.Net

IO examples

using System.IO; using System.Linq; using UAM.InformatiX.Text.Json; using UAM.Optics.LightField.Lytro.IO; using UAM.Optics.LightField.Lytro.Metadata;

The Lytro.IO namespace provides classes for working with the Lytro’s package files. Here is how to make the simplest splitter:

string path = "IMG_0000.lfp"; LightFieldPackage package = new LightFieldPackage(File.OpenRead(path)); for (int i = 1; i < package.ComponentsCount; i++) File.WriteAllBytes(path + "." + i, package.Components[i].Data);

The Lytro.Metadata namespace contains classes for working with various metadata structures in the packages. There are two levels of abstraction available. The Lytro.Metadata.Json class contains low-level definitions for the JSON data found in the packages. You can directly parse it onto or generate it from the classes. To make this process even easier, a JSON parser from the miloush.net library is included:

LightFieldComponent metadata = package.GetMetadata().First(); Json.Master packageMetadata = new Json.Master(); packageMetadata.LoadFromJson(metadata.GetDataAsString()); // for simplicity only: assuming the property is always there and the first one bool isFavorite = (packageMetadata.Picture.ViewArray[0].VendorContent as Json.LytroStars).Starred;

The remaining classes in the Lytro.IO.Metadata provide even higher level access to the metadata, making the work with Lytro generated files more straight:

PictureMetadata pictureMetadata = new PictureMetadata(packageMetadata.Picture); isFavorite = pictureMetadata.IsStarred;

Of course all the classes support modifying the data as well!

package accessors

Known package types (like raw files, pre-rendered stack files etc.) are now easy to explore using accessors, which save a lot of manual loading and metadata parsing. For example, the simplest way to obtain the depth LUT table from pre-rendered package is:

LightFieldPackage package = new LightFieldPackage(File.OpenRead("stack.lfp")); byte[] depthLut = package.AccessStacks().GetDepthLutComponent().Data; // again, no error checking

on decoding the H.264 frames

H.264 decoder is not available for portable libraries. The viewer can decode them if backed by Windows Media Foundation (excluded from Windows N versions, but can be installed with media feature pack). In order to take advantage of the existing code, feel free to reference the viewer executable instead of the library. An example of its usage is available here.

networking examples

The portable libraries have neither TCP client nor sockets support, so the classes can work with streams only. It is upon the caller to establish connection to the camera and provide the network stream, which is however easy to do. As in the case with metadata, there are several layers that can be used to accomplish the same task.

Let's start from the highest one, which is using the LytroNetClientPortableBase (similar to WebClient). This abstract class must be inherited to provide the network stream. Check examples for a primitive implementation using TCP client.

using System; using System.IO; using System.Net.Sockets; using UAM.Optics.LightField.Lytro.Net;

So you need a picture?

new LytroNetClient().TakePicture();

Also downloading files from the camera is quite easy:

LytroNetClient lytro = new LytroNetClient(); lytro.DownloadBufferSize = 512; // only to demonstrate progress monitoring, this is a small file! lytro.ProgressChanged += (sender, e) => Console.WriteLine("{0:P0} % completed", (float)e.BytesTransferred / e.TotalBytesToTransfer); byte[] data = lytro.DownloadFile("A:\\VCM.TXT"); File.WriteAllBytes("VCM.TXT", data); // save to disk // include a LytroNetClient implementation

The LytroNetClientPortableBase uses LytroRequest and LytroResponse underneath (corresponding to WebRequest and WebResponse).

LytroRequest request = LytroRequest.Create(LytroCommand.QueryBatteryLevel); LytroResponse response = request.GetResponse(stream, 4); Debug.Assert(response.ContentLength == 4); return BitConverter.ToSingle(response.Content, 0);

The lowest layer is represented by the LytroRawMessage class. You can create raw messages representing both requests and responses and access their header fields directly. The messages can be sent using the WriteTo method.

on decoding the YUY2 images

Picture thumbnails returned by the LytroNetClientPortableBase.DownloadPicture method (and the .048, .128 and .133 files) are 4:2:2 raw data in YUY2 format. MSDN has some description and formulas about these formats and YUV ↔ RGB conversion. An example implementation used by the communicator is available here.

light field examples

A complete example on getting a subaperture image from the raw file using the library is available on separate page.

silverlight plug-in

A basic Silverlight 5 plug-in with refocus ability on prerendered files was built using the library and is available for your web servers here.

changelog

2.1.0.0 (08/01/2014)

  • new XYImage/UVImage classes and FieldImage.GetSubapertureImage/GetMicrolensImage methods to easily access subaperture and microlens images;
  • accesors to simplify access to well known packages (FilesPackageAccessor, RawPackageAccessor, StackPackageAccessor, DepthPackageAccessor);
  • new FlatFieldSet class to help finding modulation images in calibration data;
  • RawImage.NormalizeBy overflows fixed and RawImage.Demodulate introduced;
  • default demosaicing algorithm switched from bilinear to Malvar et al.;
  • Json.Modes.NDFilterSpec added;
  • JSON fixes (encoded strings, surrogate pairs, stack overflow on circular root tree);
older changes...
Disclaimer: Jan Kučera and miloush.net are not affiliated with or endorsed by Lytro, Inc.