brought to you by Deparment of Optics & Acoustics

the library

Until Lytro comes with their own SDK, I am happy to share my own library:

DLLLytroCompatibleLibrary.dll (2.1.0.0)     XMLLytroCompatibleLibrary.xml

It was built as portable, which means it can be referenced by .NET Framework 4.0, Windows Store and Universal applications, Silverlight 4, Windows Phone 7, Xbox 360 and higher projects (Profile1).

The portable library assembly and documentation are released under MIT license. If you have access to the miloush.net library, you can download the unportable version here+xml.

Namespace UAM.InformatiX.Text.Json
Class JsonExtensions
Class JsonIgnoreDefaultAttribute
Class JsonMemberNameAttribute
Class JsonMemberTypeInAttribute
Class JsonMemberTypeInKeyAttribute
Class JsonTypeMappingAttribute
Class JsonTypeResolver
Namespace UAM.Optics
Struct ColorRgb128Float
Struct ColorRgba128Float
Interface IContinuous2D`1
Interface ISampled2D`1
Namespace UAM.Optics.LightField.Lytro
Class DemosaicedImage
Class FieldImage
Class InterpolatedDepth
Class InterpolatedImage
Class MicroLens
Class MicroLensCollection
Class RawImage
Class UVImage
Class XYImage
Namespace UAM.Optics.LightField.Lytro.Camera
Enum CameraMode
Class HardwareInfo
Class PictureList
Class PictureListEntry
Enum PictureListRotation
Namespace UAM.Optics.LightField.Lytro.IO
Class AccessorExtensions
Class ComponentCollection
Class ComponentReader
Class DepthPackageAccessor
Class FilesPackageAccessor
Class FlatFieldSet
Class LightFieldComponent
Class LightFieldPackage
Class PackageAccessor
Class RawPackageAccessor
Class StackPackageAccessor
Namespace UAM.Optics.LightField.Lytro.Metadata
Class DebugMetadata
Class DepthMapAcceleration
Enum DimensionsMode
Class FilesMetadata
Class FrameMetadata
Class Json
Class ParallaxStackAcceleration
Class PictureMetadata
Class RefocusStackAcceleration
Enum Representation
Namespace UAM.Optics.LightField.Lytro.Net
Class CallbackArgs
Delegate CallbackEventHandler`1
Class CameraModeChangedCallbackArgs
Class IsoSensitivityChangedCallbackArgs
Class LikedChangedCallbackArgs
Enum LoadPictureFormat
Class LytroCallbackSink
Enum LytroCommand
Class LytroNetCanceledException
Class LytroNetClientPortableBase
Class LytroNetException
Class LytroNetProgressChangedEventArgs
Delegate LytroNetProgressChangedHandler
Class LytroNetProtocolViolationException
Class LytroRawMessage
Class LytroRequest
Class LytroResponse
Class PictureCallbackArgs
Class RawCallbackArgs
Class SelfTimerCallbackArgs
Class ShutterSpeedChangedCallbackArgs
Class StateChangedCallbackArgs
Class ZoomLevelChangedCallbackArgs

IO examples

using System.IO; using System.Linq; using UAM.InformatiX.Text.Json; using UAM.Optics.LightField.Lytro.IO; using UAM.Optics.LightField.Lytro.Metadata;

The Lytro.IO namespace provides classes for working with the Lytro’s package files. Here is how to make the simplest splitter:

string path = "IMG_0000.lfp"; LightFieldPackage package = new LightFieldPackage(File.OpenRead(path)); for (int i = 1; i < package.ComponentsCount; i++) File.WriteAllBytes(path + "." + i, package.Components[i].Data);

The Lytro.Metadata namespace contains classes for working with various metadata structures in the packages. There are two levels of abstraction available. The Lytro.Metadata.Json class contains low-level definitions for the JSON data found in the packages. You can directly parse it onto or generate it from the classes. To make this process even easier, a JSON parser from the miloush.net library is included:

LightFieldComponent metadata = package.GetMetadata().First(); Json.Master packageMetadata = new Json.Master(); packageMetadata.LoadFromJson(metadata.GetDataAsString()); // for simplicity only: assuming the property is always there and the first one bool isFavorite = (packageMetadata.Picture.ViewArray[0].VendorContent as Json.LytroStars).Starred;

The remaining classes in the Lytro.IO.Metadata provide even higher level access to the metadata, making the work with Lytro generated files more straight:

PictureMetadata pictureMetadata = new PictureMetadata(packageMetadata.Picture); isFavorite = pictureMetadata.IsStarred;

Of course all the classes support modifying the data as well!

package accessors

Known package types (like raw files, pre-rendered stack files etc.) are now easy to explore using accessors, which save a lot of manual loading and metadata parsing. For example, the simplest way to obtain the depth LUT table from pre-rendered package is:

LightFieldPackage package = new LightFieldPackage(File.OpenRead("stack.lfp")); byte[] depthLut = package.AccessStacks().GetDepthLutComponent().Data; // again, no error checking

on decoding the H.264 frames

H.264 decoder is not available for portable libraries. The viewer can decode them if backed by Windows Media Foundation (excluded from Windows N versions, but can be installed with media feature pack). In order to take advantage of the existing code, feel free to reference the viewer executable instead of the library. An example of its usage is available here.

networking examples

The portable libraries have neither TCP client nor sockets support, so the classes can work with streams only. It is upon the caller to establish connection to the camera and provide the network stream, which is however easy to do. As in the case with metadata, there are several layers that can be used to accomplish the same task.

Let's start from the highest one, which is using the LytroNetClientPortableBase (similar to WebClient). This abstract class must be inherited to provide the network stream. Check examples for a primitive implementation using TCP client.

using System; using System.IO; using System.Net.Sockets; using UAM.Optics.LightField.Lytro.Net;

So you need a picture?

new LytroNetClient().TakePicture();

Also downloading files from the camera is quite easy:

LytroNetClient lytro = new LytroNetClient(); lytro.DownloadBufferSize = 512; // only to demonstrate progress monitoring, this is a small file! lytro.ProgressChanged += (sender, e) => Console.WriteLine("{0:P0} % completed", (float)e.BytesTransferred / e.TotalBytesToTransfer); byte[] data = lytro.DownloadFile("A:\\VCM.TXT"); File.WriteAllBytes("VCM.TXT", data); // save to disk // include a LytroNetClient implementation

The LytroNetClientPortableBase uses LytroRequest and LytroResponse underneath (corresponding to WebRequest and WebResponse).

LytroRequest request = LytroRequest.Create(LytroCommand.QueryBatteryLevel); LytroResponse response = request.GetResponse(stream, 4); Debug.Assert(response.ContentLength == 4); return BitConverter.ToSingle(response.Content, 0);

The lowest layer is represented by the LytroRawMessage class. You can create raw messages representing both requests and responses and access their header fields directly. The messages can be sent using the WriteTo method.

on decoding the YUY2 images

Picture thumbnails returned by the LytroNetClientPortableBase.DownloadPicture method (and the .048, .128 and .133 files) are 4:2:2 raw data in YUY2 format. MSDN has some description and formulas about these formats and YUV ↔ RGB conversion. An example implementation used by the communicator is available here.

light field examples

A complete example on getting a subaperture image from the raw file using the library is available on separate page.

silverlight plug-in

A basic Silverlight 5 plug-in with refocus ability on prerendered files was built using the library and is available for your web servers here.

changelog

2.1.0.0 (08/01/2014)

  • new XYImage/UVImage classes and FieldImage.GetSubapertureImage/GetMicrolensImage methods to easily access subaperture and microlens images;
  • accesors to simplify access to well known packages (FilesPackageAccessor, RawPackageAccessor, StackPackageAccessor, DepthPackageAccessor);
  • new FlatFieldSet class to help finding modulation images in calibration data;
  • RawImage.NormalizeBy overflows fixed and RawImage.Demodulate introduced;
  • default demosaicing algorithm switched from bilinear to Malvar et al.;
  • Json.Modes.NDFilterSpec added;
  • JSON fixes (encoded strings, surrogate pairs, stack overflow on circular root tree);
older changes...
Disclaimer: Jan Kučera and miloush.net are not affiliated with or endorsed by Lytro, Inc.