Introduction
Since the first release of Silverlight, the community has been patiently waiting for a version of Silverlight which would allow us to capture video from a user’s webcam, as well as audio from their microphone. This past month at PDC 09, our wait was over with the release of Silverlight 4 Beta 1.
In this article we will explore the ways in which Silverlight can interact with a webcam, as well as the powerful results one can produce by combining live video with pixel shaders, video brushes as well as WriteableBitmaps.
Download Source Code
Getting Started
To get started, let’s create a new Silverlight 4 project in Visual Studio 2010 and draw a 320x240 pixel black Rectangle called “rectVideo” inside of the LayoutRoot of our MainPage.xaml.
One of the coolest media features included in Silverlight, is the VideoBrush. A VideoBrush gives us the ability to paint any area of our stage with video content. Back in the earlier versions of Silverlight we were limited to painting the source of a MediaElement, such as a Video Stream being played from an external source. A lot of people used the VideoBrush as a way to create cool effects like reflections of a video playing.
With Silverlight 4 addition of Webcam support, we now have the ability to set the source of our VideoBrush to that of the live video being captured through our webcam. We will use this technique to display our video on the Rectangle we just added to our stage.
Attaching to our Webcam
In order to have some live video to display in our black rectangle, we need to attach to our camera (VideoCaptureDevice) and capture its source. The first step in doing this is to instantiate the System.Windows.Media.CaptureSource object. This class is designed to provide methods and properties used to work with audio and video capture devices. In essence, CaptureSource is like a little media player. We attach a device, such as the camera or microphone, and “Start()” or “Stop()” the capture of content through the device. CaptureSource does not give us direct access to the raw audio or video, but instead can be used as the source of a VideoBrush or a custom VideoSink. In the future we will explore how to leverage a class derived from VideoSink to receive video information and to obtain the capture graph.
The first step in setting up our CaptureSource, is to set its “VideoCaptureDevice” and “AudioCaptureDevice” properties equal to one of our attached audio or video devices. Silverlight has created a helper class called “CaptureDeviceConfiguration” which gives us two choices in how we can get access to a camera or microphone.
All available Camera’s or Microphones
The first option we have is to call CaptureDeviceConfiguration.GetAvailableVideoCaptureDevices. This method will return us a generic Collection containing each available VideoCaptureDevice. From this collection we could allow our users to manually select which device they choose to capture or we could look for the device which has the “IsDefaultDevice” property set to true.
Default Camera or Microphone
A second approach we can use is to request the default VideoCaptureDevice . Most people only have a single camera attached to their computer, so requesting the default will get us access without an additional interaction required by the user. Below is the code I use to attach our default camera to my CaptureSource and use that source to feed live video into my VideoBrush.
1: CaptureSource _capture.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
2:
3: VideoBrush videoBrush = new VideoBrush();
4: videoBrush.Stretch = Stretch.Uniform;
5: videoBrush.SetSource(_capture);
6:
7: rectVideo.Fill = videoBrush;
Requesting Access
Now that we have our webcam attached, and our rectangle’s Fill being painted by our VideoBrush, it time to request permission to use the camera. For privacy reasons each time that Silverlight wants to access ones camera or microphone, the user must explicitly give Silverlight permission.
To get this dialog to appear we need to make the following call. The first part of the condition looks to see if we have already been given access, while the second requests the dialog to appear. A true response from either will allow us to call our CaptureSource.Start() method to begin capturing our live video.
1: if (CaptureDeviceConfiguration.AllowedDeviceAccess ||
2: CaptureDeviceConfiguration.RequestDeviceAccess())
3: { 4: _capture.Start();
5: }
Taking it to the next level
Now that we know how to display our webcam in Silverlight, let’s see what we can do with the video being captured.
In Silverlight 3 we saw the introduction of Pixel Shader’s as a way to provide a custom bitmap effect to an image or video. Creating your own ShaderEffect is a great way to dynamically transform our webcams video as it is being painted on our rectangle. Instead of spending a bunch of time talking about how to create a ShaderEffect I encourage everyone to go to http://wpffx.codeplex.com/ and download the Windows Presentation Foundation Pixel Shader Effects Library. It contains just about every effect one would want to apply to our video.
To apply a ShaderEffect to our rectangle, we can either do it procedurally or simply add an Effect node to our XAML.
1: <Rectangle Name="rectVideo" Width="320" Height="240" Fill="Black"
2: MouseLeftButtonDown="rectVideo_MouseLeftButton" VerticalAlignment="Top">
3: <Rectangle.Effect>
4: <local:InvertColorEffect/>
5: </Rectangle.Effect>
6: </Rectangle>
In this example I have applied an Invert effect to my video producing the following results. As each frame of the video is painted to the rectangle the ShaderEffect is applied just prior to the render.
If I want to give my user access to a bunch of effects, then on cool approach is to create a custom list that previews the effect on a small thumbnail of the live video. Using the same VideoBrush I can paint my webcam video on small 48x36 rectangles that are being rendered with one of each of my ShaderEffects. If a user wants to see the effect on the larger video, then we can tie a MouseLeftButtonDown event on the rectangle that alters our larger rectVideo to display the effect.
1: private void Effect_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
2: { 3: Rectangle rec = sender as Rectangle;
4: switch (rec.Name)
5: { 6: case "invert":
7: var invert = new InvertColorEffect();
8: rectVideo.Effect = invert;
9: break;
10: case "mono":
11: var mono = new MonochromeEffect();
12: mono.FilterColor = Colors.White;
13: rectVideo.Effect = mono;
14: break;
15: case "swirl":
16: var swirl = new SwirlEffect();
17: swirl.SwirlStrength = 2.5;
18: rectVideo.Effect = swirl;
19: break;
20: case "tone":
21: var tone = new ColorToneEffect();
22: tone.Toned = 2;
23: tone.LightColor = Colors.Brown;
24: rectVideo.Effect = tone;
25: break;
26: case "emboss":
27: var emboss = new EmbossedEffect();
28: emboss.Amount = 15;
29: emboss.Width = .01;
30: rectVideo.Effect = emboss;
31: break;
32: default:
33: rectVideo.Effect = null;
34: break;
35: }
36: }
Capturing Stills of our Live Video
Now that we have our Shader’s being applied to our video, It would be really fun to capture still images that we can store both in isolated storage and to our file system.
In most situations, the fastest and easiest way to capture a still is to use the AsyncCaptureImage method exposed on our CaptureSource.
1: _capture.AsyncCaptureImage((capturedImage) =>
2: _viewModel.Captures.Add(capturedImage));
This method takes an System.Action<T> that generates a WriteableBitmap at the moment the method is called. The result of this call will be a WritableBitmap that we could use as the BitmapSource of an image or the source of an image we write to a second location.
Unfortunately, since this technique goes directly against the CaptureSource, it does not reflect the ShaderEffect that we have applied to our rectangle which has been painted with our VideoBrush. To get our still to have the effect, we simply have to generate the WritableBitmap from rectVideo directly. One possible downside is that the size of the still we capture will be the exact size of the rectangle we have drawn. If this was a problem we could certainly explore other ways to capture a large image using a similar approach.
1: WriteableBitmap writeableBitmap = new WriteableBitmap(rectVideo, null);
2: string name = Guid.NewGuid().ToString() + ".jpg";
3:
4: //store the image in a collection in my viewmodel
5: _viewModel.Captures.Add(new Capture() { Name = name, Bitmap = writeableBitmap });Isolated Storage
One of the other cool features I decided to add to my example, is the ability to create a list of thumbnails of each still I capture. This collection of thumbnails is displayed in a WrapPanel that has been applied to a ListBox as its ItemPanelTemplate. As I capture the still, I add it to my ListBox via binding to an “ObservableCollection”, as well as writing the image as a jpeg in isolated storage. This allows the application to maintain a persistent cache of each photo that has been captured. Next time we load the Silverlight app the list of stored image will be redisplayed in the ListBox.
Saving Still as Jpeg to File System
Out of the box Silverlight does not have native support of encoding jpegs, but a nice open source library called FJCore has been create that does the trick. http://code.google.com/p/fjcore/
1: using (IsolatedStorageFileStream isfs = new
2: IsolatedStorageFileStream(name, FileMode.CreateNew, _isf))
3: { 4: MemoryStream stream = new MemoryStream();
5: writeableBitmap.EncodeJpeg(stream);
6: stream.CopyTo(isfs);
7: }
The last and final feature I want to demonstrate was the ability to wrte my captures from isolated storage to the file system. Since I store each capture in isolated storage as a jpeg, all I have to do is locate the correct image by name, and copy the “IsolatedStorageFileStream” to the Stream that will be used to write to the file system. .NET 4.0 added a really nice helper function “CopyTo” that allows me to copy the bytes[] from one stream to another.
1: private void Button_Click(object sender, RoutedEventArgs e)
2: { 3: Capture capture = listImages.SelectedItem as Capture;
4: if (capture != null)
5: { 6: if (_saveFileDlg.ShowDialog().Value)
7: { 8: using (Stream stream = _saveFileDlg.OpenFile())
9: { 10: if (_isf.FileExists(capture.Name))
11: { 12: using (IsolatedStorageFileStream isfs =
13: new IsolatedStorageFileStream(capture.Name, FileMode.Open, _isf))
14: { 15: isfs.Seek(0, SeekOrigin.Begin);
16: isfs.CopyTo(stream);
17: }
18: }
19: }
20: }
21: }
22: }
What’s Next
Now that we have the base of a Web capture control, in future articles we can continue to add features to improve its usability. Look for articles on leveraging right click capabilities to save and delete our captures, the introduction of audio captures as well as the introduction of commanding to move further towards the MVVM pattern of design.