(X) Hide this
    • Login
    • Join
      • Say No Bots Generate New Image
        By clicking 'Register' you accept the terms of use .

Windows Phone 7.5 - Manipulating camera stream

(8 votes)
Andrea Boschin
>
Andrea Boschin
Joined Nov 17, 2009
Articles:   91
Comments:   9
More Articles
13 comments   /   posted on Nov 14, 2011
Categories:   Windows Phone

In the previous article from this series I've introduced how to access the cameras of the phone and emulate a photo camera using the Windows Phone API. These API are really straightforward and almost everyone can take pictures easily for the purposes of the application. But for the most demanding, taking pictures does not suffice.

As an example you can think to an application that needs to measure the light incoming from the lens or that is able to read a barcode without asking the user to click a button.

These two samples, but also lot of other I can imagine, require the access to the raw stream coming from the camera. Only in this way the developer can examine the stream, detect shapes, perform calculations, and so on. And only in this way he can develop the most compelling features.

Download the source code

Accessing the raw stream

For the purpose of demonstrating how to access the camera stream, I've created a simple histogram calculator app. A color histogram is a simple chart showing the distribution of colours along the pixels of the image. Lot of professional and semi-professional cameras on the market offer this feature that is useful to people to improve the quality of the pictures. The approach to use in this case, is to capture a snapshot of the raw stream and then iterate the pixels counting the single color occurrences.

Just to be clear, it cannot be a simple image capture performed using the CaptureImage() method I've described in the last article. This method is good to take still pictures to save to the media library, but it cannot be so fast as I need, to calculate the histogram on the fly on the continuously changing image.

To be fast enough I have to access the raw stream and for this purpose they exist two methods called GetPreviewBufferArgb32 and GetPreviewBufferYCbCr. They acts making a copy of the current frame, incoming from the stream, to a buffer. The two methods differ for the format of the buffer they fill. The GetPreviewBufferYCbCr is able to encode the information using the YCbCr format that is an expression of Luminance and Chrominance. In this way Cb and Cr describe the chrominance component made of a composition of red and blue and on the other side the Y describes the luminance component. The Y taken by itself is an index of light intensity.

The YCbCr format can be useful in some applications but it is for sure hard to manipulate because there is not a direct relationship between each value and a single pixel of the image. Also, for the purpose of calculating an histogram, it would be better to have the picture represented as a simple RGB buffer to iterate, to easily count color occurrences. The GetPreviewBufferArgb32 method is the best choice because it fills the buffer representing for each pixel a Int32 value that express Red, Green and Blue plus the Alpha component, determining the level of transparency of the pixel. To extract values from the Int32 you can use the following formula

  • Blue = pixel & 0xf
  • Green = (pixel & 0xf0) >> 8
  • Red = (pixel & 0xf00) >> 16
  • Alpha = (pixel & 0xf000) >> 24

Unfortunately it does not exists a "buffer changed" event so, to write the histogram example, I have to start a timer that trigger a method to recalculate le histogram every single timeout. The calculation of the histogram is not so heavy so the lenght of the timeout may be sufficently short to give to the user the appearance of a chart update almost in realtime, also if this is not exacly the truth. Here is the code to start the timer:

   1: public Histogram()
   2: {
   3:     InitializeComponent();
   4:  
   5:     // add here further initializations
   6:  
   7:     if (PhotoCamera.IsCameraTypeSupported(CameraType.Primary))
   8:     {
   9:         this.Camera = new PhotoCamera(CameraType.Primary);
  10:         this.Camera.Initialized += new EventHandler<CameraOperationCompletedEventArgs>(Camera_Initialized);
  11:         this.cameraViewBrush.SetSource(this.Camera);
  12:         this.CorrectViewFinderOrientation(this.Orientation);
  13:     }
  14:     else
  15:         throw new NotSupportedException("Camera is not supported");
  16: }
  17:  
  18: private void Camera_Initialized(object sender, CameraOperationCompletedEventArgs e)
  19: {
  20:     Timer timer = new Timer(UpdateHistogram, null, 0, 1000);
  21: }

In the constructor of the page I make the needed checks to verify that a camer is available and I also setup the brush to project the viewfinder to the screen. This part is exacly the same I've explained in the last article. Then, when the camera has been initialized I create the timer to call the UpdateHistogram method once a second. This is not really a fast update but you can try shorter values by yourself.

In the UpdateHistogram method I access the buffer using the GetPreviewBufferArgb32 method and then I calculate the chart on the returned buffer. The calculation is made iterating each pixel of the image and calculating the average value between the RGB components. Then the resulting valaue is used as an index on the array representing the counts for the 256 possible values of the average. At the end of the loop the array contains a count value for each color present in the picture:

   1: private void UpdateHistogram(object state)
   2: {
   3:     int max = (int)this.Camera.PreviewResolution.Width * (int)this.Camera.PreviewResolution.Height;
   4:     int[] buffer = new int[max];
   5:     this.Camera.GetPreviewBufferArgb32(buffer);
   6:  
   7:     int [] histogram = new int[256];
   8:  
   9:     foreach (int pixel in buffer)
  10:     {
  11:         byte r = (byte)((pixel >> 16) & 0xff);
  12:         byte g = (byte)((pixel >> 8) & 0xff);
  13:         byte b = (byte)(pixel & 0xff);
  14:         int value = (r + g + b) / 3;
  15:         histogram[value]++;
  16:     }
  17:  
  18:     int h = histogram.Max();
  19:  
  20:     Deployment.Current.Dispatcher.BeginInvoke(
  21:         () =>
  22:         {
  23:             this.Bitmap.Clear();
  24:  
  25:             for(int i = 0; i<256; i++)
  26:             {
  27:                 int v = histogram[i] * 100 / h;
  28:                 this.Bitmap.DrawLine(i, 100, i, 100 - v, Colors.Red);
  29:             }
  30:  
  31:             this.Bitmap.Invalidate();
  32:         });
  33: }

Interesting to say, the asynchronous nature of the Timer is here an opportunity to perform the histogram calculation on a separated thread without impacting the update of the user interface. Only when the buffer has been scanned and the histogram is calculated the Dispatcher marshal the thread to the user interface and the calculated values are plotted to a WriteableBitmap that shows an horizontal bar chart of 256 values. This chart is then presented to the ui with the call to Invalidate().

Saving the stream to a file

The capability of directly access the raw stream does not means we are able to save the stream to a file. Given that we are not able to record the every single change in the buffer, taking a snaphot of the stream with GetPreviewBufferArgb32 does not suffice to say we can be so fast as it is required to record a video without hiccups.

It is for this reason it has been included a FileSink class that is able to automatically read the stream from a camera and save it to the IsolatedStorage. This class is really easy to use, and it can automatically encode the raw stream to a MP4 video. This format can be direclty handled by the MediaElement, so you can easily play again a video you have recorded.

The FileSink class acts as a junction between the camera and a file in the isolated storage. To initialize the FileSink class you have to provide an instance of VideoCaptureDevice that represent the camera you have to record, and the filename of the output file. To get access to the VideoCaptureDevice you have to use the Capturesource class:

   1: this.Source = new CaptureSource();
   2: this.Source.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

The GetDefaultVideoCaptureDevice method give you a reference to the default camera but you are able to enumerate the available cameras using the GetAvailableVideoCaptureDevices method. Using the returned VideoCaptureDevice class you can detect supported formats, and other properties like the FriendlyName. Once you get the reference to the device you can connect it to the FileSink:

   1: this.Sink = new FileSink();
   2: this.Sink.CaptureSource = this.Source;
   3: this.Sink.IsolatedStorageFileName = "video.mp4";
   4:  
   5: this.Source.Start();

After the device has beend start, using the Start method, the FileSink class begins to pump data from the camera directly to the "video.mp4" file, located in the IsolatedStorage. During this process you can also redirect the output of the CaptureSource to a VideoBrush for the purpose of showing a preview of the recorded video.

   1: this.Brush = new VideoBrush();
   2: this.cameraView.Background = this.Brush;
   3: this.Brush.SetSource(this.Source);

To end the recording you have to call the Stop method on the CaptureSource and then you have to disconnect the FileSink from the source setting its properties to null. This automatically free the output stream and becomes the recorded video available for the play. If you have a preview on a VideoBrush you have also ti disconnect the Source from the brush to be sure you can restart the capture in a later time.

   1: this.Source.Stop();
   2: this.Sink.CaptureSource = null;
   3: this.Sink.IsolatedStorageFileName = null;
   4: this.cameraView.Background = null;
   5: this.Brush = null;

To play the recorded video you have to connect it to a MediaElement. This means opening a stream to the Isolated Storage file and pass it as a source for the MediaElement:

   1: this.Stream = new IsolatedStorageFileStream("video.mp4", FileMode.Open, FileAccess.Read, IsolatedStorageFile.GetUserStoreForApplication());
   2: this.media.SetSource(this.Stream);
   3: this.media.Play();

As a side note please be aware that the CaptureSource class is also able to take still pictures using the CaptureImageAsync method. It may be an alternative way to the use of the PhotoCamera class. Of course, once you have recorded a video it is in charge to you the code to upload it to the network if you want to persist it to a file on a pc. Isolated Storage in windows Phone is limited only by the size of the available memory, but it is not a good idea to waste it with huge video files.

Updated example

In the solution I provided with the last article I've added another project that shows the things I've explained in this rows. The first example shows how to calculate the histgram and if you load it on your development device you can appreciate the resulting chart. Also another example shows how to record to a file. The application alternatively record to a file and reply the recorded video. The sequence show how to correcly attach and detach the stream to avoid the lock of something.


Subscribe

Comments

  • phenry

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by phenry on Nov 17, 2011 06:41
    I'm not sure what is happening, but I'm unable to run this no matter what I do.  I keep getting "The application could not be launched for debugging. Ensure that the target device screen is unlocked and that the application is installed."  I also realized that only one project is built with an F6, the rest are not, even after doing a manual build on those other three, I get the same dialog.  Am I doing something wrong?  (and no, my Focus is unlocked and running haha)
  • AndreaBoschin

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by AndreaBoschin on Nov 17, 2011 10:36

    Hi phenry, you need to go to the Configuration Manager and activate the "Build" and "Deploy" flags for the projects you need to run. This is because I deactivate old project while adding the new to avoid multiple deploy during the development.

    Thanks

  • phenry

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by phenry on Nov 17, 2011 16:19

    Ah yes!  I figured that out late last night (config mgr chgs).  Thank you for confirming. 

    I also have another nasty HW problem, my USB port on my Focus is corroding and causing connection problems, that certainly wasn't helping neither!  DOH! 

    Thank you for your blog on this.  The specific hiccup I'm trying to solve right now is usign the LED always on.  I can make it flash, but I would like to control the flash to be on 100% of the time.  I've seen one flashlight app do it, but i don't know why.

  • jackd

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by jackd on Nov 24, 2011 08:14

    Thank you for the clear and helpful examples. I'm getting this sort of error each time I try to run the application:

    "Error    2    Could not load the assembly file:///C:\WriteableBitmapExWinPhone.dll. This assembly may have been downloaded from the Web.  If an assembly has been downloaded from the Web, it is flagged by Windows as being a Web file, even if it resides on the local computer. This may prevent it from being used in your project. You can change this designation by changing the file properties. Only unblock assemblies that you trust. See http://go.microsoft.com/fwlink/?LinkId=179545 for more information.    SLPG.Mango.ManipulateCamera"

     I don't know what's wrong. The dll is not blocked if I check the file properties. Any help is really appreciated.

  • jackd

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by jackd on Nov 25, 2011 14:35
    I solved my problem by compiling the WriteableBitmapExWinPhone.dll from the source code by myself. I couldn't find any other solution.
  • AndreaBoschin

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by AndreaBoschin on Nov 25, 2011 14:43
    the problem is that you downloaded the zip from internet so it is marked as unsecure. open properties of the zip and hit "unblock" in the right lower corner. then you can unzip abd run the project. bye 
  • Marwan

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by Marwan on Jan 27, 2012 15:36

    I created a xaml Item exactly the same as Histogram.xaml havent change anything  ! (I wanted to start by keeping things the same)
    But still it's not working,  the problem is here :

     

    this

     

     

     

    .cameraViewBrush.SetSource(this.Camera);

    it's not giving me any feedback !
    I can't find any clues to help me fixing this, any guidance is appreciated !

     

     

  • Marwan

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by Marwan on Jan 27, 2012 15:45

    I created a xaml Item exactly the same as Histogram.xaml havent change anything  ! (I wanted to start by keeping things the same)
    But still it's not working,  the problem is here :

     

    this

     

     

     

    .cameraViewBrush.SetSource(this.Camera);

    it's not giving me any feedback !
    I can't find any clues to help me fixing this, any guidance is appreciated !

     

     

  • Marwan

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by Marwan on Jan 27, 2012 15:46

    I created a xaml Item exactly the same as Histogram.xaml havent change anything  ! (I wanted to start by keeping things the same)
    But still it's not working,  the problem is here :

     

    this

     

     

     

    .cameraViewBrush.SetSource(this.Camera);

    it's not giving me any feedback !
    I can't find any clues to help me fixing this, any guidance is appreciated !

     

     

  • Marwan

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by Marwan on Jan 27, 2012 15:55

    I created a xaml Item exactly the same as Histogram.xaml havent change anything  ! (I wanted to start by keeping things the same)
    But still it's not working,  the problem is here :

     

    this

     

     

     

    .cameraViewBrush.SetSource(this.Camera);

    it's not giving me any feedback !
    I can't find any clues to help me fixing this, any guidance is appreciated !

     

     

  • mr_rishi

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by mr_rishi on Mar 09, 2012 13:47
    Hi Andrea, I want to know how to access and manipulate the raw camera stream as a texture in order to map it onto an 3D surface. any help is welcome in advance.
  • -_-

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by on Oct 13, 2012 07:43
    After the famous talk-Lady wearing her cheapest uggs uk sheepskin boots in 2003, her mission was all over Hollywood so stylish as they are comfortable cheapest uggs have. Do you? Then secure your pair of men's fashion collection in Stylebop.com!
  • tikha

    Re: Windows Phone 7.5 - Manipulating camera stream


    posted by tikha on Feb 19, 2013 04:43

    i want to use this apps histogram n combine to flash light, but always error  in syntaks"_cameraVisualizerSetSourceMethod.Invoke(_cameraVisualizer, new object[] { camera.InnerCameraObject }); " can you help me to know, where is my fault..

     

    thanks before 

Add Comment

Login to comment:
  *      *       

From this series