Tuesday, March 3, 2015

How to CORRECTLY take a screenshot using Android MediaProjection API

With the MediaProjection API in Android 5.0 it is possible to: 
capture the contents of the main screen (the default display) into a Surface object, which your app can then send across the network
The process to capture the screen contents is described in MediaProjection API Demo. This demo uses a SurfaceView to show a miniature version of the device's screen creating a Droste effect. But the surface view is not very helpful to actually use the captured screen i.e you can not get the screen capture as an image or a video. If you need to capture the screen as an image you can use and ImageReader object and pass its surface to createVirtualDisplay() method.

The problem is that it is not very straight forward to get the captured screen from ImageReader object either. The image that you get from ImageReader is raw i.e it has stride pixels in the image buffer and you need to take care of them your self. There are many people out there having difficulties getting the image from an ImageReader e.g here, here and here. The solution described in some of these Stackoverflow posts does work but it reduces the image quality significantly and also it is a lot more work and therefore it is slow. 

Below I describe a simpler method to get the image from ImageReader once you have created a virtual display using the ImageReader's surface. For completeness sake I describe the steps for creating a virtual display as well but if you are familiar with them just skip to step 3.

Step 1
To capture the screen contents you need a MediaProjection object. But first you need the user's permission to capture the screen using following code:
MediaProjectionManager projectionManager = (MediaProjectionManager) getSystemService(Context.MEDIA_PROJECTION_SERVICE);startActivityForResult(projectionManager.createScreenCaptureIntent(), PERMISSION_CODE);
This will start a dialog activity to ask user for permission to capture the display contents. 

Step 2
Once user allows screen capture, we can get a MediaProjection object in the onActivityResult() method using the following code.
MediaProjectionManager projectionManager = (MediaProjectionManager) getSystemService(Context.MEDIA_PROJECTION_SERVICE);
mProjection = projectionManager.getMediaProjection(resultCode, data);
The "data" passed to getMediaProjection() call is the intent data passed to onActivityResult() method. 

Step 3
The call to create a virtual display takes width and height of the virtual display and a surface object which will be used to save the device's screen. To the capture the screen as an image or a series of images we use surface of an ImageReader object. Create an ImageReader object as follows:
mImageReader = ImageReader.newInstance(mWidth, mHeight, ImageFormat.RGB_565, 2);
Step 4
Once we have the MediaProjection and ImageReader objects, we can start capturing screen by creating a virtual display using following code:
mProjection.createVirtualDisplay("screen-mirror", mWidth, mHeight, mDensity, flags, mImageReader.getSurface(), null, null);
Step 5
Now we can get the screen capture as image using the following code.
final Image.Plane[] planes = image.getPlanes();
final ByteBuffer buffer = planes[0].getBuffer();
int offset = 0;
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * mWidth;
// create bitmap
Bitmap bmp = Bitmap.createBitmap(mWidth+rowPadding/pixelStride, mHeight, Bitmap.Config.RGB_565);
bmp.copyPixelsFromBuffer(buffer);
image.close();
now the bitmap object bmp has the your screen capture in correct format and you can simply use Bitmap.compress() method to convert it to JPEG, PNG or WEBP format. 
The error that most people make here is that they create bitmap of the wrong size, they create bitmap of same width and height as the virtual display but the problem is that the image you get from ImageReader has some extra pixels in there on each line. So you need to account for those as well when creating the bitmap to copy the pixel data. Therefore when creating bitmap I am passing "mWidth+rowPadding/pixelStride" as width. This is for sake of clarity so it is easy to understand otherwise you can instead pass "rowStride/pixelStride".

4 comments:

Manos said...

Hi,

I have used code similar to your to capture the screen in Android 5.0. I had a problem in that the user can at any point stop projection by manually going to the notification bar and ending it. Is there a callback so that my app can identify such events and act accordingly? I tried using MediaProjection.Callback but with no success.

Jeramy Morrill said...

using the method you outlined but I'm still getting the extra size on the right edge of the screen shot. Anything else I need to do?

Bitmap __bmp = Bitmap.createBitmap(__rowStride/__pixelStride, __height, Bitmap.Config.ARGB_8888);

Abdul Aziz said...

I have implemented as you suggested by I am getting black images having sizes 2-3 KB, please can you share your code or can help??

Thanks and regards

Kamran Ahmed Ansari said...

What about `flags`? What flags are we supposed to use to capture what's on the screen?

I used `VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR` and I get images of height and width as "-1".