1. Getting Started

In this tutorial, we'll create an Android app demonstrating AprilTag detection using the ViSP SDK. We assume that you've already created the SDK using this tutorial: Tutorial: Building ViSP SDK for Android.

This tutorial assumes you have the following software installed and configured:

Android Studio
Android Emulator - If you want to test your apps on an emulator first

2. Building the project

2.1. On recent Android Studio

This section presents how to either create from scratch or import the sample project that is located in samples/android folder.

For this section, Android Studio Ladybug Feature Drop | 2024.... has been used.

We highly recommend to follow the steps presented in 2.1.1. Importing sample project if you want to have the example located in samples/android quickly built and running.

2.1.1. Importing sample project

If you carefully followed Tutorial: Building ViSP SDK for Android, when you ran build.py script, a folder named visp-android-sdk has been created in $VISP_WS/visp-build-android.

In $VISP_WS/visp-build-android/visp-android-sdk folder, you should find as many folders as ABI targets you have compiled ViSP for. To work with Android emulator, you will need to use either x86_64 or arm64-v8a folder depending on your arch.

Open Android Studio.

In the welcoming window of Android Studio, enter "File > New > Import Project..." menu.
Then, in the explorator that opens, navigate until you find the "samples" folder in the ABI folder you want to use.

Android sample project imported on a Macbook Pro M1.

The project is now ready to build and run. It will be explained in the 2.1.3. Building and running the app in a simulator section.

2.1.2. Creating a project from scratch

Open Android Studio.

Enter "File > New > New project" menu, select in the "Templates" section "Phone and Tablet". Then, choose "Empty Views Activity":
Press Next button. Give a name to the application. We suggest to choose ApriltagDetection as the Java files of the sample application use it for the Java package naming. Be sure to select "Java" as the language and "Groovy DSL (build.gradle)" for the "Build configuration language". Select the "Minimum SDK" as "API 24":
The source code of this new ApriltagDetection project is located by default in $HOME/AndroidStudioProjects/ folder.
Edit the build.gradle file of the application in order to add implementation project("visp") in the "dependencies" section.
Copy the sdk folder located in $VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/sdk at the root of the project folder, at the same level than app.
Edit the settings.gradle file at the root of the project in order to add the ":visp" project, indicating that its sources are located in the sdk folder. As shown in the next image, you should add the following lines:
include ':visp'

project(':visp').projectDir = new File('sdk')
Copy the shared library libvisp_java3.so located in the sdk/native/libs/${YOUR_ABI} folder in the destination folder app/src/main/jniLibs/${YOUR_ABI}. If app/src/main/jniLibs/${YOUR_ABI} folder doesn't exists, create it. We recall that ${YOUR_ABI} should be set to arm64-v8a or x86_64 depending on your arch.
Copy also all libvisp_*.a static libraries located in the sdk/native/staticlibs/${YOUR_ABI} folder in the destination folder app/src/main/jniLibs/${YOUR_ABI}.

Content of the app/src/main/jniLibs/${YOUR_ABI} folder after copying shared and static libraries from ViSP Android sdk.

Finally, copy the files located in $VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/java/com/example/apriltagdetection/ in the app/src/main/java/com/example/apriltagdetection/ folder of the project.
Copy also the folders located in $VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/res in the app/src/main/res folder.

2.1.3. Building and running the app in a simulator

Before running the app, we need to edit some settings to be able to emulate the camera.

To this end, in Android Studio enter "Run > Edit Configurations..." menu.
A configuration window should appear. In the "Launch Options" section, select "Specified Activity" in the "Launch" section. Then, in the "Activity" section, click on the three dots and then choose Camera Preview Activity.
Finally, click on "Apply" and then "Run" buttons. If you use a simulator, a panel should appear on the right-hand side of Android Studio and display a simulated Android device.
Our sample app indicates in the bottom how many AprilTags are detected. As shown in the next image, when presenting a 36h11 tag in front of the camera you should see that the tag gets detected.

ViSP sample app emulated with the embedded camera. It indicates that one 36h11 AprilTag is detected.

Note
At this point, if your app doesn't show the camera stream, follow 4.1. The application does not detect a camera section to modify app permissions to access the camera.
After few seconds the app should close and you should see the home screen like in the following image.

Home screen that shows the app green icon.

2.2. On old Android Studio

2.2.1. Create an Android Project

If you're new to app development using Android Studio, we'll recommend this tutorial for getting acquainted to it. Following this tutorial create an Android Project with an Empty Activity. Make sure to keep minSdkVersion >= 21, since the SDK is not compatible with older versions. You're app's build.gradle file should look like:

android {
    compileSdkVersion ...
    defaultConfig {
        applicationId "example.myapplication"
        minSdkVersion 21
        versionCode 1
        versionName "1.0"
    ...
    }

2.2.2. Importing ViSP SDK

In Android Studio, head to File -> New -> Import Module option.

Head over to the directory where you've installed the SDK. Select sdk -> java folder and name the module.

This only imports the Java code. You need to copy the libraries too (.so files in linux, .dll in windows and .dylib in mac). Create a folder named jniLibs in app/src/main folder. Then depending upon your emulator/device architecture (mostly x86 or x86_64), create a folder inside jniLibs and copy those libraries into your project.

3. Code analysis

We will now study with more details the code available in the samples/android folder.

First, in MainActivity.java, we need to load ViSP libraries:

public class MainActivity{
 
    // Used to load the 'native-lib' library on application startup.
    static {
        System.loadLibrary("visp_java3");
    }
 
    ...
}

3.1. Begin Camera Preview

In order to be able to use the camera, we need to ask user for Camera Permissions. First, in the app/src/main/AndroidManifest.xml manifest file, we need to include:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
        package="...">
 
    <uses-permission android:name="android.permission.CAMERA" />
 
    <uses-feature
        android:name="android.hardware.camera"
        android:required="true" />
 
    <application ...>
        ...
    </application>
</manifest>

Then, we need to add a runtime permission for accessing camera in MainActivity.java. Note that detection will execute only when user allows camera access:

// Check if the Camera permission has been granted
if (ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
    // Permission is already available, start camera preview
    Snackbar.make(mLayout,
            "Camera permission is available. Starting preview.",
            Snackbar.LENGTH_SHORT).show();
    startCamera();
} else {
    // Permission is missing and must be requested.
    requestCameraPermission();
}

And finally we implement a request callback listener:

public class MainActivity extends AppCompatActivity implements ActivityCompat.OnRequestPermissionsResultCallback {
 
    ...
 
    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions,
                                           @NonNull int[] grantResults) {
        if (requestCode == PERMISSION_REQUEST_CAMERA) {
            // Request for camera permission.
            if (grantResults.length == 1 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                // Permission has been granted. Start camera preview Activity.
        Intent intent = new Intent(this, CameraPreviewActivity.class);
                startActivity(intent);
            } else {
                // TODO Permission request was denied. Do something
            }
        }
    }
 
    ...
}

3.2. Starting Camera Preview

Now we will study the CameraPreviewActivity.java. This will call the camera API. The incident image is recieved as a byte array which can be easily manipulated for our purposes. We can render the resultant image as Java Bitmap in an ImageView element. In brief,

public class CameraPreviewActivity extends MainActivity {
 
    private Camera mCamera;
    public static ImageView resultImageView;
    ...
 
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
 
        // Open an instance of the first camera and retrieve its info.
        mCamera = getCameraInstance(CAMERA_ID);
        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
        Camera.getCameraInfo(CAMERA_ID, cameraInfo);
 
        if (mCamera == null) {
            // Camera is not available, display error message
            Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
            setContentView(R.layout.camera_unavailable);
        } else {
 
            setContentView(R.layout.activity_camera_preview);
 
            resultImageView = findViewById(R.id.resultImage);
 
        ...
 
            // Get the rotation of the screen to adjust the preview image accordingly.
            final int displayRotation = getWindowManager().getDefaultDisplay()
                    .getRotation();
        ...
        }
    }

Now that we get access to Camera, we need to create a Camera Preview class that'll process the image for AprilTag detection. This class is implemented in the CameraPreview.java file. In brief,

public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback {
 
    private int w,h;
    private VpCameraParameters cameraParameters;
    private double tagSize;
 
    public CameraPreview(Context context, Camera camera, Camera.CameraInfo cameraInfo,
                         int displayOrientation) {
        super(context);
 
        // Do not initialize if no camera has been set
        if (camera == null || cameraInfo == null) {
            return;
        }
 
        mCamera = camera;
        mCameraInfo = cameraInfo;
        mDisplayOrientation = displayOrientation;
 
    ...
 
        // init the ViSP tag detection system
        w = mCamera.getParameters().getPreviewSize().width;
        h = mCamera.getParameters().getPreviewSize().height;
        cameraParameters = new VpCameraParameters();
        cameraParameters.initPersProjWithoutDistortion(615.1674805, 615.1675415, 312.1889954, 243.4373779);
        tagSize = 0.053;
    }
 
    ...
 
    public void onPreviewFrame(byte[] data, Camera camera) {
 
        if (System.currentTimeMillis() > 50 + lastTime) {
 
            VpImageUChar imageUChar = new VpImageUChar(data,h,w,true);
 
            // do the image processing
            // Its working even without grey scale conversion
            VpDetectorAprilTag detectorAprilTag = new VpDetectorAprilTag();
            List<VpHomogeneousMatrix> matrices = detectorAprilTag.detect(imageUChar,tagSize,cameraParameters);
            Log.d("CameraPreview.java",matrices.size() + " tags detected");
 
            // Turn `data` to bitmap and display
            updateResult(data, matrices.size() + " 36h11 tags detected within " + (System.currentTimeMillis() - lastTime) +" ms");
 
            lastTime = System.currentTimeMillis();
        }
    }
}

Note that the detector works on grayscale images. The camera API returns values for all pixels (R,G,B,A). Depending on the image format rendered in Android, we can convert those color values into grayscale. Refer this page, for a complete list of formats. Most commonly used format is NV21. In it, the first few bytes are grayscale values of the image and rest are used to compute the color image. So AprilTag detector process only first width*height bytes of the image as VpImageUChar.

Also, we're detecting tags every 50 milli-seconds. This is simple but efficient since actual tag detection time will vary according to the image and should be an asynchronous task.

If we wanted to display the result of the conversion, we would need to change CameraPreviewActivity accordingly,

public class CameraPreviewActivity extends AppCompatActivity {
 
    ...
    public static ImageView resultImageView;
    ...
 
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
 
        // Open an instance of the first camera and retrieve its info.
        mCamera = getCameraInstance(CAMERA_ID);
        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
        Camera.getCameraInfo(CAMERA_ID, cameraInfo);
 
        if (mCamera == null) {
            // Camera is not available, display error message
            Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
            setContentView(R.layout.camera_unavailable);
        } else {
            setContentView(R.layout.activity_camera_preview);
 
            resultInfo = findViewById(R.id.resultTV);
            resultImageView = findViewById(R.id.resultImage);
 
            // init the byte array
            w = mCamera.getParameters().getPreviewSize().width;
            h = mCamera.getParameters().getPreviewSize().height;
 
            // Get the rotation of the screen to adjust the preview image accordingly.
            final int displayRotation = getWindowManager().getDefaultDisplay()
                    .getRotation();
 
            // Create the Preview view and set it as the content of this Activity.
            CameraPreview mPreview = new CameraPreview(this, mCamera, cameraInfo, displayRotation);
            FrameLayout preview = findViewById(R.id.camera_preview);
            preview.addView(mPreview);
        }
    }
 
    public static void updateResult(byte[] Src){
        byte [] Bits = new byte[Src.length*4]; //That's where the RGBA array goes.
        int i;
        for(i=0;i<Src.length;i++){
            Bits[i*4] = Bits[i*4+1] = Bits[i*4+2] = Src[i]; //Invert the source bits
            Bits[i*4+3] = -1;//0xff, that's the alpha.
        }
 
        //Now put these nice RGBA pixels into a Bitmap object
        Bitmap bm = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
        bm.copyPixelsFromBuffer(ByteBuffer.wrap(Bits));
 
        resultImageView.setImageBitmap(bm);
    }
 
    ...
}

Note the inversion in updateResult method. Visp in C++ accepts image as sequence of RGBA values but Java Bitmap process them as ARGB.

However, note that in the sample, the conversion and display of the conversion has been commented out and we instead just write how many tags have been detected.

3.3. Further Image manipulation

In this tutorial, we've developed a bare bones tag detection app. We can use OpenGL for Android to manipulate the image (for instance, drawing a 3D arrow on the tags) using the list of VpHomogeneous matrices. You can find the complete source code of above Android App here.

4. Potential troubleshoutings

4.1. The application does not detect a camera

When you try to launch the application, you may face the following error message "Camera unavailable":

It is because the permission to use the camera has not been granted.

First, click holding your mouse on the application icon. The following menu should appear:

Click on "App info". Then, click on the "Permissions - No permissions granted" item in the menu that will open:

Then, click on the "Camera" item that appears in the "Not allowed" section:

Finally, click on the "Allow only while using the app" item of the menu that appears.

You should now be able to use the embedded camera. Otherwise, check 4.2. The application displays nothing or not what is expected section.

4.2. The application displays nothing or not what is expected

When using the app, you may not see the image acquired by your camera when the application is running. This is because you must parameterize the simulator in order to use your camera.

First, click on the "Device manager" menu. It is either in the "Tools" section of the menu on top of Android Studio or on the right side of the window as follow:

Click on the three vertically-aligned dots that are beside the simulator, and then on "Edit":

Click on the "Show Advanced Settings" that is located in the bottom of the window:

In the "Camera" settings, select the camera you want to use. In this example, the integrated camera of the computer is used:

4.3. The application freezes

The application might lag in the simulator. It might be due to the amount of RAM that is allocated to the camera.

To change that, open the "Device Manager" panel. It is either in the "Tools" section of the menu on top of Android Studio or on the right side of the window as follow:

Then, click on the three vertically-aligned dots that are beside the simulator, and then on "Show on Disk".

In the files explorator, open the "hardware-qemu.ini" file using a text editor software:

Modify the value specified by the hw.ramSize item:

4.4. The application immediately closes

When you launch the application, it immediately closes without displaying anything. It might be due to an OpenMP problem.

Open the "Logcat" terminal, that is located on the bottom left of the screen:

Then, click on the icon of the application to try to launch it. If a message in red appears complaining about libopenmp.so, it is because ViSP has not been compiled with OpenMP as static library. Please open an issue on github.