Lawrence Technological University
College of Arts and Science
Department of Mathematics and Computer Sciences

Handouts

Getting started with the Raspberry Pi 3 camera as a coach in 2016

   For the busy Robofest participant, just point your Web browser at https://www.raspberrypi.org/learning/getting-started-with-picamera/ and get started. When you are finished with that follow the link https://www.raspberrypi.org/learning/push-button-stop-motion/ To make an animated video from stills taken when you press a push button attached to your Pi. The rest of this page is too offer some ideas for coaches in anticipation of the inevitable questions. It only happens that the result I show here is a 3 second video. This page is about Linux, Pi hardware and a little electronics. If you want to learn more about video, Dr. Chung is a way better bet.

   Resources:

   Time and money trade-offs:

  1. The Pi 3 is more expensive particularly if you have invested in previous models. The Pi 3 has the least likely SD socket to malfunction and thus corrupt the SD card and then cause your team to restart its project. The Pi 3's USB ports and built in WiFi and Bluetooth will help you as a coach.
  2. Speed matters when you are copying an SD card image and during the many Linux reboots your team will do. Use the Pi 3 and buy the fastest name brand SD card you can find.

   To summarize, the plan was to take a MCM Raspberry Pi 3 Model B Starter Kit and add a Raspberry Pi Camera Board v1.3 from Element14, then follow the tutorials mentioned above. Here are just the places we went off the script.

   Before going on to try video processing we compare the speed of a simple flashing light program on the Pi3 to our previous experiments with the Pi.

pi@robofestpi3:~ $ sudo apt-get install git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
git is already the newest version.
git set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.
pi@robofestpi3:~ $ git clone https://github.com/derekmolloy/exploringrpi.git
Cloning into 'exploringrpi'...
remote: Counting objects: 2602, done.
remote: Compressing objects: 100% (1974/1974), done.
remote: Total 2602 (delta 398), reused 2602 (delta 398), pack-reused 0
Receiving objects: 100% (2602/2602), 15.81 MiB | 1.75 MiB/s, done.
Resolving deltas: 100% (398/398), done.
Checking connectivity... done.
pi@robofestpi3:~ $ ls exploringrpi/chp06
bits      dht     flash_script  GPIOpython  minimal_clk  pthreads  wiringPi
callback  driver  GPIO          memoryGPIO  multicore    python
pi@robofestpi3:~ $ ls exploringrpi/chp06/flash_script
flash.sh
pi@robofestpi3:~ $ cat exploringrpi/chp06/flash_script/flash.sh 
#!/bin/bash
#  Short script to toggle a GPIO pin at the highest frequency
#  possible using Bash - by Derek Molloy
echo 17 > /sys/class/gpio/export
sleep 0.5
echo "out" > /sys/class/gpio/gpio17/direction
COUNTER=0
while [ $COUNTER -lt 100000 ]; do
    echo 1 > /sys/class/gpio/gpio17/value
    let COUNTER=COUNTER+1
    echo 0 > /sys/class/gpio/gpio17/value
done
echo 17 > /sys/class/gpio/unexport
pi@robofestpi3:~ $ ls -l exploringrpi/chp06/flash_script
total 4
-rwxr-xr-x 1 pi pi 398 Aug 15 00:04 flash.sh
pi@robofestpi3:~ $ exploringrpi/chp06/flash_script/flash.sh 
pi@robofestpi3:~ $ logout
Because the light is flashing as fast as it can, using the oscilloscope (module of the Analog Discovery 2 by Digilent) helps to show that the Pi3 is between 3 and 4 times as fast as the Pi we tested before. This test just uses 1 of the 4 cores of the Pi3. Below are pictures and diagrams of the test setup followed by the results:
wiring diagram setup detail
test setup
oscilloscope

   Now a look at video processing comparing a Picamera and a Webcam. We will be following Derek Malloy's Exploring Raspberry Pi.

# With the Pi camera installed we add a module to be able to use it
# in Linux user space.
pi@robofestpi3:~ $ sudo modprobe bcm2835-v4l2
pi@robofestpi3:~ $ ls -l /dev/vid*
crw-rw----+ 1 root video 81, 0 Aug 30 18:25 /dev/video0
# Now plug in a Webcam.
pi@robofestpi3:~ $ lsusb
...
Bus 001 Device 006: ID 046d:08dd Logitech, Inc. QuickCam for Notebooks
pi@robofestpi3:~ $ ls -l /dev/vid*
crw-rw----+ 1 root video 81, 0 Aug 30 18:25 /dev/video0
crw-rw----+ 1 root video 81, 1 Aug 30 18:26 /dev/video1
pi@robofestpi3:~ $ sudo apt install fswebcam libv4l-dev v4l-utils libav-tools
Selecting previously unselected package fswebcam.
(Reading database ... 118175 files and directories currently installed.)
Preparing to unpack .../fswebcam_20140113-1_armhf.deb ...
Reading state information... Done
...
libav-tools is already the newest version.
v4l-utils is already the newest version.
The following NEW packages will be installed:
  fswebcam libv4l-dev
pi@robofestpi3:~ $ cd exploringrpi/chp15
pi@robofestpi3:~/exploringrpi/chp15 $ ls
audio  boost  fswebcam  openCV  v4l2
pi@robofestpi3:~/exploringrpi/chp15 $ cd fswebcam
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ ls
exploringRPi.png  fswebcam.conf  fswebcamrpi.conf
# Test the Pi camera.
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ fswebcam -c fswebcam.conf
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Putting banner at the bottom.
Setting font to /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf.
Setting title "Exploring Raspberry Pi".
Setting timestamp "%H:%M:%S %d/%m/%Y (%Z)".
Setting output format to PNG, quality 0
Writing PNG image to 'exploringRPi.png'.
# Manually make a configuration file for the Webcam.
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ nano fswebcam.conf
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ fswebcam -c fswebcam1.conf
--- Opening /dev/video1...
Trying source module v4l2...
/dev/video1 opened.
Adjusting resolution from 1280x720 to 640x472.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Putting banner at the bottom.
Setting font to /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf.
Setting title "Exploring Raspberry Pi".
Setting timestamp "%H:%M:%S %d/%m/%Y (%Z)".
Setting output format to PNG, quality 0
Writing PNG image to 'exploringRPi-wc.png'.
# Compare the Pi camera and Webcam.
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ v4l2-ctl --all -d 0
Driver Info (not using libv4l2):
	Driver name   : bm2835 mmal
	Card type     : mmal service 16.1
	Bus info      : platform:bcm2835-v4l2
	Driver version: 4.4.11
	Capabilities  : 0x85200005
		Video Capture
		Video Overlay
		Read/Write
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x05200005
		Video Capture
		Video Overlay
		Read/Write
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
	Width/Height  : 1280/720
	Pixel Format  : 'JPEG'
	Field         : None
	Bytes per Line: 0
	Size Image    : 921600
	Colorspace    : JPEG (JFIF/ITU601)
	Flags         : 
Format Video Overlay:
	Left/Top    : 150/50
	Width/Height: 1024/768
	Field       : None
	Chroma Key  : 0x00000000
	Global Alpha: 0xff
	Clip Count  : 0
	Clip Bitmap : No
Framebuffer Format:
	Capability    : Extern Overlay
			Global Alpha
	Flags         : Overlay Matches Capture/Output Size
	Width         : 1024
	Height        : 768
	Pixel Format  : 'YU12'
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 30.000 (30000/1000)
	Read buffers     : 1

User Controls

                     brightness (int)    : min=0 max=100 step=1 default=50 value=50 flags=slider
                       contrast (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                     saturation (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                    red_balance (int)    : min=1 max=7999 step=1 default=1000 value=1000 flags=slider
                   blue_balance (int)    : min=1 max=7999 step=1 default=1000 value=1000 flags=slider
                horizontal_flip (bool)   : default=0 value=0
                  vertical_flip (bool)   : default=0 value=0
           power_line_frequency (menu)   : min=0 max=3 default=1 value=1
                      sharpness (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                  color_effects (menu)   : min=0 max=15 default=0 value=0
                         rotate (int)    : min=0 max=360 step=90 default=0 value=0
             color_effects_cbcr (int)    : min=0 max=65535 step=1 default=32896 value=32896

Codec Controls

             video_bitrate_mode (menu)   : min=0 max=1 default=0 value=0 flags=update
                  video_bitrate (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000
         repeat_sequence_header (bool)   : default=0 value=0
            h264_i_frame_period (int)    : min=0 max=2147483647 step=1 default=60 value=60
                     h264_level (menu)   : min=0 max=11 default=11 value=11
                   h264_profile (menu)   : min=0 max=4 default=4 value=4

Camera Controls

                  auto_exposure (menu)   : min=0 max=3 default=0 value=0
         exposure_time_absolute (int)    : min=1 max=10000 step=1 default=1000 value=1000
     exposure_dynamic_framerate (bool)   : default=0 value=0
             auto_exposure_bias (intmenu): min=0 max=24 default=12 value=12
      white_balance_auto_preset (menu)   : min=0 max=9 default=1 value=1
            image_stabilization (bool)   : default=0 value=0
                iso_sensitivity (intmenu): min=0 max=4 default=0 value=0
           iso_sensitivity_auto (menu)   : min=0 max=1 default=1 value=1
         exposure_metering_mode (menu)   : min=0 max=2 default=0 value=0
                     scene_mode (menu)   : min=0 max=13 default=0 value=0

JPEG Compression Controls

            compression_quality (int)    : min=1 max=100 step=1 default=30 value=30
                     brightness (int)    : min=0 max=100 step=1 default=50 value=50 flags=slider
                       contrast (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                     saturation (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                    red_balance (int)    : min=1 max=7999 step=1 default=1000 value=1000 flags=slider
                   blue_balance (int)    : min=1 max=7999 step=1 default=1000 value=1000 flags=slider
                horizontal_flip (bool)   : default=0 value=0
                  vertical_flip (bool)   : default=0 value=0
           power_line_frequency (menu)   : min=0 max=3 default=1 value=1
                      sharpness (int)    : min=-100 max=100 step=1 default=0 value=0 flags=slider
                  color_effects (menu)   : min=0 max=15 default=0 value=0
                         rotate (int)    : min=0 max=360 step=90 default=0 value=0
             color_effects_cbcr (int)    : min=0 max=65535 step=1 default=32896 value=32896
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ v4l2-ctl --all -d 1
Driver Info (not using libv4l2):
	Driver name   : gspca_zc3xx
	Card type     : USB Camera (046d:08dd)
	Bus info      : usb-3f980000.usb-1.2
	Driver version: 4.4.11
	Capabilities  : 0x85200001
		Video Capture
		Read/Write
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x05200001
		Video Capture
		Read/Write
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (gspca_zc3xx: ok)
Format Video Capture:
	Width/Height  : 640/472
	Pixel Format  : 'JPEG'
	Field         : None
	Bytes per Line: 640
	Size Image    : 113870
	Colorspace    : JPEG (JFIF/ITU601)
	Flags         : 
JPEG compression:
	Quality: 75
	Markers: 0x00000018
		Define Huffman Tables
		Define Quantization Tables
Streaming Parameters Video Capture:
	Frames per second: invalid (0/0)
	Read buffers     : 2

User Controls

                     brightness (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                       contrast (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                          gamma (int)    : min=1 max=6 step=1 default=4 value=4 flags=slider
                 gain_automatic (bool)   : default=1 value=1
           power_line_frequency (menu)   : min=0 max=2 default=0 value=0
                      sharpness (int)    : min=0 max=3 step=1 default=2 value=2 flags=slider

JPEG Compression Controls

            compression_quality (int)    : min=50 max=87 step=1 default=75 value=75
                     brightness (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                       contrast (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                          gamma (int)    : min=1 max=6 step=1 default=4 value=4 flags=slider
                 gain_automatic (bool)   : default=1 value=1
           power_line_frequency (menu)   : min=0 max=2 default=0 value=0
                      sharpness (int)    : min=0 max=3 step=1 default=2 value=2 flags=slider
# Install the C++ bindings for OpenCV.
pi@robofestpi3:~/exploringrpi/chp15/fswebcam $ sudo apt install libopencv-dev
pi@robofestpi3:~/exploringrpi/chp15 $ cd openCV/
pi@robofestpi3:~/exploringrpi/chp15/openCV $ ls
build        edges.png  face.cpp        filter      grayscale.png                Lenna.png  timing.cpp
capture.png  face       faceOutput.png  filter.cpp  haarcascade_frontalface.xml  timing
pi@robofestpi3:~/exploringrpi/chp15/openCV $ g++ -O2 `pkg-config --cflags --libs opencv` filter.cpp -o filter
pi@robofestpi3:~/exploringrpi/chp15/openCV $ ./filter
Started Processing - Capturing Image
HIGHGUI ERROR: V4L: Property Gain(14) not supported by device
Processing - Performing Image Processing
Finished Processing - Saving images
# Build the timing program as well.
pi@robofestpi3:~/exploringrpi/chp15/openCV $ g++ -O2 `pkg-config --cflags --libs opencv` timing.cpp -o timing
pi@robofestpi3:~/exploringrpi/chp15/openCV $ ./timing
It took 4.71522 seconds to process 100 frames
Capturing and processing 21.2079 frames per second 
# Alter filter.cpp and timing.cpp to use the Webcam
pi@robofestpi3:~/exploringrpi/chp15/openCV $ nano filter.cpp
pi@robofestpi3:~/exploringrpi/chp15/openCV $ g++ -O2 `pkg-config --cflags --libs opencv` filter1.cpp -o filter1
pi@robofestpi3:~/exploringrpi/chp15/openCV $ ./filter1
Started Processing - Capturing Image
HIGHGUI ERROR: V4L: Property Gain(14) not supported by device
Processing - Performing Image Processing
Finished Processing - Saving images
pi@robofestpi3:~/exploringrpi/chp15/openCV $ nano timing.cpp
pi@robofestpi3:~/exploringrpi/chp15/openCV $ g++ -O2 `pkg-config --cflags --libs opencv` timing1.cpp -o timing1
pi@robofestpi3:~/exploringrpi/chp15/openCV $ ./timing1
It took 18.1628 seconds to process 100 frames
Capturing and processing 5.50577 frames per second 
pi@robofestpi3:~/exploringrpi/chp15/openCV $ logout
# Sources if you don't have the book handy:
pi@robofestpi3:~/exploringrpi/chp15/openCV $ cat filter.cpp
#include<iostream>
#include<opencv2/opencv.hpp>   // C++ OpenCV include file
using namespace std;
using namespace cv;            // using the cv namespace too

int main()
{
    VideoCapture capture(0);   // capturing from /dev/video0
    cout << "Started Processing - Capturing Image" << endl;
    // set any  properties in the VideoCapture object
    capture.set(CV_CAP_PROP_FRAME_WIDTH,1280);   // width in pixels
    capture.set(CV_CAP_PROP_FRAME_HEIGHT,720);   // height in pixels
    capture.set(CV_CAP_PROP_GAIN, 0);            // enable auto gain
    if(!capture.isOpened()){   // connect to the camera
       cout << "Failed to connect to the camera." << endl;
    }
    Mat frame, gray, edges;    // original, grayscale and edge image
    capture >> frame;          // capture the image to the frame
    if(frame.empty()){         // did the capture succeed?
       cout << "Failed to capture an image" << endl;
       return -1;
    }
    cout << "Processing - Performing Image Processing" << endl;
    cvtColor(frame, gray, CV_BGR2GRAY);     // convert to grayscale
    blur(gray, edges, Size(3,3));           // blur image using a 3x3 kernel
    // use Canny edge detector that outputs to the same image
    // low threshold = 10, high threshold = 30, kernel size = 3
    Canny(edges, edges, 10, 30, 3);         // run Canny edge detector
    cout << "Finished Processing - Saving images" << endl;

    imwrite("capture.png", frame);     // store the original image 
    imwrite("grayscale.png", gray);    // store the grayscale image
    imwrite("edges.png", edges);       // store the processed edge image
    return 0;
}

pi@robofestpi3:~/exploringrpi/chp15/openCV $ cat timing.cpp
#include<iostream>
#include<opencv2/opencv.hpp>
#include<time.h>
using namespace std;
using namespace cv;

int main() {
    VideoCapture capture(0);
    capture.set(CV_CAP_PROP_FRAME_WIDTH,640);
    capture.set(CV_CAP_PROP_FRAME_HEIGHT,480);
    if(!capture.isOpened()){
         cout << "Failed to connect to the camera." << endl;
    }
    Mat frame, edges;
    struct timespec start, end;
    clock_gettime( CLOCK_REALTIME, &start );

    int frames=100;
    for(int i=0; i<frames; i++){
    	capture >> frame;
    	if(frame.empty()){
		cout << "Failed to capture an image" << endl;
		return -1;
    	}
    	cvtColor(frame, edges, CV_BGR2GRAY);
    	Canny(edges, edges, 0, 30, 3);
    }

    clock_gettime( CLOCK_REALTIME, &end );
    double difference = (end.tv_sec - start.tv_sec) + (double)(end.tv_nsec - start.tv_nsec)/1000000000.0d;
    cout << "It took " << difference << " seconds to process " << frames << " frames" << endl;
    cout << "Capturing and processing " << frames/difference << " frames per second " << endl;
    imwrite("edges.png", edges);
    imwrite("capture.png", frame);
    return 0;
}
The executive summary is that the Pi camera has better resolution and is considerably faster than the Logitech Webcam.

Revised August 30, 2016