Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Robots @ SIT

Get hold of one of the available Robots at SIT Library and start your robotics journey today!

Color Detection

Required Knowledge and Parts:

  • EV3 set
  • Digital EV3 Colour Sensor
  • SD Card

 

Application:

The digital EV3 Colour Sensor can distinguish between seven different colours and detect the iabsence of colour. It also serves as a light sensor by detecting light intensities. Students can experiment with light reflection of different colours and gain experience with a technology widely used in ndustries to programme robots to function using sorting and line-following with colours.

The Colour Sensor allows the robot to measure reflected red light and ambient light from darkness to very bright sunlight and detect seven colours plus the absence of colour. It can tell the difference between colour or black-and-white or among blue, green, yellow, red, white, and brown.

Note. A apple harvesting robot. The robot comprises a RGB-D camera for vision sensing including colour detection. A universal robot arm (UR5) is then used as the manipulator to pick the apple. From "Fruit Detection and Segmentation for Apple Harvesting Using Visual Sensor in Orchards," by H. Kang, and C. Chen, 2019, Sensors, 19(20), p. 4599 (https://doi.org/10.3390/s19204599). CC BY 4.0.

Color Sensors - Basic

This is an introductory video on how to use the EV3 colour sensor in 3 different modes:-

  • colour mode
  • reflected light mode
  • ambient light mode

Project - Solving Rubik's cube (Python)

This project use color/light sensor to recognise the position of the cube and pass the data to the cube solving algorithm to calculate the actions of solving the cube. The complete product works as below video.

 

Items required for this project:

  • A computer with Windows 10 / Mac OS [M1 Chip Devices not tested yet; use at your own risk]
  • Visual Studio Code (use this link to download)
  • EV3 Brick (with 6 AA Batteries)
  • MicroSD Card (Minimum capacity of 4GB and a maximum capacity of 32GB)
  • MicroSD Card Reader (if your computer does not have an SD card slot)
  • Mini USB cable (included with Lego Mindstorms Set)
  • A MicroUSB Flasher Tool (use this link to download)

Follow the instruction of this link to setup your development environment.

Step 1: Assembling of MindCub3r robot

The robot MindCub3r can be assembled with the following guide from David Gilday - MindCub3r Assembly Guide using EV3 Home Set

Step 2: Setting up SD card and Python

In order to utilize and program the robot in Python, MicroPython is needed. MicroPython is Python for Microcontrollers, In this case, MicroPython for EV3 is needed. The MicroPython image and guide for EV3 can be seen in the following link - MicroPython image and flashing to SD card guide

Detail Steps

  1. Download the EV3 MicroPython microSD card image from the link mentioned above in the Introduction. There is no need to unzip the file
  2. Download and install a microSD card flashing tool. In our case, we used balenaEtcher
  3. Insert the microSD card into your computer or card reader. In our case, a 16GB SD card is used
  4. Launch the flashing tool and install the downloaded EV3 card image into the microSD card. Once installed, eject the SD card
  5. Ensure that the EV3 Brick is turned off. Important: Attach a tape to the back of your SD card to easily remove the card from the EV3 brick. Insert the SD card into the EV3 brick
  6. Turn on the EV3 brick and wait for it to boot up until it reaches the menu screen

Step 3: Connect EV3 Brick to the Internet via USB cable

Internet connection is needed to update the software as well as to install the dependencies and Python code. The guide can be found here, Internet Connection via USB Guide

Detail Steps (Windows)

  1. Using the mini-USB cable that came with the EV3, connect the the EV3 brick to your computer. The computer will take a minute to install the EV3 driver.
  2. Open Devices and Printers in the Control Panel under Hardware and Sound.  (Or just type Devices and Printers in the start menu.) The computer will detect the EV3 as Remote NDIS Compatible Device.
  3. Right-click the Remote NDIS Compatible Device and select Network Settings. This takes you to the Network and Sharing Center.
  4. In the Network and Sharing Center, make a note of which connection is your Internet connection (Access type: Internet) and then click on Change adapter settings.
  5. There will be a connection that says Remote NDIS Compatible Device. This is the EV3. Let’s rename it so it is easy to identify. Right-click the adapter and select Rename. In our case, we renamed it to EV3.
  6. To share our Internet connection with the EV3, double-click the Internet network connection that you noted earlier (not the one we just renamed, but the one with the Access type: Internet). This will show the connection status.
  7. If you have more than one other connection, you will want to verify that this connection is indeed the Internet connection. Then, click on the Properties button.
  8. In the window that opens, select the Sharing tab and check the box that says Allow other network users to connect through this computer’s Internet connection. If you have more than two network connections, it will ask for a “Home networking connection”. If you see this, select your EV3’s network adapter from the list (this will be called EV3 if you renamed it earlier in the guide).
  9. Click OK when you are done and close the status window as well. Windows will automatically reconfigure your other network connections.
  10. Now, we need to do some setting up on the EV3. On your EV3 Brick, go to Networking and select Manage connections…, then select the Wired connection.
  11. Check the box for Connect automatically. This way you don’t have to connect manually after you reboot. Go to IPV4, select Change, cycle through the list to select DHCP. Then select Connect. State should change to Online to indicate that your EV3 is connected to the Internet.

Notes

  • Connecting to the Internet is hit or miss, you may have to keep resetting the network shared connection by turning off and on the Allow other network users to connect through this computer's Internet connection at your computer, and reconnecting the wired connection at your EV3 Brick
  • How to check if EV3 Brick is connected to the internet: On the EV3 Brick, go to Networking > Manage connections > Wired > IPV4, in the display, Gateway should contain an IP address. If it does not, your EV3 Brick is not connected to the internet. Additionally, you should be able to ping the EV3 Brick from your computer.

Step 4: Connect to EV3 terminal using Secure Shell (SSH)

Using a SSH client such as PuTTY to connect to the EV3 Brick to install additional software and run the program. In our case, PuTTY is used as our SSH client. PuTTY is an open source SSH and telnet client. PuTTY can be downloaded here, Download PuTTY. The steps to connect PuTTY to the EV3 Brick can be found here, SSH connection to EV3 Brick.

Detail Steps

  1. Download a SSH client in your computer. In our case we use PuTTY. Once downloaded, run PuTTY
  2. In the PuTTY Configuration window, type in ev3dev for the “host name”. Then click the Open button to connect
  3. Once you are connected, type in the ev3dev username (robot) and the password (maker) if you haven’t changed it yet and then you should be logged in.

Notes

  • The first time you connect, you'll get a warning about the new fingerprint. This is normal. Just click *Yes* to continue. You won't see this again unless you re-flash your SD card
  • Default username: robot
  • Default password: maker

Step 5: Download and installation of software

The Python program for the Rubik's cube solver that we will be using is here, Mindcuber Code Program. In that link, there is a guide to the installation and running of program. With the internet connection connected at the EV3 Brick and SSH terminal established, use the SSH terminal to perform the below steps.

Detail Steps

Installing kociemba

kociemba is a two-phase algorithm for solving of 3x3x3 Rubik's cube, created by Herbert Kociemba. The algorithm is briefly explained here, Herbert Kociemba's Algorithm This algorithm is then ported to a Python package with two equivalent implementations in C and Python. The library that we are using is from here, dwalton76's kociemba python library, which was originally created by muodov. Firstly, update of the operating system and installation of the dependencies of kociemba is performed before installing kociemba.

In your SSH terminal, perform the below steps:

  1. sudo apt-get update
  2. sudo apt-get install build-essential libffi-dev
  3. cd ~/
  4. git clone https://github.com/dwalton76/kociemba.git
  5. cd ~/kociemba/kociemba/ckociemba/
  6. make
  7. sudo make install
Installing rubiks-color-resolver

This library is found here, dwalton76 rubiks-color-resolver python library. The purpose of this library is to resolve the detected Red, Green, Blue (RGB) values to the six cube colors of Rubik's cube. This library analyzes all RGB values and assign one of the six colors of the cube to each square. It then uses the Traveling Salesman algorithm to sort the colors. It is intended to work with kociemba. Firstly, pip for Python 3 will be installed, also known as pip3. Using pip3, rubiks-color-resolver package is then installed from its git repository.

In your SSH terminal, perform the below steps:

  1. sudo apt-get install python3-pip
  2. sudo pip3 install git+https://github.com/dwalton76/rubiks-color-resolver.git

Installing MINDCUB3R program

The program is mentioned at the download and installation of software section. There are two different code programs, namely calibrate_white.py and mindcuber.py. The purpose of the calibrate_white.py is to allow the robot to measure and calibrate its color sensor by measuring the RGB values of the white center of the Rubik's cube. It is a good idea to run calibrate_white.py whenever you move to a room with different lighting. The mindcuber.py is the main program where it controls the robot, scans the colors of the 6 different faces of the Rubik's cube, performs algorithms to solve the Rubik's cube, and controls the robot to turn the Rubik's cube to the solved state. Firstly, the MINDCUB3R repository is cloned from Github. Then a command to kociemba will be made to initialize the kociemba program.

In your SSH terminal, perform the below steps:

  1. cd ~/
  2. git clone https://github.com/ev3dev/ev3dev-lang-python-demo.git
  3. cd ~/ev3dev-lang-python-demo/robots/MINDCUB3R/
  4. kociemba DRLUUBFBRBLURRLRUBLRDDFDLFUFUFFDBRDUBRUFLLFDDBFLUBLRBD

The fourth step is essentially to run the kociemba program by giving a series of string representation of the colors of each of the 54 squares of a 3x3x3 cube. The letters represents different moves which is known as the Rubik's Cube Notation. An explanation of the notation can be found here, Rubik's Cube Notation. With the first run of the kociemba program, it takes some time to process, ~30 seconds, where it build a series of tables that caches to the filesystem. After the first run, the kociemba program will then run faster.

Step 6: Running of MINDCUB3R program

calibrate_white.py

As mentioned previously, this program is to calibrate the robot's color sensor by measuring the RGB values of the white center of the Rubik's cube. It is advised to run this program if you move to a room with different lighting. It would be more accurate if a piece of white paper is attached on the white center of the Rubik's cube for this calibration section. This is because the white center of the Rubik's cube contains other colors which can affect the reading of the color sensor.

To run this program, perform the following steps in your SSH terminal:

  1. cd ~/ev3dev-lang-python-demo/robots/MINDCUB3R/

  2. ./calibrate_white.py

mindcuber.py

As mentioned previously, this is the main program where it controls the robot to compute and solve the Rubik's cube. It takes some time for it to start as well as the computation of the solution of the Rubik's cube. Before starting the program, if you have attached a white paper on the white center of the Rubik's cube for the calibrate_white.py, remove the paper as this affects the flipping of the cube.

To run this program, perform the following steps in your SSH terminal:

  1. cd ~/ev3dev-lang-python-demo/robots/MINDCUB3R/
  2. ./mindcuber.py

Do note that the MINDCUB3R drains battery quickly and is not perfect as some issues can arise. Some issues detected were:

  • Inaccurate color reading by the color sensor
    • Red & Orange (Similar colors)
    • Blue & Green (Similar colors)
  • Bad lighting, where it can affect the color sensor
  • Robot flipper not robust enough to flip the Rubik's cube, causing the cube to remain at the same position affecting the rest of the process

Changing the speed of flipper in mindcuber.py

It was discovered that the default program sometimes fail due to the flipper not able to flip the Rubik's cube. An adjustment to the flipper speed was made and it shows an improvement to the flipping of the cube. The robot was tested three times and it successfully flips the Rubik's cube til completion of every trial.

To change the speed of the flipper, perform the following steps in your SSH terminal:

  1. cd ~/ev3dev-lang-python-demo/robots/MINDCUB3R/
  2. nano mindcuber.py
  3. Press and hold the "down" key on your keyboard til you see two variables, flip_speed = 300 and flip_speed_push = 400. Change the value of both to flip_speed = 700 and flip_speed_push = 800
  4. Once changed, press "CTRL-X" on your keyboard to exit
  5. It will issue a confirmation message on whether you want to save your changes. Press "y" on your keyboard to save the changes
  6. You can proceed and run the program as described at the mindcuber.py section

Explanation of calibrate_white.py code

The calibrate_white.py code can be found here, calibrate_white.py code.

Imported Libraries Purpose
from mindcuber import MindCuber

This library uses the code from the mindcuber.py found in the MINDCUB3R program repository, specifically the MindCuber class

import logging Logging purposes of events, logging documentation
import sys Provides various functions and variables that are used to manipulate different parts of the Python runtime environment. This program uses this library to exit the program if an error occurred. sys documentation

With the imported MindCuber class, the program uses its functions to perform following tasks:

  • Runs wait_for_cube_insert, where it uses the robot's infrared sensor to detect if the Rubik's cube is present on the robot's Rubik's cube holder. The infrared sensor detects based on the distance of the cube. Once the Rubik's cube is detected, it uses the robot's flipper arm to position the Rubik's cube for the color sensor to scan and moves the flipper arm away afterwards.
  • Adjusts the color sensor arm to the middle square of the Rubik's cube. It then calls the calibrate_white() method of the ColorSensor library from ev3dev as seen from here, calibrate_white() method. This method determines the maximum RGB raw values of the white square of the Rubik's cube and assigns the RGB values to red_max, green_max, and blue_max variables, which helps to scale future RGB raw values.
  • Writes the red_max, green_max and blue_max variables in a txt file which will be used by the main program. The program then removes the color sensor arm back to the original position, and shuts down the robot.

Explanation of mindcuber.py code

The mindcuber.py code can be found here, mindcuber.py code.

With the start of this program, a MindCuber() class object is created that initializes various variables and methods. Python class and instance variables are present in the MindCuber() class. These are object-oriented programming concepts which you can read it in more details in this link, Python Class Variables vs. Instance Variables.

After initializing the variables and methods, the program perform following tasks:

  • Checks whether the Rubik's cube is present on the turntable by using the infrared sensor distance detection. Once the Rubik's cube is detected, it uses the flipper to position the cube in the expected position.
  • The program scans the 6 faces of the Rubik's cube utilizing the various motors and the color sensor. Upon every square, the RGB value detected by the color sensor is recorded and saved in the order of the scan_order list. The program then uses the imported RubiksColorSolverGenericmethod and input the scanned information. With the scanned information, the program utilizes kociemba and algorithms to derive the solution of the Rubik's cube.
  • With the solution data, the program then resolves the Rubik's cube. Utilizing the flipper and turntable, the Rubik's cube is then sorted to the derived solution. Once completed, the robot is then shutdown which stops the various motors and sensors. The Rubik's cube has now been solved by the robot!

Below is the explanation of each class and variable in this program. 

Imported Libraries Purpose
from ev3dev2.motor import LargeMotor, MediumMotor, OUTPUT_A, OUTPUT_B, OUTPUT_C, SpeedDPS Utilizes the various motor classes and methods of the robot, ev3dev motor classes documentation
from ev3dev2.sensor.lego import ColorSensor, InfraredSensor Utilizes the various sensor classes and methods of the robot, ev3dev sensor classes documentation
from pprint import pformat Short for Pretty Printer, this library helps to format print output where it notably helps to print more complex data types such as dictionary and list, pprint documentation
from rubikscolorresolver import RubiksColorSolverGeneric Utilizes rubiks-color-resolver library that was installed previously
from subprocess import check_output Allows the production and management of processes to perform various tasks, subprocess documentation
from time import sleep The library time offers many different time-related methods. For this program, it utilizes the sleep method where it helps to delay the robot for timing purposes, sleep() method documentation
import json Short for JavaScript Object Notation, JSON is a lightweight data-interchange format to allow humans to easily read and write as well as to allow machines to easily parse and generate, json documentation
import logging Logging purposes for events, logging documentation
import os Provides operating system dependent functionalities such as file path retrieval and file creation. Hence, it can run smoothly in different operating systems. os documentation
import signal This library is similar to a notification of an event. When an event occurs in the system, a signal is generated to notify programs about the event. For this program, signal is used to stop and shutdown the program and robot if an event happened that causes the program to be terminated or interrupted. signal documentation
import sys Provides various functions and variables that are used to manipulate different parts of the Python runtime environment. This program uses this library to exit the program if an error occurred. sys documentation
import time As mentioned above, this program only uses the sleep() method. In this instance, time.sleep() is written instead.

 

MindCuber() Python Class Variables

Class Variables Purpose
scan_order A list of numbers determining the order to scan the Rubik's cube
hold_cube_pos Sets a value to have the Rubik's cube to be in the targeted position of the turntable
rotate_speed Sets a value on the speed of rotating the cube
flip_speed Sets a value on the speed of pulling back the cube
flip_speed_push Sets a value on the speed of pushing back the cube

MindCuber() Python Class Instance Variables & Methods

Class Instance Variables Purpose
self.shutdown Default value is False. When robot is shutting down, this becomes True to stop further action from the robot
self.flipper

The flipper uses the large motor arm of the lego set and is connected to port A of the EV3 Brick. Hence, this detects whether the connected motor is a large sized motor and assign its operation to port A

self.turntable The turntable uses the large motor arm of the lego set and is connected to port B of the EV3 Brick. Hence, this detects whether the connected motor is a large sized motor and assign its operation to port B
self.colorarm The colorarm uses the medium motor arm of the lego set and is connected to port C of the EV3 Brick. Hence, this detects whether the connected motor is a medium sized motor and assign its operation to port C
self.color_sensor Assigns the various operations of the color sensor
self.color_sensor.mode Sets the mode of the color sensor to read the raw values of RGB components
self.infrared_sensor Assigns the various operations of the infrared sensor
self.init_motors() Initializes the various motor arms by turning them on and off, as well as resetting them
self.state A list of Rubik's cube notations
self.rgb_solver Default value is None. When program runs, the result of the imported RubiksColorSolverGeneric()method will be assigned to this variable
signal.signal(signal.SIGTERM, self.signal_term_handler) Initializes a signal handler to stop the program and the robot if a termination event has occurred
signal.signal(signal.SIGINT, self.signal_int_handler) Initializes a signal handler to stop the program and the robot if an interrupt event has occurred
filename_max_rgb Filename of the txt file generated after running of the calibrate_white.py