Introduction

In August 2022, elections were held in Kenya to decide the next leaders, the main one being the presidential elections.

The way elections work is, that there are 46,229 polling centres where elections take place. After the voting period ends (typically around 5pm on election day) the polling station officials commence counting of votes.

They then record the tally manually on a physical form. This form is referred to as Form 34A.

The forms are then uploaded to a  public portal where anyone can download and view them.  See sample below

This project uses AI to read the Form 34As, extract the tally from the handwritten digits and display the results on a public portal. This was phase one of the project.

As you can already guess, this isn't going to be very accurate and needs some way of validating the AI results. Here comes phase two, to validate the AI's  output; I resulted to crowsource the verification of the AI results.

This was done via public facing portal here

Phase One: Using AI to read data from the form

This was probably the hardest part. Firstly I tried feeding the form directly to already trained OCR extractors. I tried both AWS Textract and GCP Document AI but did not get the best results.

The output I was getting was just RAW text with also of "noise" hence extracting the actual numbers from the OCR ouput was really hard.

From the example Form34A above, our area of interest is essentially the top left quadrant with these values:

As you can tell running OCR on the whole form isn't useful as we'll get a bunch of unwanted text "noise" making it harder to extract the final numbers that we want. Here is a sample output

For this specific form, you could easily use REGEX to extract the numbers, but more often than not the regex expression would not work, since the numbers that we are interested in would appear in different positions in the output of different forms.

So I decided manually crop the image and just feed this section to AWS Textract.

This yielded much better results. The output of this was the following:

No. of Valid Votes Obtained
199
091
000
001

But now here comes the question. How do we crop 46,000 forms?

We could build our own object detection model from scratch (takes a very long time to setup) or we could use a pre-trained one.

YOLO

You Only Live Once , I mean "You only look once" is a state-of-the-art, real-time object detection system. That is extremely fast, it is typically used in for identifying real-world objects, like in Autonomous driving, Security and Surveillance etc.

The basis of how it works, is you give it an image and it gives you a bounding box (coordinates) where it "thinks" the object is found. If you want an in-depth guide on how YOLO works see this video below by Aladdin Persson

Since YOLO is trained to identify real-world objects (cats, dogs, people...), it's not going to work out of the box for our form. We need to retrain it, not from scratch but train it on new information (transfer learning).

For this we are going to use the latest YOLOv5 and another website called makesense.ai  . We will be using makesense.ai (an image annotation tool) to create coordinates of our crop region. In my case, I did about 100. (70 for training and 30 for testing)

The way the model is evaluated is based on a simple formula called IoU (Intersection Over Union).

The formula is  (Intersection Area) / (Union Area) . In the image below the green box represents our true bounding box and the red represents the AI prediction while training.

Hence, values the closer the IoU value is to 1, the better.

So after training the model for a while I was able to get an IoU of about 0.9 which is really good.

The next step is to crop the images using the coordinates given. Luckily YOLOv5 has a handy command for doing that. Now that we have the cropped images, we can now feed them to AWS Textract to get the text predictions. Once this was done it was a simple REGEX to get the actual numbers for each candidate and create a tally. When I created the tally I noted that the tally was quite off by a lot, for example, I got a tally in the billions which is not possible. This was due to some text predictions looking like this "No of Obtained Votes 677000001 002", I could try using a better regex expression but there could be a million edge cases, which brings us to phase two.

Phase Two : How to deal with wrong predictions.

We need a way to verify to correct wrong predictions and also fill missing data that might have been missed by the AI. So I decided to create a simple portal to crowdsource the verification part.

When users submit the form above, the system picks the value with the highest frequency, ensuring that even if someone enters a wrong value, the tally will more or less be accurate since the assumption is that most people are honest

In hindsight, I might as well have started with this phase, and skipped the whole complex AI part. Though, a huge advantage is that during the verification process most of the values are already prefilled, and all that the user has to do is to click the Save/Verified button.

Within a few days, I was able to get about 4,000 users who verified the results.

Findings

About a week after the election day, the results were officially announced and "our results" weren't that far off. Comparing the crowdsourced results vs the official results  we had a margin of error of about 0.3-0.4%  which is not bad at all. I'm quite grateful to all who participated in the verification.

For the AI OCR vs the crowd-sourced results, I had exact values predicted 97% of the time, also not bad.

What's next? The next election is a bit far, though I have the data available for anyone interested in training a better OCR AI, especially since this whole project was done in hurry(in about 4 days) and had to deliver before the official results were out 😂😂.

If you are interested here is the repo with the data. Your task is to train an AI and see if you can achieve better results. Hoping to see a custom pipeline from downloading images, cropping then OCR. Especially, without using cloud-based solutions for the OCR part. To start you can checkout Keras OCR Captcha model. Create an issue on the repo to share and discuss your results. I have truncated the data to prevent cheating 😂😂. May the best AI win