Support Ukraine

What do you get when you combine IMatch[a] with Google's Inception V3[b] net, modified for multiple labels[c]? A fantastic playground for image recognition!

​1. Overview

This is an experiment in using neural nets for image recognition in IMatch, and also an experiment in how such a workflow could be implemented. The application consists of an IMatch "app" that runs in IMatch, and a Windows-native application that runs TensorFlow for the actual image recognition.

The IMatch app consists of three modules: Labeling images, Reviewing suggested labels, and manage the Models.

A Model consists of an image recognition net and workflow parameters - where it gets its training data from, what categories it should label images with, and how the review process works.

​2. Downloads

Download IMception 1.0

Source code[e] | Repository[f]

​2.1. Installation

The following applies to the Windows zip package.

1. Unzip the package.

2. The subdirectory imatch-app/ contains the IMatch application and should be placed where you keep your IMatch apps - typically C:\ProgramData​\photools.com​\imatch6​\webroot​\user.

3. The subdirectory tf-server/ contains the application used to run TensorFlow. Put it anywhere you like, for example C:\Program Files​\nu.sutic.imception.

Since this is the first Python-based application I distribute, I may have missed some dependencies for the Python interpreter. If you have problems, try installing Python 3.6.4[g]. It should give you any dependencies you need.

​3. Quick Start

1. Download and install the application.

2. Download flower_photos.tgz[h]

3. Unpack flower_photos.tgz in a folder of your choice.

4. Add that folder to your IMatch database.

5. Create a category hierarchy with one top category and subcategories matching the sub-folders of the flower_photos.tgz package. For example ImageLabel​|daisy, ImageLabel​|dandelion, and so on.

6. Assign the photos in each sub-folder to the corresponding subcategory.

   

7. Start the Imception application in IMatch.

8. You will be asked to provide the directory where you installed the executable. Do so, then press OK. (If you copied the tf-server/ folder as in the example above, enter C:\Program Files​\nu.sutic.imception.)

9. Click on Create New, and input the name Flowers. (The folder should be flowers.)

10. In Categories, enter a search term that will match the categories created in the above step. For example, ^ImageLabel|.

11. Press Test and verify that the list includes all the categories with flowers that you created, but no other categories.

12. In Review Category, enter AI-review.

13. In Review Substitutions, enter ^,AI-assigned|

14. In Approval Substitutions, enter ^AI-assigned\|,

15. Press Test below Approval Substitutions. The message box should show the following flow:

    • ImageLabel​|daisy → ImageLabel​|daisy → AI-assigned​|ImageLabel​|daisy → ImageLabel​|daisy

    • ImageLabel​|dandelion → ImageLabel​|dandelion → AI-assigned​|ImageLabel​|dandelion → ImageLabel​|dandelion

    • ImageLabel​|roses → ImageLabel​|roses → AI-assigned​|ImageLabel​|roses → ImageLabel​|roses

    • ImageLabel​|sunflowers → ImageLabel​|sunflowers → AI-assigned​|ImageLabel​|sunflowers → ImageLabel​|sunflowers

    • ImageLabel​|tulips → ImageLabel​|tulips → AI-assigned​|ImageLabel​|tulips → ImageLabel​|tulips

    That is, images found to belong to the training category ImageLabel​|daisy are first normalized to ImageLabel​|daisy (no change), then placed in the category AI-assigned​|ImageLabel​|daisy for review, and when approved put in ImageLabel​|daisy.

16. Press Train, and go for a cup of coffee or something. A dialog box will pop up and inform you of the training progress. It takes about 30 minutes, and the application will need Internet access if this is the first time you train a net, as it will download the un-trained model.

    

17. Switch to the Label module.

    

18. Go to the top category you assigned flowers to, and select a bunch - say 10-20.

19. Press Preview. A single image will be labeled and you will see the suggested labels along with confidence percentages. Green means that the label was above the confidence threshold, meaning that the model would have suggested you label the image with it.

    

20. Now it's time to label some images for real. Press Label All Selected Images, and wait for it to complete.

21. Switch to the Review module.

22. Go to the AI-review category. You will see all flower images that the net could produce suggestions for.

23. Select the first image.

    

24. Click on the label to move it from the thanks-but-no-thanks set to the approved set.

    

25. Press Apply to apply all approved categories and discard all rejected categories. The image is also removed from the AI-review category.

26. Repeat for all files in the AI-review category.

​4. Workflow

The workflow is based on the net doing an best effort at labeling images, and then have an optional review phase where those labels are either approved or rejected.

Each step in the workflow consists of either assigning images to a category or unassigning them from one. For example, when recognizing labels, the recognized labels will be categories to which the image will be assigned. When a label for an image is approved, the image is unassigned from the corresponding category and moved to another category that indicates "approved label".

For example, a model may recognize the label banana. When it thinks an image has a banana in it, it will assign the image to the category review|banana, to indicate that it has seen a banana, but this requires review. If the label is approved, the image will be unassigned from review|banana and put in the plain banana category.

For the application to figure out all these category names, it uses substitutions. First, the model is trained on training categories. These are images that are known to be correctly labeled - for example "Training|apple" contains all images that are known to contain apples. These are the raw categories that come out from the image recognition.

The second step is a set of normalizing substitutions. Here we can, for example, strip the Training| prefix. Then we apply review substitutions to figure out which category the image should be in when we have detected a label, but that label is not human-approved. We can, for example, prepend a Review| prefix - making our detected label Review|apple. (If we don't want to review anything, we can simply not make any changes at all here - the image goes straight to the final category.)

Finally, when the label has been reviewed, a set of approval substitutions tells us the category where images that ended up in Review|apple should go when they are approved. In this case, we can remove the Review| prefix, putting the image in the apple category.

​4.1. Label

The Label module is where you test your model and label images.

​4.2. Review

The Review module is where you review the results of labeling images.

You can review one or more images at the same time. In the case of a single image, approving or rejecting labels work just as you would expect: approved labels are assigned to the image, rejected labels are removed. When you review more than one image, the approval or rejection applies to all images - an approved label will be applied to all images for which it has been suggested, and a rejected label will be removed from all images for which it has been suggested. If you have two images, apple.jpg and banana.jpg, and the suggestions Apple for apple.jpg and Banana for banana.jpg, approving both labels will assign Apple to apple.jpg and Banana to banana.jpg - but not apply Banana to apple.jpg or vice versa.

​5. Models

The Models module is where you configure the recognition models and their workflow parameters.