Table of Contents
This is an experiment in using neural nets for image recognition in IMatch, and also an experiment in how such a workflow could be implemented. The application consists of an IMatch "app" that runs in IMatch, and a Windows-native application that runs TensorFlow for the actual image recognition.
The IMatch app consists of three modules: Labeling images, Reviewing suggested labels, and manage the Models.
A Model consists of an image recognition net and workflow parameters - where it gets its training data from, what categories it should label images with, and how the review process works.
This is experimental-grade, not production-grade software. The purpose of this software package is to explore and play with image recognition by neural nets in IMatch, not to provide a product. It is offered here in the hope that others may want to see for themselves what the result is on their data. The software is offered under the Apache 2.0 License[d], which, to quote the license means that it is offered WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
In short: Make a backup of your data before using this. (And perhaps use it on a separate IMatch database.)
The following applies to the Windows zip package.
Unzip the package.
imatch-app/contains the IMatch application and should be placed where you keep your IMatch apps - typically
tf-server/contains the application used to run TensorFlow. Put it anywhere you like, for example
Since this is the first Python-based application I distribute, I may have missed some dependencies for the Python interpreter. If you have problems, try installing Python 3.6.4[g]. It should give you any dependencies you need.
3. Quick Start
Download and install the application.
flower_photos.tgzin a folder of your choice.
Add that folder to your IMatch database.
Create a category hierarchy with one top category and subcategories matching the sub-folders of the
flower_photos.tgzpackage. For example
ImageLabel|dandelion, and so on.
Assign the photos in each sub-folder to the corresponding subcategory.
Start the Imception application in IMatch.
You will be asked to provide the directory where you installed the executable. Do so, then press OK. (If you copied the
tf-server/folder as in the example above, enter C:\Program Files\nu.sutic.imception.)
Click on Create New, and input the name Flowers. (The folder should be
In Categories, enter a search term that will match the categories created in the above step. For example, ^ImageLabel|.
Press Test and verify that the list includes all the categories with flowers that you created, but no other categories.
In Review Category, enter AI-review.
In Review Substitutions, enter ^,AI-assigned|
In Approval Substitutions, enter ^AI-assigned\|,
Press Test below Approval Substitutions. The message box should show the following flow:
That is, images found to belong to the training category
ImageLabel|daisyare first normalized to
ImageLabel|daisy(no change), then placed in the category
AI-assigned|ImageLabel|daisyfor review, and when approved put in
Press Train, and go for a cup of coffee or something. A dialog box will pop up and inform you of the training progress. It takes about 30 minutes, and the application will need Internet access if this is the first time you train a net, as it will download the un-trained model.
Switch to the Label module.
Go to the top category you assigned flowers to, and select a bunch - say 10-20.
Press Preview. A single image will be labeled and you will see the suggested labels along with confidence percentages. Green means that the label was above the confidence threshold, meaning that the model would have suggested you label the image with it.
- Now it's time to label some images for real. Press Label All Selected Images, and wait for it to complete.
Switch to the Review module.
Go to the
AI-reviewcategory. You will see all flower images that the net could produce suggestions for.
Select the first image.
Click on the label to move it from the thanks-but-no-thanks set to the approved set.
Press Apply to apply all approved categories and discard all rejected categories. The image is also removed from the
Repeat for all files in the
The workflow is based on the net doing an best effort at labeling images, and then have an optional review phase where those labels are either approved or rejected.
Each step in the workflow consists of either assigning images to a category or unassigning them from one. For example, when recognizing labels, the recognized labels will be categories to which the image will be assigned. When a label for an image is approved, the image is unassigned from the corresponding category and moved to another category that indicates "approved label".
For example, a model may recognize the label
banana. When it thinks an image has a banana in it, it will assign the image to the category
review|banana, to indicate that it has seen a banana, but this requires review. If the label is approved, the image will be unassigned from
review|banana and put in the plain
For the application to figure out all these category names, it uses substitutions. First, the model is trained on training categories. These are images that are known to be correctly labeled - for example "Training|apple" contains all images that are known to contain apples. These are the raw categories that come out from the image recognition.
The second step is a set of normalizing substitutions. Here we can, for example, strip the
Training| prefix. Then we apply review substitutions to figure out which category the image should be in when we have detected a label, but that label is not human-approved. We can, for example, prepend a
Review| prefix - making our detected label
Review|apple. (If we don't want to review anything, we can simply not make any changes at all here - the image goes straight to the final category.)
Finally, when the label has been reviewed, a set of approval substitutions tells us the category where images that ended up in
Review|apple should go when they are approved. In this case, we can remove the
Review| prefix, putting the image in the
The Label module is where you test your model and label images.
The Review module is where you review the results of labeling images.
You can review one or more images at the same time. In the case of a single image, approving or rejecting labels work just as you would expect: approved labels are assigned to the image, rejected labels are removed. When you review more than one image, the approval or rejection applies to all images - an approved label will be applied to all images for which it has been suggested, and a rejected label will be removed from all images for which it has been suggested. If you have two images,
banana.jpg, and the suggestions
banana.jpg, approving both labels will assign
banana.jpg - but not apply
apple.jpg or vice versa.
The Models module is where you configure the recognition models and their workflow parameters.