In this article, I would like to show you how to build a custom model using AI builder for form processing and use it in PowerApp.
What is Form processing model?
As per Microsoft docs,
Form processing lets you read and save information from standard documents such as invoices or tax documents. When you automate this process, you can save valuable time by reviewing, extracting, organizing, and saving the data automatically by using Power Automate and Power Apps.
In other words, Form processing model processes forms or documents that have data in key-value pair. It extracts the document data and makes it available for any further use. Since Form Processing is a custom model, it must be trained before it can be used. This training is done by using existing sample documents for analysing data. Once trained, new documents may be input into the model to extract information. The accuracy of outcome of this model depends on the sample documents used for training the model.
Model requirements and limitations
The details around model requirements and limitation are available on Microsoft Docs. However, following are some of the highlight points:
Language
Latin (English) alphabet
Format
- JPG
- PNG
Size
Maximum file size for all training documents: 50 MB
Maximum PDF length: 500 Pages
JPG and PNG between 530 × 100 and 4200 × 4200 pixels
Unsupported Types
Form processor was created to extract text items usually typed or written into a field, like first name, last name, date of birth etc. The AI builders form processing model does not process all the controls placed in a form. Some of these unsupported control types include the following
- Complex tables (nested tables, merged headers or cells, and so on)
- Check boxes or radio buttons
- Signatures
- Fillable PDFs
Model output
The most interesting form processing model output from an input document or image is as follows:
Results
A list of values extracted from the input document or image by AI the model height.
What are we going to build?
We will build a custom model for processing invoices. We will then add this custom form processing component into a new canvas PowerApp. Here is the result of our trained and tested model in PowerApp for a given invoice document.
Building the custom Form Processing Model
Please follow the steps below to create and train your custom form processing model:
Enable AI Builder: The very first thing to ensure is that AI builder is available and enabled for the environment that you are working in.
In order to do this, sign in to Power Platform Admin center and go to Environments -> <select environment> -> Settings -> Features.
In the Features page, enable the Create AI Builder preview models if it is not is enabled.
Create Form Processing AI Model : Sign in to PowerApps maker portal and then go to ‘AI Builder’ -> Build -> Form processing.
When the screen as shown below opens up, give a suitable name for the model.
Remember that AI Builder Form Processing is a premium feature.
We can clearly see that a minimum of 5 sample documents are required to train this model. In the next step, we will make use of the sample documents provided by Microsoft for this purpose. For now, click the “Create” button to go to next screen.
Download sample documents: As shown in the screen shot below, click on “Get Sample Documents” link or button to download the sample documents from Microsoft. We will use these sample documents to train our model.
Prepare sample documents: Please ensure that the images or documents that you upload meet the requirements specified here. The model will fail with the errors if the documents do not meet the requirements. Here is a screen shot of sample invoice document in PDF format that we will upload for training our model.
We will use the five sample invoice PDF files as sample data for our Form processing model.
Add Documents: Click on ‘Add Documents’ button and select all 5 sample pdf files for upload from your local storage. As you can see, the source for these documents may be SharePoint or Azure Blob Storage.
Upload Documents: Click on ‘Upload Documents’ button to upload all documents you just added.
Click on “Close” button once the documents have been successfully uploaded.
Upload Documents: Click on ‘Analyze’ button to save the model and start analyzing the documents.
This process may take some time depending upon how many documents have been uploaded for training the model.
Select form fields: Once the analysis is complete, we need to select data that we want to extract from the document.
Click on the tile to open our form for selecting fields.
We can either select all the fields or just the ones that we need for our purpose. Here we have selected the Invoice field to start with. Click all the other fields except the table showing item, quantity, rate and amount. Click on ‘Done’ button to save the field selection for model. Click on ‘Next’ button to train the model using uploaded sample documents.
Train, Test and Publish Model: Click on ‘Train’ button to train the Form processing model.
Once again, depending on how may documents have been uploaded and how many fields have been selected, the training process may take some time to complete.
Once the training is complete, click on the “Go to Details page button”.
You can see a warning that says “The model isn’t published yet”. It is important to publish the model before it can be used no apps and flows.
On this page, we can quick test and publish our trained model now. We can test the model by using ‘Quick test’ button. To conduct this test, simply upload a new sample document and check if it extracts the data from the fields we selected during model training. Repeat testing with few more documents and ensure consistent results are obtained.
Click on Publish button to publish the model when you complete your testing.
Congratulations…! You have successfully built your form processing AI model.
Download Form Processing PowerApp
Download the sample Power App from here and import it into your environment. Note: Use #ModernWork as the password to access the sample Power App package.
Edit the PowerApp in designer to explore the controls and code associated with this simple one screen app.
Conclusion
In this article we saw how to use Form Processing AI model of AI builder to process our images or documents, analyse them, and extract required data from them. Once this data becomes available, we can consume it in, for example Microsoft Power Automate, to further fulfil our business requirements.
While the possibilities are endless, I hope this article encouraged you to start exploring AI builder capabilities. This solution can be easily extended using Power Platform and other Office 365 services, and I would like to hear your thoughts on this.
Hope you liked this article! Please share your valuable comments and feedback with me on LinkedIn.