Background

As you probably know, the General Data Protection Regulation, better known as GDPR, is being adopted by the European Union and comes into effect on 25 May 2018. Although this is a European regulation, if your organization retains any personally identifiable information (PII) relating to clients or individuals who are located in Europe then it affects you.

We published a Blog Post earlier this year discussing what the changes are and how they can affect you. Here's a link to that Blog Post for reference.

One important and challenging aspect of the GDPR is the need to be able to answer Subject Access Requests. Starting in May, any European citizen (a data subject) will have the right to obtain confirmation as to whether or not their PII is being processed, where, and for what purpose. They will also have the "right to be forgotten." In other words, you may need to search for all occurrences of a person's PII, and optionally ensure that it is all erased.

This Blog Post will show how you can use Teamstudio Configurator's "Search" functionality in order to create an IBM Notes Application for searching for user information across multiple databases and provide you with a report of what was found.

Normally with Configurator you would perform searches on design elements in a single Notes application and optionally replace anything that was found with alternate code. That is the most common use of Configurator. But in the following example we will detail how you can utilize Configurator API calls to perform searches on the data contained within multiple Notes applications using a simple interface and creating a report on the search results.

Prerequisites

Before you begin you will need to make sure you have the following prerequisites in place:

IBM Domino Designer V8.5.3 and above
Access to a Domino server to create the new Notes Application
The latest version of Teamstudio Configurator (Edition 32.1x as of the time of writing)
Access to the IBM Notes Applications where client data resides
Time, patience and an interest to learn

Step 1: Initial setup of database

This is a simple process of creating a new IBM Notes Application with the following design elements:

Form for main UI (the "Wizard")
Form for viewing reports
View for seeing previous reports
View for storing scan requests
Frameset for displaying main UI Form when the database is opened
Agent for performing the search and reporting on the results

Step 2: Setup the "Wizard" form

For this step we will create a very basic Form with only the fields needed to use the values entered to perform the search using Configurator.

Below is a list of the Properties for the Form and the Fields required:

FORM:
Name = Wizard
Alias = Search

FIELDS:

NAME	ATTRIBUTES	DESCRIPTION
Server	- Dialog List	This will be used to select a Server to search on
Paths	- Text - Multi-value - Separate values with new line	This will be a list of the databases including their paths that you want to search
Criteria	- Text	This will be the Search Criteria

Also, we will need a Button that will run the code. Label the button "Run Search".

Here is an illustration of how the basic Form should look:

Step 3: Interface functions

We need to include some code on the Wizard form to make life easier for users and for us as developers.

The first thing we want to do is make it simple for the user to select the server that they want to search. The easiest way to do this is to use a Dialog List field that pulls in the names of all IBM Domino servers from the Domino Directory. For simplicity's sake I have included the @Formula to use below:

server:=@Subset(@MailDbName;1);
db:="names.nsf";
view:="Servers";
column:=3;
list:=@DbColumn("":"cache";server:db;view;column);
@If(@IsError(list);"- Unable to locate list of Server names -";list)

Step 4: Setting Up the remaining design elements

Now that we have the Wizard UI form in place, we need to make a few more design elements to make sure we have everything we need.

First we will need two views, one for the documents that will contain the information from the Wizard UI form and another to display the results of the searches.

For the first view, create it with these settings:

Name: Searches
View Selection: SELECT Form="Search"

COLUMNS:

Column Title	Column Value	Description
Status	@If(Status="";"Pending"; Status)	This will display the "Status" of the Search Request. We will change this value during the Search Process.
Date Requested	@Created	The date and time when the Search Request was made
Requestor	@Author	The person who created the Search Request
Search Criteria	Criteria	The search criteria from the Wizard

For the second view, create it with these settings:

Name: Reports
View Selection: SELECT Form="Memo"

COLUMNS:

Column Title	Column Value	Description
Date Searched	@Created	The date and time when the Search was made
Requestor	@Author	The person who performed the Search
# DBs Searched	DBsSearched	This is the number of databases that the search was performed on. This will be set during the Search Process.
# Search Hits	SearchHits	This is the number of occurrences of the Search Criteria being found in the databases searched. This will be set during the Search Process.
Search Criteria	Criteria	The search criteria from the Wizard

Second, let's setup how the application will be launched. This requires a Frameset to be created with these simple settings:

Name: Wizard

# of Frames: 2

Alignment: Large Top, Small Bottom

Attributes of First Frame:

Type: Names Element – Form
Value: Wizard

Attributes of Second Frame:

Type: Named Element – View
Value: Reports
Frame Border Attributes:
Caption Formula: “Past Searches”
Show: Both
Align: Top

Once the Frameset is in place you can adjust the IBM Notes Application Launch Options to automatically open this Frameset to display the Wizard and the past search reports.

Third and finally we need to create a simple Form for displaying the Search Results Reports.

Below is a list of the Properties for the Form and the Fields required:
FORM:
Name = Report

FIELDS:

NAME	ATTRIBUTES	DESCRIPTION
Server	- Text - Computed When Composed	This is the Server that was searched
Paths	- Text - Multi-value - Separate values with new line - Computed When Composed	This is the list of the databases including their paths that were searched
Criteria	- Text - Computed When Composed	This is the Search Criteria
DBsSearched	- Number - Computed When Composed	This is the number of databases that were searched
SearchHits	- Number - Computed When Composed	This is the number of search "hits", or items found, based on the Search Criteria
Description	- Text - Computed When Composed	A simple description of the Search Results report
Body	- Rich Text - Computed	The Rich Text Field which will contain the results of the search, including a simple descriptor for the Document located and a link to the Document itself

All of the above fields will have a value that references itself.

Here is an illustration of how the basic form should look:

Step 5: The Teamstudio APIs

Before any of the above design elements can be used we need to add the Configurator API design elements as well. These are supplied with every installation of our Teamstudio Notes Tools. They are contained within a series of LotusScript Script Libraries and can be found in the Teamstudio Reports template that is deployed with the Teamstudio Notes Tools called "tmslogs.ntf". The Teamstudio Reports template can be found in the Notes Data directory on your computer.

To retrieve the design elements, simply follow these steps:

Open "tmslogs.ntf" in IBM Domino Designer
Navigate to the Script Library design elements
Locate and copy the Script Library called "Configurator" onto your clipboard (select the Script Library and then press "CTRL-C"
Open your new IBM Notes application and navigate to the Script Library design elements
Paste the "Configurator" Script Library into your IBM Notes Application (CTRL-V)

That's it! Your new IBM Notes Application now has all of the code needed to run Configurator from your own LotusScript code.

If you would like to read up on what API calls you can make take a look at this Knowledgebase Documentation on the Configurator API. We will be referencing this documentation (and others) once we start using the Configurator API calls.

Step 6: Tying It All Together – The Code

Now that we have all of the pieces in place, it is time to piece it all together.

The first thing we will need to do is create the code that will be placed into the button that we created on the Wizard form. This code will first need to validate that the fields in the Wizard have values and then call the agent that we will be making shortly. For this we can use @Formula language since it is simple to code.

Below is the code that can be placed into the button:

@If(Server="";
    @Return(@Prompt([Ok];"No Domino Server Selected";"You must select a Domino server to search before continuing"));
    @Success); 
@If(Paths="";
    @Return(@Prompt([Ok];"No Database(s) Listed"; "You must list at least one database to search before continuing"));
    @Success);
@If(Criteria="";
    @Return(@Prompt([Ok];"No Search Criteria"; "You must indicate what you want to search for before continuing"));
    @Success); 
@Command([FileSave]); 
@PostedCommand([ToolsRunMacro];"(Perform Search)")

Now let's start creating the agent that will perform the actual search using the Configurator API.

First let's start with the Agent Settings:

Name: Perform Search
Type: LotusScript
Trigger: On event – Agent list selection
Target: None

In the Agent Options section add a reference to the Configurator API Script Libraries as shown below:
Use “Configurator”

We’ll need some Global Declarations that will be used in the Functions and Subroutines. Here’s what we need:

Dim session As NotesSession       ' current Notes Session
Dim db As NotesDatabase           ' current database 
Dim wizardDoc As NotesDocument    ' this is the document that was saved when the Wizard was filled in 
Dim server As String              ' this will be used to store the server to search 
Dim paths() As String             ' this will be an array of the database(s) to search 
Dim criteria As String            ' this will be the search criteria 
Dim dbCount As Integer            ' this will be the count of how many dbs were searched 
Dim resultCount As Long           ' this will be the total number of search "hits" 
Dim results() As Variant          ' this is an array of documents where we can store the search results 
Dim report As NotesDocument       ' this will be the Search Results Report document 
Dim body As NotesRichTextItem     ' this will be the content of the Search Results Report

We now need to create the function that will retrieve the information that was entered into the Wizard. This will access the view that we created to display the search requests and load the values from that document into the global declarations that we created. Here’s a link to download this function to see what it should look like.

Next, we need the function that will perform the search itself. This is where the calls to the Configurator API will come into play.

To familiarize yourself with what the Configurator API can do be sure to visit our documentation on the Configurator API.

The function will need to use the information stored on the search request document that we retrieve in the previous function that we created. Once we have that information we can create a loop to go through that information and perform the actual search using Configurator.

The Configurator API call can only run in serial, which means that for each IBM Notes Application you listed to search, the Configurator Search call will be invoked. Now there are two side effects to this, one we can deal with, one we can’t:

For each time the Configurator Search call is made a new Configurator Report document is created in the Configurator Report database that we will be indicating in the call. Although this is extra overhead we can deal with this and the results it contains within our LotusScript code.
When the Configurator API call is made, the normal Configurator Status window is displayed. This is nice to have so that you can see the progress, but the one caveat to this is that for each Notes database that you are searching you also need to click the “OK” button on that status window to continue. This also means that this has to happen in a foreground process which also means that there needs to be someone watching for when the “OK” button needs to be clicked. This is the one side effect that currently we cannot avoid (although I know our CTO’s cellphone number and will be calling him about this after I have completed this blog post :) ).

So, even though we have these side effects, we can still get this app running. We will need a new function to perform the search, so let's create one called "PerformSearch". This new function will have a series of variables to capture and use the following information:

A counter for the loop
References for retrieving the Configurator Report db and the Report document(s)
A variable for the Configuration Report Title (this will also be used to locate the Report document)
Various others for capturing the results of the search from the Configurator API call

Again, to make your life that much easier, here’s a link to download this function to see what it should look like.

Ok, so now we have a function that will loop through all of the IBM Notes applications that you want to search a text string within. We made the call to the Configurator API and passed in some parameters to control how the Configurator application will run, specifically:

CONFY_RUN_SILENT: This will run Configurator without presenting the initial dialog. We don’t need that dialog since we are pre-populating the values that Configurator requires to run
CONFY_SEARCH_DATA: This will ensure that we are only searching through Notes Documents and not the design of the IBM Notes application
CONFY_RUN_NO_SIGN: This will skip the signing option that can occur when Configurator runs. The last thing we want is for the Notes documents to be re-signed

There are numerous other options that can be passed into the Configurator API call but they are all to do with searching through design elements so we wouldn’t need them for this process. You can review all of the various parameters and flags on our Configurator API documentation page.

The returned result from Configurator is actually only a Notes document which reports on the results of the search. The Notes document contains all of the information about the search which was performed, including the Note IDs for the Notes documents with the search criteria found, but with one caveat: the results are stored within a Rich Text field.

This poses a problem since we want to retrieve the Notes documents that contained the Search Criteria. But not to fear, with LotusScript there is always a way around everything. For this caveat we need to create a function that will parse through the text that is contained within the Rich Text field and extract the Note ID values for the Notes documents that Configurator found the search criteria on.

Although this sounds a bit tricky it is actually very simple as long as you know what you are searching for. In this case there is a consistency to what is placed into the Rich Text field that is contained on the Configurator Report. For each discovered Notes document there will be a line in the report similar to this:

Document 'Untitled Contact' (0x00004952)

To deconstruct this line we need to know what it represents:

Document: indicates that the search located a Notes document
Untitled: this would normally be the "Window Title" if it was set for the Notes document
Contact: this is the value in the Form field
(0x00004952): this is the Note ID for the discovered Notes document, (kind of). The beginning part “(0x” is actually added for our internal purposes and, when removed form the value, the remainder of this value represents the Note ID and we can use it to locate the actual Notes document.

So what can we do with this? We need to create a function that can parse through the Rich Text and extract only the Note ID values so we can locate the Notes documents that they represent. Using some standard LotusScript calls we can go through the Rich Text and extract those values very simply.

And once again, here’s a link to download this subroutine to see what it should look like.

This subroutine traverses the content of the Rich Text, retrieves the Note IDs and places them into an array. Then we look into the Notes database that was searched and locate the Notes documents referenced in the Configurator Report and place them into an array of Notes documents.

But now what are we going to do with the results? Well we need to report on them in a way that the user can easily see the Notes documents and perform any additional tasks on them, such as delete them if your customer has requested this as part of their GDPR rights. This means we need to create one last function to create our report. Since we have a collection of all the Notes documents this is very straightforward. The Function will need to perform the following tasks:

Create our Report document if is does not already exist
Add values to our Report so that we know what was searched for and where
Increment some reporting values (such as number of occurrences of the search criteria)
Add values into a Rich Text field so that we can view the results

Pretty straightforward stuff. How straight forward? Well here's a link to download this function to see what it should look like.

The final step in the agent is calling this code. That's very straight forward as there are only four lines of code to make in the "Initialize" event:

Sub Initialize
     Set session=New NotesSession
     Set db=session.Currentdatabase
      If Not RetrieveSearchRequest() Then Exit Sub
      If Not PerformSearch() Then Exit Sub
  End Sub

This will format the results in a fashion that will be usable by the end user who requested the search. Here's an example of the output:

Step 7: What To Do With The Results

So now you have a report which indicates where your search criteria was found in specific IBM Notes applications and documents. Now what?

Depending on your GDPR policies and what you intend on doing with the results is truly up to your organization. With the code supplied within this blog post you can now use the resulting output to enhance this Notes application to provide advanced functionality, such as:

Format a report that can be provided to the client
Enhance the output to be more specific to your applications
Change the functionality to target known Notes databases that contain client information
Create a routine that will also delete the Notes documents with the client information

… and the list goes on an on. You will need to determine what your GDPR needs will be and then use the code within this blog post to do what you need. As one final parting gift, I have compiled all of the above code into a reusable IBM Notes application that you can use as your starting point. Here’s a link to download the IBM Notes Application. Happy coding!