In the world of software procurement it's one thing to buy software from a company that looks to have a good product, but it's completely different when the company that makes the software also uses it internally for the same reasons you do.

That's exactly what we do here at Teamstudio. Most of our IBM Notes developer tools also have corresponding IBM Notes databases that contain many, many reports. We use Teamstudio CIAO! for source code control and Teamstudio Build Manager to create the shippable IBM Notes databases to our clients. We also use the remainder of our Notes Developer Tools line of products to ensure best practices are in use while building these IBM Notes databases.

With the addition of our latest product, Teamstudio Export, we have chosen to use it for archiving our old IBM Notes databases in much the same way that our clients use it as well. On this blog post is a summary of the steps that I took to archive our old IBM Notes-based CRM system and the lessons that I learned.

Step 1: Preparation

This is a crucial step. In order for Teamstudio Export to run successfully you need to make sure there are no problems within your IBM Notes databases.

I started by performing a series of maintenance tasks. I first made new replicas of the IBM Notes databases onto an IBM Domino server that is "closer" to my computer (that being on the same local area network). Before making the new replicas I also ensured that I had enabled "Full Access Administration" within my IBM Domino Administrator client when connected to the IBM Domino server where the existing IBM Notes databases reside. This will ensure that the new replicas have all of the documents, regardless of whether they contain security fields or not.

By creating these new replicas many of the known problems within an IBM Notes database "should" be corrected. Having been in the IBM Notes/Domino industry since 1993, I'm always super careful about stuff like this. So I also performed the following maintenance tasks on all of the IBM Notes databases before starting:

Fixup
Updall
Compact

These are all server tasks that can be started on all Notes databases in a given directory. These will ensure that there is no corruption left in the databases. Here’s a great article from IBM about fixing corrupted IBM Notes databases.

Step 2: Creating the Teamstudio Export Output

Now that the IBM Notes databases are free of any problems we can begin the process of creating the Teamstudio Export output. You will quickly realize that this is the easiest of all the steps to perform!

Once you launch Teamstudio Export you will be presented with a screen similar to the one below (this is assuming that Teamstudio Export has already been installed and configured):

If you don't see your newly created replicas you will need to refresh the list of Notes databases that are displayed in Teamstudio Export (from the menu select "Window->Refresh").

Once this has been refreshed the next process is very simple. Once I located the databases that I needed to archive I selected each one at the same time by holding down the CTRL key and clicking on the rows of information (as shown below):

Once the databases are selected the archiving process is as simple as right-clicking one of the selected rows and then selecting "Archive" from the menu that is displayed. This is the time to go get a coffee since the overall accumulated size of the Notes databases is just over 3GB in my case. Since the archive process is running on the Domino server on my LAN, the process will take approximately 2 hours. Teamstudio Export runs in its own process so you can still use your Notes client for your regular daily work.

The three most common questions about the Teamstudio Export output files are:

How long does the process take?
How large are the output files?
What affects the speed of creating the output files?

Knowing that these are very important questions for clients I recorded all of this information during my archiving process. Below are the statistics for the seven Notes databases that I used during this project:

All sizes are in Megabytes (MB), all times are in Minutes
Hardware specifications used for Teamstudio Export Process are as follows:
- Microsoft Windows 7
- Intel Core i7-7700HQ @2.8GHz
- 3.5GB RAM
- 500GB SATA Hard Drive
- 1000Base-T network connection

Looking at the above statistics it is very clear to see that when a Notes database has a large number of attachments, the time to process will take longer and the end results will not decrease significantly in size. This is due to the amount of space needed to store the attachments whereas when an attachment is added to a Notes document it is automatically compressed.

Step 3: Modifying the HTML Archive Interface

Now that we have a full set of XML archives and the HTML Archive Viewer files, we can view the initial HTML representation of the documents that were in the Notes database.

The initial UI format for the HTML Archive Viewer has two distinct characteristics:

By default, only non-hidden views are displayed.
When viewing documents, we only render the field name and the contents of the fields that are saved on the document.

The first characteristic we feel is more than adequate for every day use since this would be the equivalent to using the Notes menu item "View -> Go To…" to display a full list of the non-hidden views in the database.

For the second characteristic, we chose not to even attempt to parse the form designs in an attempt to obtain the labels for the fields. Why, you ask? The simple answer is that in Notes development there is no correlation between field and the text that is adjacent to it which is being used to label the field. In attempts to acquire the label for the field we found that the accuracy was very low (less than 40% accuracy), mainly due to the fact that each Notes developer has a different approach to the placement of a field label (left, right, above or below). And, since there are no real "best practices" for this (unless you adhere to UX standards) we opted not to include even a partial attempt to label the fields since that would cause more confusion. (Imagine if only half of the labels could be determined and the other half had no labels. I’m sure that would generate an immediate support call for us!)

Instead, we provide the ability to modify a special configuration file where you can specify exactly which views and which fields per form to display. You can even change the display order for the fields and include some basic styling if you like.

This file is written in JavaScript, so if you fancy yourself a JavaScript developer, the sample code and instructions should be very familiar to you.

But the nature of Teamstudio Export is that an IBM Domino Administrator will be using the product, not a developer. I am not saying that an Administrator would not have the knowledge to code in JavaScript, but it is much less likely.

Knowing this, I created a simple utility in a Notes database which can walk you through the process of selecting the fields to display, providing a label for the field, indicate some basic styling, re-sort the fields in the order you most like and select the views to display. Once these items have been configured, the utility will generate the needed JavaScript code to modify the UI of the HTML Archive Viewer.

Although this utility does require that you have knowledge of the fields that need to be displayed in the UI, it does not require Domino Designer or knowledge of Notes/Domino development.

Below are a series of images from our "Staff" Notes database. The images are of the initial HTML Archive UI, the UI of the original form with a document opened, the utility to modify the UI layout, and then finally the end result:

Step 4: Making the Output Available to End Users

Now that the UI for the HTML Archive Viewer is in a state that you can provide it to your end users, you need to provide them access to the files so they can see the output.

There are different methods for doing this:

Move all HTML directories and files onto a network file server and then provide links to the "index.html" file for each HTML archive.
- With this method, security at the operating system level can be applied.
Install a simple HTML web server to assist with common access to the HTML files.
- Examples would be:
  - Apache HTTP Server
  - XAMPP
  - Nginx
  - LIGHTTP D
- These web servers allow you to secure a web page so that your users must supply a user name and password for access.
Use a Domino server to secure and render the HTML files.
- This will allow you to use the hardware and software resources you already own during the process of decommissioning your Notes applications.

For my purposes I used the latter of the three examples above. Although we no longer need the Notes databases on the server, we still use our Domino server for our work on our own products and will always have an Domino server running.

It is a very simple process to use the Domino server as a pure web server. Within the directory structure on the Domino server you will find a directory specifically meant for HTML pages. You can find it on your IBM Domino server in the following area:
<IBM Domino Data>\domino\html\..”

This is in fact where the Domino web-based help files also reside. You can make a new directory within there for placement of the HTML Archive Viewer files. Once your new directory is created, you can simply move/copy your HTML Archive Viewer files and folders into this new directory. Be sure to keep the directory structure that is already contained in the HTML Archive Viewer directories; this is how the HTML code knows where everything is.

Once the directories are in place, you can easily secure the new directories from your Domino Administrator client. You will need to create a file protection document to place security on the "index.html" file. This will allow you to indicate which users from your Domino Directory you would like to grant permissions to see the HTML Archive Viewer.

Here are the steps I took in order to secure the files. These steps assume that the Domino server is already setup to use web configuration documents that are in the "Internet Sites" view. If this is not the case then you will need to configure that option first.

Open Domino Administrator and then switch to the "Configuration" tab
On the navigator, select "Web\Internet Sites"
Open the Internet Site document for the Domino server where the files will be hosted
From the action bar click "Web Site->Create File Protection"
Provide a description if desired. I used "TS Export Output Files\db title" (where "db title" is the title of the Notes database)
For "Directory or file path:" enter the relative path to the "index.html" relative to the Domino data directory (e.g. "domino\html\export_output\office\CRM\Staff_nsf.tse\index.html"
Click on the "Set/Modify Access Control List" button and select the users that you want to have access to the HTML Archive Viewer. Note: the users will only need "Read/Execute access (GET method)"
Save & close the document
Restart the HTTP task

Now when you provide the URL link to the HTML Archive Viewer it will be rendered by the Domino server’s HTTP task and will force the user to authenticate to the Domino server before they can gain access to the HTML Archive Viewer.

And that’s it! Now you can see the process that we undertook in order to archive, retire, and still provide viewable access to the data from an IBM Notes database. There are definitely still a lot of things you can now do with the end result, but for now while you are learning more about the structure of the XML and how to change the UI of the HTML Archive Viewer this should get you well on your way to providing access to and a meaningful representation of the Notes databases that you have archived.

As always, feel free to contact us to chat about this, or any aspect of managing or migrating your Notes & Domino apps. To start the conversation, email me directly or click on the button below. We look forward to hearing from you!

Contact Us