Client Test Results

Accessing Web Services from FileMaker

12 August 2010

Web Services Plugin from FMNexus

Testing

A test bed was set up consisting of a number of machines available on MaJic’s internal network

These ‘machines’ were a mix of physical hardware and terminal services sessions.

MaJic also created a test file.

The file consisted of a simple schema that replicated a portion of a client’s FileMaker files that would collect web services result data

The file was populated with arbitrarily generated example data which equalled 186k+ records and which covered random criteria making up potential requests for data.

Testing would happen on two levels:

• Simultaneous requests would be made from multiple users over a short period (1 hour)

• Full load test would be run from a single user over a protracted time frame (2 days)

All results returned were read and recorded.

The looping script which called the plugin functions also had to compile the data to be sent from each record and parse the returned xml and write these results into each record

So, any times mentioned with regard to processing covered these scripted steps as well as the time it took to send the message to the web service and get the response back from the web service

Any errors returned were recorded and any timeouts or plugin failures were also recorded.

The initial test run sent approximately 30k requests from 5 different machines simultaneously over 1 hour – a total of 150k requests sent from FileMaker.

Some records contained deliberately erroneous data and machines had the plugin deliberately failing by leaving the trial version of the plugin in place which would timeout after 1 hour).

The second test run was a full load test and involved sending approximately 112k records sequentially from a single machine over a 2-day period.

The second test did not contain any erroneous data or expected plugin failures.

Both of these tests would load-test the plugin and provide metrics that would allow us to draw conclusions about the plugin’s appropriateness for the specified use.

Results

(note: Success = valid requests, Data error = deliberately erroneous or missing criteria, Plugin error = plugin timeout)

••The first test yielded the following results (5 users & 150k records)

The results across all 5 machines were returned exactly as expected.

Of the 150k records, all but 2 records returned a result in 2 seconds or less and the Plugin error volume identifies all requests sent after the trial plugin had timed out.

••The seconds test yielded the following results (1 user & 112k records)

There were a small number of unexpected errors returned by the plugin.

All of these errors were that the plugin ‘Could not invoke the web service’.

These errors were quite compacted and, although we could not reproduce them again, they could have been caused by one or more of many aspects (see ‘conclusions’).

Records that were processed immediately following those records that produced the error were all processed successfully.

Of the 112k records all but 165 records returned a result in 3 seconds or less (and all but 9 in 6 seconds or less).

The results across all records showed a consistent response time and the plugin showed no signs of slowing after 112k records or showed any sign of stress under load working within the FileMaker application.

Conclusions

The results of our tests speak for themselves.

The FMNexus Web Services plugin is far and away the obvious solution for the client.

The errors in the second load test unfortunately cannot be reproduced but we do know that they are to do with the communication between the Web Services plugin and the web service. The communication is taking place across the internet where any number of factors could interrupt the communication, including some of the following

• Problem with router at MaJic’s site
• Problem with router at client’s site
• Latency of data transferred over the internet causing timeout
• Data packets lost over the internet
• Backup, virus checking or other system process taking place on machine hosting the web service

Finally, it is worth noting that, where an error occurred, the plugin returned a detectable error message that the FileMaker application can be scripted to react to in whatever way is deemed suitable to client

----------------------------------------------------------------------------------------------------

It is fundamentally understood that the web service which was being polled should be able to satisfy very high volumes of requests for data and this is borne out by the responses to requests received during testing and by how timely and consistent they were.

The real point about this testing was primarily focussed upon the operation of the plugin itself working within the API architecture of the FileMaker application.

This was what proved to be a problem-free integration.

The plugin operates within the FileMaker calculation engine, is called via script steps and is therefore procedural.

Thus any request will take the time it takes for the controlling script to complete - and the call to the web service is just one aspect of that script

Overall, the positive test results for Web Services plugin are qualitatively backed up by the fact that the plugin is currently in its third version and has been through years of real world use.

In as far as MaJic can determine, the plugin has proved to be a robust tool and has stood up perfectly within the tests that have been run.

These tests have far exceeded the throughput expected during normal day-to-day use although it must be noted that they cannot replicate real world conditions

That said, and given the results achieved, MaJic would have no doubt attached to recommending that this plugin seems to be entirely fit for purpose.

Harry Catharell

MaJic House