Wednesday, January 31, 2007

Functional and load testing of ASP.NET Ajax applications – part I

Introduction

Velodoc is developed in ASP.NET C# 2.0 with Visual Studio .NET 2005.

VSTS comes with unit testing, web testing and load testing, considering load testing is actually an execution environment for unit tests and web tests.

Unfortunately Visual Studio web tests work at the protocol level, recording HTTP traffic in order to replay it. The same applies to Redgate’s ANTSLoad and many other load testing tools. Velodoc is an Ajax application and this does not work.

We have started our search for a solution from the following lists of testing tools:

Obviously there are tools like Mercury Winrunner/Loadrunner and the IBM Rational equivalents which certainly cope but they are too expensive.

Our requirements are the following:

  • Test ASP.NET applications
  • Works on Windows XP with IE
  • Compatible with NetAdvantage controls, iFrames, file uploads and Ajax
  • Scripts IE including IE dialogs to drive functional UI tests
  • Several instances can be executed concurrently to create load/stress tests
  • Uses a familiar technology (low learning curve)
  • Open source is a plus and a requirement if the supplier has not been around for a long time.

There are three types of solutions which cope more or less with these requirements:

  • Macro recorders/players
  • Test environments
  • Web application scripting frameworks

Macro recorders/players and test environments

Searching on Tucows, Download.com and other shareware web sites reveals loads of macro recorders. Most of them hook the message pump and replay the windows messages. Obviously, the result is not good. We have tried a dozen of them but only iOpus iMacros 5.2 could do a decent job at automating a file upload from https://www.velodoc.net/. The code is reproduced below:

TAB T=1
TAB CLOSEALLOTHERS
URL GOTO=https://www.velodoc.net
SIZE X=876 Y=627
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:aspnetForm ATTR=ID: SenderTextBox CONTENT=test1@acme.com
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:aspnetForm ATTR=ID: RecipientTextBox CONTENT=test2@acme.com
TAG POS=1 TYPE=TEXTAREA FORM=NAME:aspnetForm ATTR=ID: MessageTextBox CONTENT= Test<SP>with<SP>iMacro
FRAME F=1
WINCLICK X=100 Y=429 CONTENT=
WINCLICK X=100 Y=429 CONTENT=C:\test.bin
FRAME F=0
TAG POS=1 TYPE=INPUT:CHECKBOX FORM=NAME:aspnetForm ATTR=ID: SendTermsCheckBox&&VALUE:on CONTENT=YES
WINCLICK X=490 Y=513 CONTENT=

I see at least three drawbacks to using iOpus iMacros as a functional test tool:

  • The lack of assertions and reporting capabilities makes it insufficient for functional testing;
  • The lack of infrastructure makes it insufficient for load testing;
  • The proprietary language reduces the possibilities despite the fact that it can call external scripts or be called via a COM component.

I have also tried several test environments mentioned in the above lists. Generally, recording the file upload test on https://www.velodoc.net/ has always proved difficult but I cannot tell whether it was feasible or not for some of them. I have limited myself to 2 hours per evaluation and the inherent complexity and learning curve was not worth digging passed first impression.

Web application scripting frameworks

I have looked at two open-source frameworks, which both script IE to execute functional tests on the web application:

  • IEUnit 2.3, a framework based on JavaScript
  • WatiN 0.9.5, a framework based on C# .NET

Both frameworks are very easy to start with due to a familiar language and an intuitive API. Additionally both frameworks provide the necessary assertions to report on the success or failure of a test fixture.

Scripting a file upload on the Velodoc home page with IEUnit entails the following JavaScript code:

_.openWindow(https://www.velodoc.net);
_.setField("SenderTextBox",
test1@acme.com);
_.setField("RecipientTextBox",
test2@acme.com);
_.setTextArea("MessageTextBox", "Test with IEUnit");
var fname = "C:\\test.bin";
var cmdShell = new ActiveXObject("WScript.Shell");
cmdShell.Run("C:\\EnterFileName.sbk " + fname, 0, false);
_.setFrame(0);
_.clickObjById("FileInput");
_.setFrame(-1);
_.setCheckBox("SendTermsCheckBox",true);
_.clickObjById("SendWebImageButton");

Where EnterFileName.sbk contains:

var fpath = " " + WScript.Arguments(1);
var popupWin = _.waitForWindow("Choose file", 30000);
_.setFrame(0);
_.findWindow(popupWin, "Edit").sendText(fpath);
_.findWinButton(popupWin, "&Open").click();

Scripting a file upload on the Velodoc home page with WatiN entails the following C# code:

using (IE ie = new IE(https://www.velodoc.net/))
{
ie.ShowWindow(NativeMethods.WindowShowStyle.Maximize);
ie.TextField(Find.ById("SenderTextBox")).TypeText("
test1@acme.com");
ie.TextField(Find.ById("RecipientTextBox")).TypeText("
test2@acme.com");
ie.TextField(Find.ById("MessageTextBox")).TypeText("Test with WatiN");
Frame f = ie.Frame(Find.ById("FileUploadFrame"));
FileUpload fUp = f.FileUpload(Find.ById("FileInput"));
fUp.Set("C:\\test.bin");
ie.CheckBox(Find.ById("SendTermsCheckBox")).Checked = true;
//ie.TextField(Find.ById("SenderTextBox")).FireEvent("onKeyUp");
ie.Button(Find.ById("SendWebImageButton")).ClickNoWait();
SimpleTimer t = new SimpleTimer(60 * 60);
Span s;
do
{
s = ie.Span(Find.ById("DownloadLinkLabel"));
if (!String.IsNullOrEmpty(s.Text))
goto EXIT;
System.Threading.Thread.Sleep(1000);
} while (!t.Elapsed);
throw new WatiN.Core.Exceptions.TimeoutException("...", 60 * 60));
EXIT:
Assert.AreEqual(true, s.Text.Contains(
https://www.velodoc.net/));
}

Conclusion

The beauty of WatiN is that it builds on top of C# and .NET framework. Accordingly you get not only a rich language but also a rich environment. Test code built with WatiN can be executed within NUnit and Visual Studio unit testing projects which means that they can also be executed for load tests which I am going to try next although we know already that launching several instances of IE has its own limitation.

6 comments:

Unknown said...

Did you ever get Watin to work with load/stress testing? It appears all instances of IE are using the same session.

Unknown said...

I am not sure you can really do load testing with WatiN and any functional testing tool.

Regarding web functional testing tool check out InCisif.net.

InCisif.net is an functional web testing tool for the .NET Platform integrated with Visual Studio (express included) and using the
C#, VB.NET or IronPython languages.

Jacques L. Chereau said...

This has been written in January 2007 with a beta of WatiN 1.0. There were not many solutions to achieve load testing of Ajax components at the time.

Yes, this was working. You needed to configure each instance of IE to run in a separate process. At the time I would launch up to 10 IE sessions from WatiN on a single PC and I would generally use 3 to 5 PCs.

There are better and easier ways to achieve load testing of Ajax components now that we are in 2010.

Rich Clark said...

"There are better and easier ways to achieve load testing of Ajax components now that we are in 2010."

Could you please give some examples of the tools that make load testing easier now as I am also looking into load testing for an ASP.NET web application.

Thanks,
Rich Clark.

Jacques L. Chereau said...

Load testing tools like Visual Studio Team Test record and play back traffic.

They are quite expensive and I have experienced issues with synchronizing Ajax calls. For example let's suppose you have a page which loads and makes Ajax calls to populate certain areas. The load testing tools would sequence the requests on different threads too quickly and the Ajax calls depending on the page being loaded would fail.

Although I have not put Visual Studio 2008 and 2010 Team Test through their paces, I have been told they have improved regarding these issues and you might benefit from them if you can afford the capex.

You also now have SOAP load testing tools which did not exist at the time to specifically stress test your web services called by Ajax components.

As far as I am concerned, I am part of a small team, and we cannot really afford to record both functional test cases and load test cases. So we focus on building good functional test scripts which we run as load tests as explained here.

What is not explained here is that we quickly identify bottlenecks and we also run series of gets/posts on these bottlenecks using tinyget.

The combination of the two with a good set of performance monitor counters as explained in http://www.microsoft.com/downloads/details.aspx?familyid=8A2E454D-F30E-4E72-B531-75384A0F1C47 gives us all we need although purists could object we are not using true load and stress testing tools.

Offshore Software Testing said...

Hi, Your blog is really wonderful. I liked this article since it is very useful to me. Thanks for sharing it. Bye.