Careers   |   Events   |   Contact   |   402.238.1399   |

Connect with us on FacebookConnect with us on LinkedInFollow Us on Twitter

Category List

Tag List (4)
requirements (2)
react (1)
microsoft test manager (2)
test agent (2)
web (1)
sql server 2008 (2)
lunch and learn (2)
windows 8 store app (2)
event-handling (2)
API Tests (1)
vsdbcmd (2)
M Query (1)
team foundation server 2012 (4)
MFA (1)
preview (1)
installation (2)
faq (2)
nintex (4)
business insight (2)
team build (2)
Quality (2)
cascading-dropdown (2)
visual studio (8)
dependency-injection (2)
AAD (2)
nebraska code camp (2)
mvvm (2)
opensource (1)
sql saturday (2) (2)
Infrastructure (1)
artifacts (1)
reporting (2)
team deploy (2)
web deploy (1)
javascript (3)
necc (2)
agile (12)
Functions (2)
PBI (1)
adfs (3)
Cosmos DB (1)
razor (4)
team web access (2)
test cases (2)
Template (1)
azure mobile services (2)
video (2)
deliveron alm delivery guidance (2)
user profiles (4)
pipelines (1)
ssrs (4)
table-valued-parameters (4)
continuous inegration (2)
msi (2)
site collections (2)
team foundation server 11 beta (2)
Web API (1) (2)
2013 (2)
ARM Template (1)
Node.js (1)
powershell 2.0 (8)
deployment (2)
visual studio 2010 (18)
webparts (2)
release management (7)
microsoft alm rangers (2)
Visual Studio 2017 (4)
security (1)
xaml (2)
REST (1)
sql server (2) (6)
rest assured (1)
coded ui tests (6)
Powershell (1)
json (2)
owasp (3)
lab management 2010 (2)
AzureAD (1)
Visual Studio Online (6)
Data Analytics (1)
action-filters (2)
github (1)
TestArchitect (5)
tfs 2013 (2)
vsts (22)
tls 1.2 (1)
ssl (1)
entity-framework (6)
code-first (4)
netstandard (2)
database projects (4)
tokenization (1)
ssas (2)
jquery (4)
Collaboration (2)
load tests (2)
alm rangers (1)
automatedui (1)
DevSecOps (2)
MSBuild (1)
web application firewall (1)
power tools (2)
Multi-Factor Authentication (1)
sp1 (2)
Office 365 (1)
sharepoint 2010 (10)
whitelist (1)
alm (9)
webpack (1)
dbpro (2)
continuous integration (2)
lync (2)
sharepoint (5)
web performance tests (2)
burndown (2)
work item (2)
outlook (2)
windows azure (2)
insiders (1)
visual studio 2012 (10)
serverless (3)
kanban (1)
visual studio code (1)
tfs (11)
team foundation server (6)
zap (1)
whitesource (1)
service-fabric (1)
load testing (2)
scrum (8)
test controller (2)
Azure (14)
c# (13)
fields (2)
angular (1)
association (1)
licensing (2)
data warehousing (2)
nunit (1)
certificates (2)
my work (2)
exchange (2)
pdf (2)
pipeline (1)
webs (2)
onenote (4)
silverlight (2)
home projects (1)
gulp (2)
Keith Holt (1)
Azure Functions (1)
angular 2 (3)
Nuget (1)
storyboarding (2)
Mike Douglas (4)
connect() (1)
testing (5)
single page applicaiton (1)
tfs odata (2)
sharepoint 2007 (2)
Automated Testing (10)
example (2)
tfs 2010 (2)
bi (2)
subsites (2)
database publishing wizard (2)
xunit (1)
Build (6)
workflow (2)
Analytics (2)
TFS 2015 (6)
microsoft case study (2)
Meetings (1)
webcast (4)
visual studio team services (1)
ninject (2)
Power BI (3)
top 5 (1)
tfs 2012 (4)
bdd (2)
spc14 (4)
performance (3)
sharepoint 2013 (6)
wit (2)
Decisions (2)
fluentvalidation (2)
team foundation 2012 (2)
business intelligence (2)
application insights (2)
bundling (2)
tags (2)
selinium (1)
AzureAD Admins (1)
selenium (4)
performancepoint services (2)
deliveron agile delivery process (2)
HOLs (1)
swagger (1)
Document (1)
alerts (2)
planning poker (4)
IntelliSense (1)
test automation (1)
feedback (2)
tfs 2017 (1)
tips (2)
dns (2)
visual studio 11 beta (2)
mstestv2 (1)
OAuth2 (1)
gherkin (2)
MVP (2)
Java (1)
Big Data (2)
CI (3)
azure resource manager (1)
wiki (1)
DevOps (14)
lab environments (1)
Business (2)


Running Cross Browser Tests in VSTS with Multiple Agents

Aug 12, 2018

In 2017 I wrote an article on the MVP Award blog that showed you how to create a CI / CD pipeline that can execute cross browser UI tests.  Since that time, Microsoft has deprecated the testing tasks that I showed in that post.  Earlier in the year, I wrote a post on replacing the deprecated tasks but as a person pointed out in the comments on the original article, I didn't mention how to use the new tasks to do execute cross browser tests.  In this post I'll show two ways to accomplish cross browser testing with the new Visual Studio Test tasks in VSTS.

In Visual Studio Team Services (VSTS), the build and release pipelines support running a series of tasks in parallel. The series of tasks are grouped together in an Agent Phase.  The Agent Phase specifies an agent pool, the UI Test Agent pool in this case, to execute the series of tasks.  In the Execution plan settings you specify if and how you want the phase to execute in parallel. There are two types of parallelism supported, Multi-configuration and Multi-agent.  Both of these can be used for achieving cross browser testing.  I will walk through both of these options to show you how they can be used and which option is better for executing cross browser UI tests.  


Multi-configuration parallelism

Choosing Multi-configuration parallelism allows you to specify "IE" and "Chrome" as configuration inputs and split the test run across 2 "virtual" test agents where each one of the configuration values is set to one of the two values.  This allows one test agent to run all of the tests for IE and one for Chrome.  Below shows that this works great as long as you only ever need 2 test agent virtual machines (VMs).  If you configure 4 test agent VMs this will only utilize 2.  Although I like how this option looks, I don't recommend using this option for executing UI tests.



Create a variable with the same as as the Multiplies setting and set it to IE,Chrome. The Agent Phase will create two separate instances of all of the tasks, one for each multiplier configuration setting.


In the Visual Studio Tests task, the Browser variable can be used to pass in the appropriate value to the test project.


Every test has been executed for each configuration



I mentioned "virtual" test agents because even if there is only one test agent VM, this this will still split out the run into the two runs but wait for the next available agent in the queue.  Therefore if you only have one agent, it will run through all the tests with IE and then it will run all of the tests for Chrome.

Multi-agent parallelism

This type of parallelism will split the test run by the number of test agent VMs specified.  As the number of tests increase, the total time to execute the tests will continue to increase.  By using this type of parallelism, you can continue to add more test agent VMs to continue to keep the total run time short.


With this configuration, each agent will need to run both browser configurations.  To do this, add a second Visual Studio test task and change the settings of each to run one specific browser configuration.



The test results look almost the same for multi-agent parallelism.




For using only two test agent VMs running to run Chrome and IE UI tests, either option of parallelism will provide similar results.   As you increase the number of UI tests causing increased execution times.  You will want to increase the number of test agent VMs to be able to run more tests in parallel to decrease the overall run time.  Using the Multi-agent parallelism, you can continue to add more test agent VMs to your VSTS Agent Pool and increase the maximum number of agents to run all of your UI tests most effeciently.

Let me know if you have any questions or comments on Twitter




Category: DevOps

Mike Douglas


He is a solution consultant focused on helping organization build high quality application effectively utilizing DevOps practices. He is a Microsoft DevOps / ALM Ranger and you can reach him on Twitter at @mikedouglasdev

We believe in helping our customers create software solutions in a better way.
We do this by having a project delivery process and technology expertise that ensures we are solving the right problem in the right way and driving the most business value.