VSTT: Service Background

I just realized that I've been talking about OfficeLive, but I haven't given you any idea of the user load I'm testing for, or what our performance goals are.

The most used part of our system is a user registration database.  This database is used by OfficeLive, Office Online, and Office 2007 online registration.

The user estimates for all of these services combined is approximately 70 million users over a 1 year period, which equates to 5,833,334 per month. This equates to ~2 users added every second. This number is just an average, though. We have to test to our system to our peak load, not the average load.

Our stress tests have tested our system to 2500 concurrent users.  We estimate that we will be able to support triple that number in production, based on the resource utilization we see with 2500 users.

Building a system to support 70 million entries in a database system comes with a huge set of challenges. I'll continue to share the problems we face, and the solutions we find, as we conquer them.

VSTT: Working with a test rig

In VSTT, you will quickly find the need to generate large numbers of concurrent users.  Generating this level of load will be impossible with a single machine.

VSTT enables this through the use of a "test rig".  You can read the MSDN documentation, on MSDNWiki, here. You might even notice a community comment by yours truly on the wiki site.

Basically, the way it works is that you have one boss machine, called a controller.  The controller works with zero or more worker machines, called agents.  These machines are associated together during installation of the Team Test Controller and Team Test Agent. Both are available on your distribution media for VSTT or VSTS.

Once you have the agent and controller software installed in your test rig, you're ready to generate massive amounts of load.

Now we get to the interesting part.  How do you know how many users you should simulate? It is best to start with a clearly defined goal for the number of concurrent users your software or service should support. My suggestion is to start a load test with half that number of users. Then, you can use a step based load pattern to gradually increase that number of users, until you reach your goal level.  I believe it's a good idea to make your steps as large as you can.

For instance, if you are going to supoprt 1000 users, and you're running a 10 hour test, start with 500 users. Use goal based load pattern, and set your step amount to 100, and the step duration to 90 minutes. This will cause you to reach your target of 1000 users at 7.5 hours into the test.  That allows you to test each step long enough to see if there are problems with the new load level. You will also get to test at your target load level for 2.5 hours, which will provide you with a good baseline for future testing.

If your test fails at any step, you can start a new test, and test at that user level, or slightly below it, and begin to narrow down the reason for your test failing.

The only caveat is if your test fails because you run out of resources on your agents.

In one part of the OfficeLive service, we started testing at 500 users.  We gradually increased the number of users from 500 up to 2500. Once we started to get closer to the maximum number of users we thought we could support, we started changing to a goal-based load pattern. We set the goal based pattern to monitor one of our SQL backend machines, and to adjust load until the processors were at 90% utilization. Every time, however, our test would fail shortly after passing 2500 users.  The test failed because we were running out of memory on one of our agents.

You have two options if this happens. If there is a particular agent that is causing you problems, you can adjust the load weighting for that agent, so that it gets fewer users assigned to it.  If however, all of your machines are low on resources, you have reached the limit on the number of users you can simulate, and, consequently, the amount of load you can place on your system.

In our case, all of our agents run out of memory, so we have found that running with more than 2500 users, on our particular hardware, is not possible.

Hopefully, I've shed some light on how to determine the maximum load you can generate, given your specific hardware assets.

VSTT: Custom counter sets

When you are building a load test in VSTT, you can specify sets of counters to be collected as a group. VSTS provides a bunch of default counter sets, like ADO.NET, and IIS, and SQL.  Those counter sets are great for default programs and installations.

If your service or application, like mine, creates and publishes custom counters, you may find a need to create your own counter sets. Once you've done so, you will almost certainly want to share those counter sets among your colleagues.

If you look in Program Files\Microsoft Visual Studio 8\Common7\IDE\Templates\LoadTest\CounterSets, you will see several .CounterSet files.  They are XML files (with a really easy schema), which define what counters and instances should be collected.

If you edit a custom counter set through the UI, you should find a new file in that directory. 

If you don't want to deal with the UI (it can be slow), then you can create the xml file directly.  This will enable you to build the custom counter sets as part of your build process, or part of the development process.  Manipulating the XML is shockingly easy, due to the simple schema.

You can then check that file into your source control (you are using source control, right?), and all your buddies can copy it to the correct directory on their computer.

The next time they start VSTT, they will see the counter set in the UI, in the list of available counter sets, and can map those counters to computers that you will be monitoring in your load test.

Announcing a new series

The next post will be the start of a new series.  I'll be writing about some of the lessons I have learned while working for Microsoft, helping to build and test Office Live (www.OfficeLive.com).

I will be focusing on stress testing web applications and services, using the tools included with Visual Studio Team Edition for Testers.

First some terms and acronyms. I will abbreviate VS Team Edition for Testers as VSTT. You can also use Visual Studio Team Suite(VSTS). The load  and web testing framework included with VSTS and VSTT is called Ocracoke internally to Microsoft.

Ocracoke is the name of an island in North Carolina, where VSTS was built.

So, stay tuned, and hopefully you'll find some use from these posts.

Upside-down respect

I noticed something at the MSFT company meeting a couple weeks ago that totally shocked me.

During the company meeting, most of the really high-level executives gave a speech.

I was sitting surrounded by a bunch of people who have been MSFTies much longer than I have (most of the company has been there longer than I have). When the executives started speaking, and the demos were over, most of the employees around me kind of tuned the execs out. They speak corporatespeak, like all executives, and most peons I know don't care about corporatespeak.

There were two exceptions. The one that didn't surprise me,and shouldn't surprise you, was Steve Ballmer. SteveB is a madman, and he single-handedly raised the energy level in the entire baseball stadium.

The second one really surprised me. In every other company I have ever worked for, the HR chief was the least listened to person in the company. HR types generally speak a language that not only do peons not care about, but which irritates them.

Lisa Brummel, on the other hand, is different.  When Lisa came out on stage, the entire stadium quieted, and you could sense the interest that people had in her words. Most of the guys around me leaned forwards in their chairs, as if to hear her better.

One of the more senior guys around me made an interesting comment, and one that I have never heard applied to any HR rep at any other company I've worked at. He said something to the effect of "I respect Lisa Brummel more than any other executive at this company". That is a shocking statement, applied to an HR chief.  More shocking (at least to me), is that I agreed with him.

Today, there is an article in the Seattle Times about Lisa Brummel, and some of the changes she is driving. The changes she has introduced have been met with some confusion, as us peons learned how they affected us. However, once we had some meetings about the new changes, everyone I've spoken to was extremely happy about the changes, and basically all said something like "about time".

Thank you, Lisa Brummel, from all of us peons.

More Vista happiness

The screenshots I include with this post don't exactly match the story, but you'll be able to figure it out.

I was messing around with XNA on Vista the other day, and realized something.  I created a new folder, and the icon for that folder was the generic, "I'm a folder" icon.

Then, I put a picture in that folder.  I half expected the folder's icon to change, such that it showed that there was a picture in the folder.

However, what I didn't expect, and was pleasantly surprised to learn, is that the icon was changed in a way that matched the actual picture in the folder.  The picture is a blueprint of a porsche 550. You can see, in the icon for that folder, the outline of the blueprint image, and can definitely see that the icon reflects what is in the folder.

In another folder, I added a couple of items. The icon for that folder showed that there were two items in the folder. It gets even cooler when you have subfolders.

This is going to be a huge timesaver, once we can get our heads wrapped around all the extra information that is in the icon.  Looking for a picture, but you don't remember where you put it?  Just breeze through the folder icons, and you've got a chance to find it.  Can't remember which folder you put that distinct word doc in? Check out the folder icons to find it.

The more I use Vista, the more I like it.

Taking a tradition too far

Traditions are fun.  I'm a big fan of traditions, especially ones that are humorous in some way.

Here at Microsoft, on your anniversary, employees traditionally put one pound of chocolate goodness (M&Ms are very common) outside their door for every year they have worked in the borg.

We have a guy on our team, Mikky, who has been at Microsoft 15 years. Today, there are 12 pounds of M&Ms outside his door (the last three pounds are in his office; they wouldn't fit in the bowl).

This is definitely a funny example of a tradition that has been taken too far.  Personally, I would have tried to find denser chocolate, so the 15 lbs wouldn't be so large.

Congratulations, Mikky.  We sure are glad you've stuck around so long.

Several updates

The guy who writes one of the blogs I read on a regular basis, the Daily WTF, is starting a new business.  He's going to create a type of job board, but with a new (at least as far as I know) twist.  He's going to advertise the jobs on blogs, using a format similar to google adwords and such. Phil Haack is jumping on board, too.

I am totally interested in this, and want to get on board (the bloggers who do the advertising get paid).  To that end, I'm going to start posting more frequently, and more of my posts are going to focus on technical issues and challenges, especially using .NET.

For the meantime, I'll be focusing on stress/perf testing, and also on game development using XNA.  Since XNA is finally in beta, I can post some code samples and such, and start gettting the word out about it.  I've started an XNA game, so expect to see more about that.

Also, I've been contacted by the Coding4Fun guys about writing a series of articles or two about XNA and game devlopment.  I'm going to partner with another Microsoftie, Jason Cahill, to build a game over a series of articles.  We just started talking about it yesterday, so I don't have a lot of details, but I do know that we're going to build a tic-tac-toe game. We wanted to choose a game that nearly everyone would already know how to play, so we could focus on the development challenges, and not defining or explaining a lot of rules.

I'll keep you posted on our progress there, and I'll probably put up some entries dealing with the issues we face getting that to work.

Some super cool Vista features

I've been using Vista for a while now on my laptop and second desktop at work.  I don't actually use those machines very often, which is why I haven't mentioned my experiences with Vista up to now.

At the company meeting Thursday, they highlighted some features that I think are absolutely amazing.

When you click the Vista Ball to pick a program, you can just start typing the name of your program. So, for instance, to open Live Writer (which I'm using to write this post), I hit the little button, then typed 'writer'. When the Windows Live Writer shortcut was highlighted, I hit enter. Live Writer opened up, and I started typing. This doesn't seem important until you get a bajillion items in your start menu (Mike). You can do a full 'contains' search on your entire program menu.

The second feature that amazes me is the Reliability Monitor. This is an amazing feature for network admins. I am shocked that MSFT hasn't pushed this feature more in the news and in marketing messages. We are getting beat up in the press all the time, but we don't talk about these little features that our customers are going to love. The reliability monitor keeps track of all types of system failures and such.  That's not too amazing, the great part is that it also keeps track of all installations. So, when your mom (or father-in-law) calls you and tells you that her computer suddenly got slow for no reason, and started crashing, you can go back and see exactly what she installed that caused the problem. I think this feature is going to blow the socks off of all the network admins in the smaller companies around the world; I certainly could have used it while I was doing that job.

Exceedingly good movie

Courtney and I just finished watching Running Scared. I expected this movie to be the standard Paul Walker action flick; light on acting ability, heavy on fun and explosions.

The movie was shockingly good. There were twists and turns right until the end.  The kid from Ultraviolet was in the movie, and he had the worst possible 24 hours. The movie was really well done, and Courtney and I highly recommend it.

Some amazing artwork

I got a link sent to me at work today.  There's a Marine Warrant Officer who has done some amazing artwork, which will be featured in the new Marine Corps Museum.

Check it out.  It's really poignant.  http://mdfay.blogspot.com/

Pilgrimage

On November 10th, 2007, I will be making a pilgrimage.  Where, you ask?  Here: http://www.faircount.com/web04/usmc/index.htm.

If any of you other prior service Marines want to join me and my family, drop me a line, and we'll organize bit of a reunion.

Send a message of thanks

Courtney found a super-cool site on the net. Xerox is sponsoring a program to send a free postcard to our troops overseas. All you have to do is pick the postcard you want (all were originally drawn by kids), put in your name and hometown, and pick a message. Let's all do this.

http://www.letssaythanks.com/Home1280.html

Well, duh

This post is in the spirit of Courtney's series of posted IM conversations.

 

Courtney says:

I'm brining Shawn home

will you please tell him?

Cullen says:

are you bringing me home?

Courtney says:

no, you rode the motorcycle

Cullen says:

oh yeah. forgot that

 

Sometimes, I shock myself with my own stupidity.

Gulping the drink

Ever since Courtney's post on Gulping the Drink, I've realized something.

When I drink water, I always gulp the drink. Good thing I rarely pour Mike a drink, huh?

How much is enough?

Recently, there has been a lot of talk among tech bloggers about open source projects, and contributing to them. Phil Haack has sounded off about this subject a lot lately, as have most of the bloggers around, including me.

So, I decided to give back a bit. I've started my own open source project, with one of the guys at work. And, I've decided to help with integrating Wix into Visual Studio more. And, I've decided to help with a game that an acquaintance recently made open source. I've found and filed a couple of bugs with SubText, and made a couple of feature requests.

Both Wix and the game have a model where the guys that are running the project don't allow you direct access to the source control system until you have proven that you are committed, and that you know your stuff.

Here's the question: How many bugs do you have to find & fix before you're cool enough?

I've submitted a couple of help fixes/updates to Wix, and submitted several bug fixes to the game (4 bug fixes and 1 feature addition, actually). Currently, I have to submit my changes to someone on the team that has commit rights to the source control system, and let them actually do the submitting.

For Wix, I definitely feel like I'm not at the point yet where I need to be given commit access.

Yesterday, while working on HA!, I found several times that I was being hampered by not having access to the TFS database. For example, I'd like to refactor some of the code, to make it easier to navigate and make the design more OO. Ultimately, I'd like to shelve a changeset that has all the refactorings complete, and let Chris look over the changes, and approve or not. Right now, that would entail manually uploading each file that changed to CodePlex, through a not-great interface, including a comment with each file upload. Poopy.

How many more bug fixes do I need before I get checkin rights? We'll see what Chris has to say, since he reads this blog. Btw, Chris, I don't have your email address, or I'd have just emailed you today, and asked for access; plus, I think this is an interesting question to ask the coordinators of open source projects. What is the barrier to entry as a contributor?

Irritating Malware - Family should read this

Courtney's computer recently got infected with a really irritating virus.

Every time she opened IE, she got a warning that her computer might be infected, and she should run a "system integrity scan wizard".

This is ironic, because the infection she had was the program that was offering to scan her computer, and remove the infection.

Here's how you get rid of it. Be really careful if you do this.

Start-Run
Type regedit, and hit enter
Go to HKEY_CURRENT_USER/Software/Microsoft/Windows/Current Version/Run
Look for a program on the right side that is just random letters stuck together, ending with .exe. In Courtney's case, the path (value) for this program was under c:\documents and settings\local settings
Write down the full path to the file, and pay particular attention to the filename.
Hit Ctrl-F
Type in the file name, hit enter
Every place you find the name, hit the delete button, to remove the reference.

Then, go to the folder that you wrote down, and delete the .exe file that you just removed from your registry. A restart, and everything should be fine.

Final SpaceRocks! update

Yesterday, I decided to stop development on SpaceRocks. I have learned pretty much everything that I can learn from that project, and am at the point of diminishing returns. I implemented nearly every 1.0 feature, and some of the 1.5 features.

The game has definitely been a success for me. I spent less than the target 80 hours in development, and have a playable game (albeit with crappy graphics).

Why is development stopped you ask? I'll tell you.

1. I really feel like I've learned nearly everything that I can from the game. I'm ready to start on a 3d project, so I can learn the hard stuff.
   
2. With the release of the XNA beta, I'm going to start work on a new game, that will be built via XNA, allowing it to run on XBox and PC.
   
3. There is no way to monetize SpaceRocks. In addition to being one of hundreds of clones, I'm pretty sure there would be licensing problems if I tried to sell it.
   
4. My dad made a request for a game several years ago. Back then, I couldn't conceive of how to build what he was asking for. Yesterday, during the keynote at GameFest, I thought of a game concept that mostly provides what he was asking for, while also having the potential to be a money-maker. At least, a money-maker if you consider that my dev costs should be near zero.
   

If those aren't enough reasons for you, try to convince me otherwise. As far as I can tell, no one ever actually played the game. I'd be totally willing to open source the project, if any developers are interested in contributing to the game, and building it further.

If you are a developer, and would like to see the source code to the game, just let me know. I'd be happy to provide it to you.

GameFest Report

First off, let me tell you, MSFT knows how to throw a conference. They provided breakfast, lunch, drinks, snacks, and even free ice cream! The lunch food was amazing, and who doesn't like free ice cream?

There are a lot of cool-looking games coming out from MSFT Game Studios. I am particularly happy about seeing Shadowrun come out. They also advertised Forza2, Call of Duty 3, and Gears of War, as well as several others that I'm not as excited about.

There is at least one really disturbing game coming out. Dead or Alive Extreme 2. I am convinced, after seeing their trailer, that they have more than one programmer working full time on physics simulation for boobs. Theoretically, this is a beach volleyball game. Most of the 4 minute trailer focused on boob shots, and slow-mo running and jumping. These guys were clearly inspired by Baywatch. It's shocking how many of the 'volleyball' players in this game have DD boobs. All of them move in very realistic ways when the girls jump and wrestle and giggle. This game is clearly targeted at the 14 year old boys who need spank material, and dirty old men.

Most of the sessions at the conference were good. I attended a couple of boring ones, and one that was worthless. I went to a hands-on lab(HOL) that used PhysX (a physics modeling package). The HOL didn't have any directed activities. They showed us three demos, and then said "ok, go ahead and play with it". Guess what, if we don't have this product, we don't have any idea how to use the product. Dumb ass.

We got some decent swag; a cool laptop bag, and a 1GB USB key that contains all of the slides from the conference pre-loaded.

MSFT announced XNA studio, which will be in beta in Sept. The full version will be available holiday 2006.

In addition, Garage Games has ported their Torque game engine to C#/XNA, and will be releasing TorqueX along with the XNA release. I am super excited about this; TorqueX will be the first commercial .NET game engine (that I'm aware of), and if it's as good as Torque is supposed to be, then I'll be a happy developer.

MSFT is also working on Live Anywhere, which brings the XBox Live experience to pc games, and allows for cross-platform multiplayer gaming. Shadowrun will be the first game that supports cross-platform gaming. They also talked about a version for smart phones. The use case they gave was modifying your Forza2 car from your phone, and playing on your XBox/PC later. That is certainly exciting.

There has been a ton of good information about developing for Vista. I learned all about how to get your game to show up in the game explorer, and capitalize on the rich user experience that Vista provides.

Garage Games gave a talk about how to be successful selling downloadable games for the PC and the XBox. They should know; they wrote Marble Blast Ultra, which is the highest revenue-producing game on XBox Live Arcade. I talked to them about getting a beta of the TorqueX engine. They were all super nice guys, and very easy to talk with. It seems like they brought their whole company to the conference. I know they have several devs, two co-founders, and the CEO here.

I got to meet Rico Mariani today! He's a MSFT architect that is responsible for CLR performance. He has an incredibly interesting and useful blog, Rico Mariani's Performance Tidbits. I highly recommend it, if you are writing managed code and want performance.

Apparently, GameFest is a good draw for getting a diverse crowd. I sat down for lunch today with 5 or 6 guys from Garage Games, and one Microsoftie. About 20 minutes later, they all left the table. I was prepared to sit and stare at the looping game ads for the next 45 minutes, when a young woman asked if she could sit next to me. Turns out that she is one of the like 5 female game developers in the world. As many of you already know, most developers are younger men, with limited social skills, and sometimes interesting odors.

In addition to being a female game developer, whose husband is a Bungie game developer (meaning he works on Halo), Heidi also worked on a game that I totally loved; MechCommander 2. I mentioned that I really liked the interface of MC2 a lot more than the original, and she casually mentioned that she wrote that. How cool is that? I met someone who actually worked on a shipping, commercial game, which I totally enjoyed playing. I think I'll have to reinstall MC2, and play some more, in honor of Heidi.

She also mentioned that they might be looking for a tools dev in the future. Remember that, Matt; if you're mean to me, I'll get a job at a game company, and manage to avoid the high pressure game development at the same time.

I'm waiting now for the hands on lab for Direct3D10 shaders to start. This particular lab is going to be useful for me, since my next game will have to use shaders for lighting and such. XNA is not, as far as I know, going to support the fixed function pipeline, so I have to figure out how to implement lighting and particles and a bunch of other stuff in shaders.

Too much?

You people know me. Is this t-shirt too much? It's the ultimate inequality.