A SQL Table Manhunt


As I mentioned in my last post I recently took on the exciting new position of Chief Software Architect at Hive7, Inc. We're building all kinds of great stuff. Our most popular game Knighthood has over a million registered users and over 100,000 daily actives. This game is growing quickly. Over 125,000 people added the game two weeks ago, and over 150,000 added it in the last week. The game came into existence in December.

This massive growth leads to some exciting scalability challenges. I'll be spending a lot of time talking about that in the future. Today, is a simple tidbit related to databases. Our current performance bottleneck is with database write I/O. We have enough memory in the systems and a caching layer, so the disks barely need to read. Tracking this down is a whole other post, but it's fairly simple. Once we knew we were write I/O limited we set out to find out why.

The original DB physical layout started out pretty simple. There was one File for data, one for logs. In the next 3 or 4 revisions more and more files were created. Why? Well, so we could run this nifty little query and find out which of our db tables/indexes/etc were causing the write bottlenecks:

select 
    db.name as DbName, 
    f.name as FileName, 
    f.physical_name as FilePhysicalName, 
    vf.TimeStamp, 
    vf.NumberReads, 
    vf.BytesRead, 
    vf.IoStallReadMS, 
    vf.NumberWrites, 
    vf.BytesWritten, 
    vf.IoStallWriteMS, 
    vf.BytesOnDisk 
from fn_virtualfilestats(-1,-1) vf 
    inner join sys.databases db on db.database_id = vf.DbId 
    inner join sys.database_files f on f.file_id = vf.FileId 
order by vf.NumberWrites desc


If you have physically separated your various database tables and indexes into different files, the output from this function will give you all kinds of useful information about which ones are most accessed, and which put the most strain on your I/O subsystem. Optimizing it, of course, is up to you. :)

If you enjoy big scale, fast moving, tough problems, we're hiring for a Lead Web Designer and Brilliant Lead DBA/Sysadmin and Web Games Developer (.NET)!

author: JD Conley | posted @ Wednesday, March 19, 2008 2:54 PM | Feedback (0)

kick it on DotNetKicks.com

C# 3.0 Overview


It's been forever since my last post. I promise I'll do better. I've just been juggling three jobs. ;) But that has changed (more on that soon)!

The last couple of nights I did the same talk at two different user groups. Sacramento .NET User's Group and the Central California .NET User's Group. Thank you guys for having me, and not throwing any tomatoes. I think we had a good time at both events. Though, I did take up the whole two hours both times.

The talk was based on Jon Skeet's upcoming book titled C# in Depth which I had the pleasure of reviewing and providing technical feedback. We went through the evolution of C# from 1.0 to 3.0, explored a bunch of the new features, played Human LINQ (hilarious). Oh yeah, it was pointed out to me in the Sacramento group that the word "jumped" should really be "jumps". That's what I get for copying my work! hah! If anybody in the Sacramento area wants to help, I'd like to do it again and video tape it...

Stuff to Download
  • C# 3.0 Overview Presentation – In both talks I didn't have enough time to bore you guys with the "in depth" slides. Pick up Jon's book to learn the nitty gritty about how all that stuff works.
  • A Sorted Affair 2 – - A few weeks ago I published the first version of this on my blog. This one is much cooler.
  • Human LINQ – The code we executed with our Human LINQ provider.
  • Sort Performance – A quick exploration of the relative sorting speeds using different sort methods.


I'll write up another quick blog in a bit on the sort performance. It's quite interesting, indeed.

author: JD Conley | posted @ Thursday, February 28, 2008 10:14 AM | Feedback (0)

kick it on DotNetKicks.com

A New Kind of Application Server


As you probably know, I'm a cofounder of Coversant which, at its heart, is an XMPP development platform. Most of our larger customers (thousands of simultaneous users) are ISV's that have built on the SoapBox Platform®. We allow you to easily develop XMPP applications using .NET technology.

A really long time ago, I wrote about some possibilities for using the SoapBox Platform including examples of what our customers were doing at the time. This was before there microblogging was popular, or I probably would have used that example too. :)

The last couple of weeks there seems to be quite a bit of buzz around the subject of using XMPP as an application server, and that gets me really excited! A friend/competitor Matt Tucker of Jive Software wrote in his company blog about how XMPP is the future for cloud services. A "real" online author (aka not a member of an XMPP company) even picked up Matt's article and ran with it. Yesterday, a little buzz hit Slashdot when another friend/competitor Mickael Raymond of Process One wrote about introducing the XMPP application server (when I wrote this, it seems Process One was experiencing a bit of the Slashdot effect -- hopefully by the time you read this it will be gone and you can read his article), which is an exploration of building a Twitter-like microblogging system on top of their XMPP server. Great stuff, indeed!

This is wonderful news and very validating for me personally! It seems after six years of committing to the infant technology, I wasn't crazy after all, and XMPP is a good platform for presence/messaging systems! And if you're in the market for .NET based XMPP solutions, head on over to the SoapBox Developer site. :)

author: JD Conley | posted @ Monday, February 04, 2008 9:52 AM | Feedback (2)

kick it on DotNetKicks.com

A Sorted Affair: History of the C# Sort


Next month I'll be giving a talk at the Sacramento .NET User's group titled C# 3.0 Overview where I'll be presenting the great new features in the third version of C#. I've been developing .NET/C# software since the first pre-release copies of Visual Studio .NET reached MSDN. It's been fun to see the fledgling C# language evolve with the times. In just a few short years it's gone from what many considered to be an uninspired Java clone to a highly productive, unique, language.

Today I'm going to spoil the opener code sample to my user group presentation. Let's take a journey through the life of C# as a developer with a simple task: sorting a list! I even made a super 1337 Winforms app you can play with.

List Sort GUI

For the purposes of this example, we're going to use a simple class called Person. We'll be sorting the list by the person's first name.

public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public override string ToString()
    {
        return FirstName + " " + LastName;
    }

    public override int GetHashCode()
    {
        return FirstName.GetHashCode() ^ LastName.GetHashCode();
    }

    public static ICollection<Person> GetPeople()
    {
        return new List<Person>
        {
            new Person{FirstName="Ray",LastName="Ozzie"},
            new Person{FirstName="Larry",LastName="Ellison"},
            new Person{FirstName="Steve",LastName="Jobs"},
            new Person{FirstName="Bill",LastName="Gates"},
            new Person{FirstName="Britney",LastName="Spears"}
        };
    }
}


C# 1.0

In the beginning, there was C# 1.0. C++, VB, and COM be damned! For the majority of Microsoft based development, this new environment was a godsend. We enjoyed the benefits of a garbage collected, managed runtime, with the speed of native code. Life was good, but more than a bit clunky.

For the prim and proper developer out there, the ArrayList just didn't cut it. We used CollectionBase and implemented our own strongly typed collections. Here's my collection, in all its glory!

public class PersonCollection
    : CollectionBase
{
    public PersonCollection()
    {
    }

    public PersonCollection(IEnumerable people)
    {
        foreach (Person p in people)
            base.InnerList.Add(p);
    }

    public int Add(Person p)
    {
        return base.InnerList.Add(p);
    }

    public void Remove(Person p)
    {
        base.InnerList.Remove(p);
    }

    public Person this[int index]
    {
        get
        {
            return (Person)base.InnerList[index];
        }
        set
        {
            base.InnerList[index] = value;
        }
    }

    public void Sort()
    {
        base.InnerList.Sort();
    }

    public void Sort(IComparer comparer)
    {
        base.InnerList.Sort(comparer);
    }

    public void Sort(int index, int count, IComparer comparer)
    {
        base.InnerList.Sort(index, count, comparer);
    }
}


Ok, we've got a collection of people. Now, we can finally sort them, C# 1.0 style! There's a class called Sorting in the download that these methods are a part of.

public PersonCollection SortCS1(PersonCollection people)
{
    PersonCollection sortedList = new PersonCollection();
    foreach (Person p in people)
        sortedList.Add(p);
    sortedList.Sort(new PersonFirstNameComparer());
    return sortedList;
}

private class PersonFirstNameComparer : IComparer
{
    public int Compare(object x, object y)
    {
        return ((Person)x).FirstName.CompareTo(((Person)y).FirstName);
    }
}


Goodness, that's a lot of code! No wonder we didn't sort anything in C# 1.0. Not only do you have to create a custom collection class, there is also a custom Comparer class! And there's no getting around this when you want to sort by a member of a complex business object.

C# 2.0

C# 2.0 came along with the .NET Framework 2.0 and a host of improvements. Features like generics, anonymous methods, custom iterators, and partial classes changed the face of the language for the better. No longer did we have to spend time writing the same code like Collection classes over and over, and could focus more on getting things done.

public IEnumerable<Person> SortCS2(IEnumerable<Person> people)
{
    List<Person> sortedList = new List<Person>(people);
    sortedList.Sort(delegate(Person x, Person y) { return x.FirstName.CompareTo(y.FirstName); });
    return sortedList;
}


Wow, so compared to C# 1.0's 60+ lines of code, in 2.0 we wrote one. Very nice. Though, that anonymous method syntax still really bugs me. It's ugly! Why should I have create a full method signature just to do a simple one line function. Ah yes...

C# 3.0

Enter C# 3.0. This is by far the biggest leap forward in C# to date. With the inclusion of lambda expressions we're now able to do some real functional programming, without the anonymous method gunk to get in the way. Check out the list sorting now!

public IEnumerable<Person> SortCS3(IEnumerable<Person> people)
{
    return people.OrderBy(p => p.FirstName);
}


Doesn't that look so much better than the 2.0 version? And it's light years better than the 1.0 version. Clear, concise code. It isn't cluttered up with extra, unnecessary syntax. It's just business. The way it should be.

Since 3.0 also comes with the LINQ syntax, I thought I'd throw that example in there as well.

public IEnumerable<Person> SortCS3Linq(IEnumerable<Person> people)
{
    return from p in people orderby p.FirstName select p;
}

I admit, in this casee the LINQ syntax weighs you down, but you can't say it's not clear what's going on. Throw some cross joins and advanced filters in there and it will start to look at lot more appealing.

The End

C# has changed quite a bit over the last six years, and I can't say I disklike any of it.

author: JD Conley | posted @ Wednesday, January 30, 2008 11:43 PM | Feedback (6)

kick it on DotNetKicks.com

Unadulterated Expurgation!


Alright, so, occasionally I post something funny that happened recently. Yesterday I received a spam company broadcast email from an anonymous customer of mine. You know who you are! :)

Team Anonymous Company,

In each office Anonymous Company employees have inhabited, many attempts have been made to keep our refrigerators clean and sanitary. Keeping clean refrigerators only works if we all participate – yet despite the best team players on the planet, we’ve fallen short of the goal. Common comments run the gamut from, "I’ll never put 'my' lunch in there," to the more lyrical, "There’s a fungus among us!"

As a result of this circumstance, the many must suffer the failings of a few – meaning:

Every Friday afternoon at 4:00 p.m. our Sergeant of Sanitation, will remove and dispose of everything from every refrigerator – as in: any one thing, each thing, the sum of all things, collectively all, the aggregate of all, anything at all, the totality of all within, including or comprehending all things, a comprehensive sweep of any and all things, whatever it is, whatever is left, nothing will be left, the sum total of all things, the Full Monty, the whole enchilada (frozen and unfrozen), the whole shebang. In short, every Friday at 4:00 p.m. all the refrigerators will experience an unadulterated expurgation!

Many thanks in advance for your cooperation and understanding,

The Management



Good thing I don't use the fridge...

author: JD Conley | posted @ Friday, December 14, 2007 8:34 AM | Feedback (0)

kick it on DotNetKicks.com