Using Twitter for classroom queue management

Problem: 

I teach a hands-on multimedia class of 25-30 students. During lab times, the students often need help. I was finding that as I was helping one student, their neighbor would grab me as soon as I was done, and I could never get to the other side of the room.

I wanted a way to create a queue that made it fair to all students.

Constraints:

• Students didn't need an account or access for it
• Would always be available, so that if they were having a problem before class, they could be first in line
• There was a mobile option, so I could check the queue while walking around the room. I didn't want to have to go back to the computer to determine who was next
• I could get a notification if I didn't have the queue running on my machine
• Students saw their location in the queue. 
• It was easy, without many moving parts. This was something that I needed to stand up in a couple of hours and work reasonably well.  

Solution: Twitter as the backend

Twitter offered everything that I needed. 

• Already managed and ordered timestamps
• Already had access and notifications on my phone
• Some students already had accounts, but a dev twitter account and simple custom interface would allow for anybody to access it
• Search for the special hashtag, and the queue comes up. 

And as an added bonus, I got a chance to play with Node.js, socket.io, JQuery, jade templating, Heroku hosting and a bunch of other technologies which were new to me. 

The end result is that the students LOVE it. They recognize that it is fair, and that it is really easy for them to use at any time. 

Inspired by Google, my UI was super simple :

The tweet looks like "Drew Shefman needs help #uhmultimediaHelp @UHMultimedia 1390595370659"

• Drew Shefman is obviously the first and last name. 
• #uhmultimediaHelp is the hashtag that I've choosen. (University of Houston multimedia). This is ultimately my search term. 
• @UHMultimedia is a DIFFERENT twitter account than the dev account. It is the twitter account I use for my class for announcements. I include this in the message, so that I will receive email notifications. 
• The number at the end is the timestamp in milliseconds. I found that if a student asked for help twice, that it was not showing up a 2nd time, because twitter prevents a duplicate tweet in a short time period. 

So here is the flow: 

1. On page load and before page display, twitter.search() for #uhmultimediaHelp
2. Filter out  any tweets older than 5 hours (my class is only 3 hours long)
3. Display results on page, showing name and formatted time (HH:MM)
4. Student enters name and hits "I need help" 
5. Socket broadcasts new entry and it gets prepended to the list element
   (this is primarily so that there is immediate feedback, and the impatient students know not to submit multiple times)
6. twitter.updateStatus()
7. On twitter submit success, repeat twitter.search()
8. On search success, socket broadcasts a refresh with the resultant search list.
   (I perform this seemingly duplicate step of refresh so that if any students are using their personal twitter account, with the hashtag, they will get pulled into the queue. I decided to not use twitter.stream() api as it added a significant layer of complexity. The majority of my students opt for my interface versus using their accounts, so this is a reasonable solution for now. This decision will fail if the number of personal accounts outweighs this queue, as there is only a twitter refresh when a student submits through the queue. 
9.  Then, starting from the bottom of the list, I can help the students in order. Woot!

Technical challenges: 

 This was my first foray into a decently complex HTML5  / Node app. So starting out *everything* seemed to be a technical challenge.  So outside of googling nearly every single line of code I was typing, here are the specific things that I ran into. 

• I choose ntwitter as the node package to leverage for the twitter api. Unfortunately it had not been updated to Twitter's new 1.1 API. So I had to fork it, and modify it for that. You can get that version on https://github.com/dshefman/ntwitter. The change to the 1.1 was simple and you can see it at this commit. 
• I tried using the twitter.stream API, but it requires a single instance. But each connection (webpage) was trying to create a new stream. After much researching, there was an idea to create two Heroku instances, one for the stream and one for the app. This was going to be too much work, so I took a different path.
• On Heroku, I pushed up my repository without following the instructions, basically leaving out the init(). Boy it didn't like that. Basically had to delete my whole Heroku account and start over. 
• My timezone is -5 from GMT, which is what twitter uses. Not a problem, until 7pm (my class is from 5:30 - 8:30pm). At 7pm CDT, that is 12am GMT (00:00:00), in which all of my displayed times switched to -5. It was a simple solution, but something I didn't account for when I was testing it during the day. 
• Initially I didn't provide fast enough feedback, and the students would submit a half a dozen times before it would show up on their screen. Then they were embarrassed for submitting so much. 

Technologies Leveraged (Each was decently new to me):

• CSS 
• Jade templating 
• JS
• Jasmine BDD
• Karma test runner (continuous mode is SUPER cool)
• Node.js / Express
• Socket.io
• Twitter API via ntwitter
• Heroku node hosting
• JQuery

Usage example

Code:

You can find the project at:
https://github.com/dshefman/TwitterHelpQueue

There is one file missing: twitterAPICredentials.js
You will need to add your own API keys if you want to use the code.

Since I'm fairly new to all of these technologies, I gladly welcome feedback / best practices for improvement.  Thanks! 

SCNA 2013 review

I attended Software Craftmanship North America (SCNA), this past November (2013) and wanted to share some of the major things that I got from this conference.

Software Craftmanship is a philosophy about developing software and raising the bar of the entire profession. The craftmanship manifesto augments the agile manifesto and adds that the craftsman also values well-crafted software, steadily adding value, community of professionals, and productive partnerships.

SCNA is the conference for raising the bar. It is a language / technology agnostic conference. It is a community of people who share the above values. The opening remarks of the conference are "Your annual attitude adjustment," which is absolutely true. I find the conference humbling and inspiring and everyone there cares about my growth as a developer. Thank you Twin Technologies for sending me.

---

The conference started with a reprimand from renown author and personal inspiration Robert Martin. His admonition was regarding the heathcare.gov debacle and how it's fallout has and will negatively effect our profession.

The take away from his presentation was regarding the "responsibility of knowing". Somewhere, some developer knew that heathcare.gov, wasn't tested and ready for production. Relating this to the Challenger explosion, Robert Martin asserts that we, professional software developers, MUST be able to stand firm when we "know" that something isn't ready.   A technology problem has now become a policy problem, where technology is preventing a legally created policy from happening. He cited some great references to the  Developer Bill of Rights and Client Bill of Rights.

---

Sara Gray, then presented a fantastic analogy regarding writing new worlds. Her statement is that you create the [programming] world that you live in, and are you thinking about the others (including your future self) that will have to live in that world. The analogy that she used was Harold and the Purple Crayon. You get to create your world and everything that you need, but sometimes you accidentally create monsters.

Sara asserts that like parts of a sentence, that there are parts of code. There are named things, like variable, parameters, and methods. There are pieces of knowledge, which should be extracted and then named. There are changes of state, which could be represented as "who changes state", and might be a good place for helper classes. Finally, there is the "speaking voice", which I believe is the overall flow.

---

Ken Auer says that you can't live on stackoverflow or github alone. The goal is to raise the bar, not win an arguments. When addressing poor code, the goal should be to raise the bar, to transfer knowledge in a constructive way, and not win the argument by proving your superior knowledge. He gave his mantra, which was "make it run, make it right, make it fast", but recognize that "make it right" is limited by the most skilled participant. Based on the Dreyfus learning models, the novice has to be "in sight" of the expert, in order for them to grow. One of the nuggets that I always look for at conferences is potential interview questions. Ken provided a good one. "What software have you shipped?"

---

One of the most thought provoking, and immediately applicable sessions was given by Corina Zona, entitled "Schemas for the Real World". Her assertion is that giving a form field a name or restricted answers limits its scope, and that you could declare a person invalid. People are never edge cases. Some examples would be a gender field of male/female; a sexuality field of hetero/bi/gay; drop down boxes for religion, race, or relationship status. The user often has to choose to be inauthentic to fill in these fields, saying "this isn't really me, but it is the only option". Why can't all of these fields be text inputs? What questions are we really trying to answer? Maybe the question that you are trying to answer, instead of asking for gender, ask "what pronoun do you prefer?" Based on her research, when given a free text option for gender, only 40% responded with "m, f, male, or female".


Here is another example of potential invalidation: Facebook's relationship status. Does it alienate people not in or not looking for a relationship? When does a "widowed" status change to "single"? How is the "Married, Separated, and Looking" status represented?

The cool thing about this session, was that that evening I had dinner with some college friends who work in admissions in higher education. We discussed that this was the exact thing that they *battle* with their IT group all of the time. It is very unusual to have a long lasting conversation about the things that I've learned in a conference with non-developers.

The other neat aspect of this, was that I was able to put it to use in the next form that I created. I'm organizing a parent / child dance for my daughter's school, and on the form, instead of having check boxes for mother and father, I have the following field: "Relationship of the attending adults to the child(ren)." This will hopefully give me the answers that I'm looking for... answers like "mother and father, step-father, mom1 and mom2, grandmother, etc". It provides a very rich data set.

---

The panel on software quality had some interesting insights. Does quality even exist? Given that it, quality, is an individual definition, it might not even be describable until you can see it or see its absence. Interestingly enough, is that quality has no value today. It has value tomorrow; it is something that you pay forward. Simplicity, which is often the sign of good quality, takes experience because it is hard. Testing is a good *tool* that might be an indicator of quality, but it is not a rule that tests mean high quality.

---

Dave Thomas, one of the Pragmatic Programmers,

talked about the unknown knowns and that we should teach people what we were taught, but infuse them with what we have learned. Given the matrix of what you know that you know, what you know that you don't know, what you don't know you don't know, and the what you don't know that you know; it is this last one, the unknown knowns that might be the most valuable. It is the cumulative integration of your experiences, and the critical piece which we have to try to share. A simple example to references is how do you recognize a face, or describe how you walk. Thinking about them actually breaks your ability to use the knowledge. The parts that we need to transfer, is why did you structure the code this way instead of that, or why does this code make you feel dirty?

---

Sandro Mancusio provided some insight into some of the criticism and rebuttal that the craftmanship movement has encountered. 

The first criticism is the "craftmanship is just XP rebranded". Craftmanship is an ideology not a methodology like XP. It is about principles not practices. Practices are chosen based on the value that they bring. 

"Craftmanship is an elitist movement." Actually, we (the crafts-people) are fully inclusive, trying to raise the bar across the industry. We recognize and support the need of novices. We NEED them for everyone to grow. 

"First crafted code, then whatever the client wants." We practice writing quality code, so that time is not a factor to deliver quality. Quality becomes inherit in our work. 

"Pragmatism over religion." The message is professionalism, it is not TDD or practices. "What does it mean to be a craftsman", is not universally definable, but a personal definition. How it is done is as important as having it done. 

One thing that struck me as particularly powerful, was his mentor's description of his first attempt at code as "disrespectful". Out of all of the adjectives available to him.... "bad, inefficient, ugly, unmaintainable", he choose disrespectful. That is quite an interesting context. 

---

On the nature of software development, by Ron Jefferies and Chet Hendrickson, they assert that most agile teams have it wrong. Coding is a team sport, and that everyone on the team should be somewhat capable of "scoring" even if that isn't their primary duty. Backlog refinement shouldn't be the sole job of the product owner, but a team problem-solving effort. The whole team needs the vision of the product, the "what are we building". The team is actually the product owner, and the product owner is technically the "product champion" 

---

Finally outside of what I've already written about, there were some notable quotes that I wanted to share. 

• Trust-driven development. How transparent can you be to build trust? 
• "Does my commit increase or decrease the entropy of the system?"

Oh and one last thing. I was able to participate in a code retreat at 8th Light. I'm very envious of their bookshelf.

Sequence Based FlexUnit integration testing

Did you know that sequence based testing is built into FlexUnit? I was relieved to find this link in the documentation (http://tutorials.digitalprimates.net/flexunit/Unit-13.html). If you go to that page, please scroll to the bottom section about "Using Sequences".

One of Twin's clients needed a real time update for currency exchange. Basically in our system, instead of a continuous feed, the business requirement was that every week the admin would enter the new exchange rates (there were only 5 that were relevant). Once saved, there was supposed to be a real time push notification to all open clients so that they could update their forecast data. 

Here is what my integration test needed to prove: 

• Get the current exchange rates from the service
• Save a new exchange rate
• Receive the push notification
• Validate that the new rates has been recorded into model under the correct date.

var rtmpManager:ExchangeRateRTMPManager = new ExchangeRateRTMPManager();
var sequence:SequenceRunner = new SequenceRunner(this);
    sequence.addStep(new SequenceCaller(eventDispatcher, getInitialExchanges, [weeks] ))
    sequence.addStep(new SequenceWaiter(eventDispatcher, INITIAL_RATES_EVENT,TIMEOUT, handleTimeout));
         
    sequence.addStep(new SequenceCaller(eventDispatcher, saveCurrency ,[er]))
    sequence.addStep(new SequenceWaiter(eventDispatcher, SAVE_CURRENCY_EVENT,TIMEOUT, handleTimeout));
  
    sequence.addStep(new SequenceDelay(15*1000)) //Delay 15 s to see if we can get a notification in that time
    sequence.addAssertHandler( validateRTMPManagerReceivedPushNotifcation, null);

    sequence.run();

Tests passed, and it worked beautifully!

Permissions Modelling in Flex

Permissions is something that most of my projects have needed. I've been iterating on a design for many projects, and I've finally come up with something (leveraging some work that my Twin teammates have produced)  that is scalable, testable, and easy for my team to implement. I've written this up in the context of Flex / Actionscript, but there is no reason why it couldn't apply to other languages.

Just to clarify my requirements, here are the user stories that I'm working with. 

• Should be testable.
  As the developer, I want to be able to declare which permissions are present during testing and development independent of my actual permissions on the system. (In other words - decoupled)  
• Should be scalable
  As a client, I want to be able to add/remove permissions at any time without major work/rework in the application
• Easy to understand
  As a new or jr. developer, I want to be able to understand and implement permissions in a best practices way

What I came up with, was a hierarchical string based system. It looks like the following:

public class PermissionTaskConstants
    {
        public static const DEVELOPER:String = "developer"
        public static const ADMIN:String = "admin";
        public static const ADMIN_USER:String = "admin.user";
 public static const ADMIN_CURRENCY:String = "admin.currency"
        public static const LOGIN:String = "login"
        public static const PROJECT:String = "project"
        public static const PROJECT_EDIT:String = "project.edit"
        public static const STAFFING:String = "staffing";
        public static const STAFFING_EDIT:String = "staffing.edit";
        public static const STAFFING_ALLOCATE:String = "staffing.edit.allocate"

    }

In this way, if you were assigned a permission of "staffing.edit.allocate", you would inherit all of the rights of "staffing" and "staffing.edit". This satisfy's the scalable requirement. It is trivial to add or remove permissions, as it would only mean adding / removing strings from the permission path.

These permissions would live in a "permission registry", which would contain an array of strings. This registry could be populated from a service call to a database, or it could be filled at run time. Filling it at run-time would satisfy the "should be testable" requirement. The code is availabe at the bottom, but I've set it up so that it dispatches events when permissions are changed.  A developer can listen for those events and update the UI. Although the typical workflow is that after login, we retrieve the permissions from the database, and the registry is populated before any of the UI is displayed.

IMPORTANT: Decouple the registry from the view!

So this is super important. In order to make this testable and decoupled, never, ever, ever reference the registry in the view. Said differently MXML files should have NO KNOWLEDGE of the PermissionsRegistry.

Here is the recommended workflow. You have a MXML file that represents the view. Just for clarification, this view is in charge of the layout and interactions with the user. There should be no business logic in the view.

Then you have an presentation model (PM) that is in charge of all business logic. This is likely a pure actionscript file. Business logic means conditionals, domain level events, and in this case permissions. The PM receives a copy of the permission registry through dependency injection. The PM then exposes the specific permissions that the view needs through methods.

The view creates its own Bindable public properties that represent the various permission states, and then reads them from the PM. (Please forgive the incorrect capitalization in the MXML objects, the css "brush" for this formatting strips that out)

    <![CDATA[
        import mx.controls.Alert;
        import mx.events.FlexEvent;

        public var pm:PermissionsPM;
        [Bindable] public var canReadProjects:Boolean;
        [Bindable] public var canEditProjects:Boolean;
        [Bindable] public var canReadStaffing:Boolean;
        [Bindable] public var canEditStaffing:Boolean;

        private function creationCompleteHandler(event:FlexEvent):void
        {
            pm = new PermissionsPM();
            canReadProjects = pm.canSeeProjects();
            canEditProjects = pm.canEditProjects();
            canReadStaffing = pm.canSeeStaffing();
            canEditStaffing = pm.canEditStaffing();

        }
        ]]>
    
    
    

package
{
 import com.squaredi.permissions.PermissionTaskConstants;
 import com.squaredi.permissions.PermissionsRegistry;

 import mx.collections.ArrayCollection;

 public class PermissionsPM
 {
  [Inject]  public var permissionsRegistry: PermissionsRegistry;

  public function PermissionsPM()
  {
   permissionsRegistry = new PermissionsRegistry();
   var permissions:Array = [PermissionTaskConstants.PROJECT_EDIT, PermissionTaskConstants.STAFFING]
   permissionsRegistry.permissionsList = new ArrayCollection(permissions);
  }

  public function canSeeProjects():Boolean
  {
   return permissionsRegistry.hasPermission(PermissionTaskConstants.PROJECT);
  }

  public function canEditProjects():Boolean
  {
   return permissionsRegistry.hasPermission(PermissionTaskConstants.PROJECT_EDIT);
  }

  public function canSeeStaffing():Boolean
  {
   return permissionsRegistry.hasPermission(PermissionTaskConstants.STAFFING);
  }

  public function canEditStaffing():Boolean
  {
   return permissionsRegistry.hasPermission(PermissionTaskConstants.STAFFING_EDIT);
  }
 }
}

A couple of other special features about the code. As a developer I can manually add and remove permissions from the list. When I do this, it sets a "developerUpdated" flag, which will ignore setting the whole array. Here is the use case for that. I login to the system but the service call to get permissions is delayed (for some reason). Then I want to verify how the system looks for other users, and add specific permissions, then the service call comes back and could potentially overwrite the values that I've modified. So there is code in place to prevent that.

public function set permissionSource(v:Array):void
        {
          if (developerUpdated) { return} //Don't allow it to be overwritten if the developer has already set individual permissions for testing
          permissionsList.source = v;
          permObj = createPermObj(_permissionsList)
          notifyUpdate();
        }

        public function addPermission(permission:String):void
        {
            var idx:int = permissionsList.getItemIndex(permission);
            if (idx <0)
            {
                permissionsList.addItem(permission);
                permObj = createPermObj(permissionsList)
                developerUpdated = true;
                notifyUpdate();
            }
        }

The other feature that is in the code, is an interface for extracting the data. This could be used in the case of a value object (VO) that returns from the database that has a list of permissions for a given user. This VO is an object, and doesn't have the necessary string representation for permissions. The permission registry accepts an PermissionExtractor helper, that can convert the VO into permissions strings.

package com.squaredi.permissions
{
    public interface IPermissionExtrator
    {
        function getPermissionString(perm:Object):String; // returns a.b.c
    }
}

All of the code and an example can be found at: https://github.com/dshefman/FlexPermissions

Quick and Free, Remote-Team Planning Poker

Recently I was on a hybrid team, made up of representatives from Twin Technologies and 2 other companies. We were all over the country, and we needed a way to play planning poker effectively. Given the other tools that we were using, adding an additional tool, credentials, IT clearance, etc. wasn't feasible.

So we found this great blog post from fourkitchens about using a Google Spreadsheet for planning poker.

To play planning poker, we all call into the phone / voip conference number and open the google doc. After we discuss each story, everyone enters their story point vote, but doesn't press Enter right away. The scrum master can see that everyone has entered a value when their respective box is highlighted grey. Once everyone has made a choice, the scrum master calls “3, 2, 1: Go” and everyone hits Enter. All values appear on everyone’s screen and the planning poker game continues as normal.

We color coded the cells based on conditional formatting (Format -> conditional formatting). Each Fibonacci number got its own color. Invalid numbers remained white. This gave an easy visual representation on the distribution of values.

Now you might have noticed the "Average" line in the photo. We were spending too much time discussing and coming to a unanimous agreement on these estimates.  Based on recent recommendations in the Agile world, everyone doesn't have to agree on a number. The numbers do have to be within a spread of 3 though. For example, 2,3,5 or 3,5,8 are valid spreads. The spread has to be there present though: 2,5,5 or 3,3,8 are not valid spreads and would need to be discussed. (The photo above would represent a invalid spread and we'd have to have more discussion until we could bring that in).

With the adopted averaging the numbers, we would round up to get our estimated point value. Usually what would happen, is that as the team size would unfortunately fluctuate and /or different people could or couldn't attend the planning, we would change what was being averaged. The picture above shows the average only from Rick, Drew, and Larry, and the resulting estimate would be 2 story points (rounding up, or Math.ceil(1.333) :) ) .

Given this system of averaging, it is likely and common to end up with non-Fibonacci totals for our estimates. That is okay. Since the estimates have no time based correlation, they are only used to determine our velocity and therefore our commitment, using Fibonacci only numbers as the inputs and the average as the output works out really well. Fours, sixes, and sevens in out estimate gave us good insight into velocity.

Now just to repeat how agile estimates are supposed to work. The estimate is RELATIVELY sized based on team decided criteria. These UNIT-LESS numbers are used solely to determine velocity. It will likely take 2-3 sprints to even figure out what the velocity is. The estimate HAS NO CORRELATION to time. Once the velocity has been determined, then it is used to make the COMMITMENT. The team says, that we can do (x) story points per sprint, so we will commit and GUARANTEE that (x) will be done by the end of the sprint. That is why averaging works in this case, because the numbers roll into the commitment, and not into any time units.

This system is fast and simple. Adding or removing team members is a no-brainer. Estimating was no longer a chore, nor took up very much time. We could get in and out very quickly and get back to work. I hope that it works as well for you.

Remoting Landmine without a stack trace or debug ability

Summary: 

While using AMF remoting, if you get the following error: 

TypeError: Error #1034: Type Coercion failed: cannot convert Object@1456c7b9 to mx.messaging.messages.IMessage

And you've overridden the  hashcode to your java VO objects, you will end up in an un-debuggable situation. 

Solution:  Remove the hashcode override. 

BackStory:

We are migrating database schemas, so I had set up unit tests to compare the results from one schema to the other. I am using JUnit and assertEquals. 

According to the documentation, I should override the equals() method on each object, as that is what the assertEquals uses for the comparison. This was tedious, but then I discovered that Eclipse had a generate equals() and hashcode() methods. Eclipse comments say to always include hashcode when overriding equals. Ok. I confirmed that on google, all things checked out. 

What happened though, is that serializing from Java to Flex went fine, but on the return trip --- BOOM. 

The error was an UncaughtErrorEvent.UNCAUGHT_ERROR. The request made it out, according to Charles webproxy, but never actually made it to the endpoint on the other side (Based on the debugging breakpoint). Since it never made it to the service, the normal fault responder didn't fire. Fortunately, I'm trapping all uncaught error messages in Flex. There was no stack trace in Flex or in Java. Just this error thrown from somewhere.

The only information I could gather from Charles was that it was a StackOverflowError. 

I only discovered this information by rolling back one commit at a time, then cherry-picking one file at a time, then one method at a time. Painful! 

If you run into this, I hope that I've included enough keywords that you find it quickly and don't waste a whole day like I did. 

Software Estimating Tips / Tricks / Traps

This post is broken down into the following areas

1. The presentation given at
      360Stack on Aug 6, 2013
      Houston TechFest on Sept 28, 2013
      Software Craftmanship North American on Nov 9, 2013
      Agile Leadership Network - Houston on Feb 20, 2014
      PMIHouston - June 8, 2015
2. The highlights from the presentation, in case you don't want to read the slides
3. Resources
4. Questions that were asked during the presentation
5. Backstory to the presentation
6. Cross-links
7. Feedback and comments

Software Estimating: Tips for upping your SWAG

Here is my presentation

• Software Estimating Presentation (click through version)
• Software Estimating Presentation (PDF) - 50 min version
• Software Estimating Presentation (PDF) - 75 min version
• Software Estimating: Lightning Talk (PDF)

I'd love to hear your feedback. 

9 Tips for improving your software estimating abilities

1. Determine if you are supposed to be estimating or hitting a target
2. Use units that are consistent with the estimates underlying accuracy
3. Explicitly express the uncertainty in your estimates
4. Collect and use historical data
5. Count when possible; compute when you can't count
6. Do a pre-mortem
7. Envision someone else (not you) doing the work
8. Remember the most commonly forgotten project related tasks
9. NEVER TRY. Do or do not. Manage your commitment and your implied commitment with vigilance. 

Resources:

• Software Estimation: Demystifying the Black Art by Steve McConnell
• Clean Coder by Robert Martin
• Thinking Fast and Slow by Daniel Kahneman

• http://psygrammer.com/2011/03/12/misunderestimations-part-1/
• http://psygrammer.com/2011/03/16/biases-create-misunderestimations-part-2/

Questions from presentation

• Could you talk about larger projects, like bigger than six months. Do these techniques still apply?
   The "Demystifying the Black Art" book is absolutely structured to discuss which techniques work at which stage of the project. It also categorizes by project size, what stage in the project, and type of project (iterative or sequential).
• How do you build in scope creep into your estimate?
  Put that into your explicit range as a +2 months for 20% new features, or if you are unsure about the client have a +6 month or whatever risk quantification. 
• Do I charge for estimates?
  I wish clients would pay for it because it is a lot of work, but sadly, no I don't. 
• Estimating based on historical data... isn't that like your stock prospectus "Past performance doesn't guarantee future performance"?
  Not really, the market is driven by so many factors, but this isn't the case, this is YOUR historical data. Unless your team has had a complete overhaul of people, or you have done a Matrix like download of new skills, your team's historical data is an incredible accurate model to base similar projects against. But beware of the intuitive trap, use concrete numbers (size, number of stories, number of screens, etc) to do the comparison. If you are using intuition or memory for the comparison, you will likely not get a good estimate. 
• How much effort do I put explaining this [the concepts in this presentation] to the client?
  I make sure it comes out in my language. First I clarify if I'm creating an estimate or hitting a target. If I'm creating an estimate, I make sure that I use estimate and commitment in the same time frame.  Something like the estimate is 5 months, but I can commit to you that I'll deliver something of value, that is less than the whole thing, to you in three months. Give me 2 weeks to get into it, and I can give you a good sense of what that might be. 
  
  The overall trap, is that you want to avoid committing at the big end of the cone, when it is most uncertain and unclear about what the project is about. 
• Do you give one estimate to the employees, and an later estimate to the client, so that way the employees finish early?
  Personally, this question made me uncomfortable.  My answer is no, I see no reason to withhold information or give misinformation to my team. As an Agilist, my client is involved all the time. I show them something every week. There is no question to anybody, either on the team or the client, the exact status of the project and if we are on target or not. When my team makes a  commitment, we treat that as such. I can ask my team "I have your word that we will complete all of the things that we say by the time that we said".  If your team is not at a place to give their word, to make that commitment, then it is up to you to determine how you would manage your team and client interactions.
• Tangent : The research and the book "Drive" by Daniel Pink was mentioned (and I highly recommend it)
• If estimates are non-negotiable, what happens when you get a new piece of information?
  If you get new information that improves the clarity of the project, and moves you farther into the cone, then by all means, re-estimate. Remember you *should* be re-estimating several times before the end of a project. Estimates are not fixed. 

Backstory on this presentation

I had a great time presenting last year at 360Flex, and I wanted to present again. I knew that since the feel of the conference was changing to be less Flex centric, I wanted to make sure to choose a topic that was generic enough.  At the time, I was working on this crazy estimate with 100+ highly detailed requirements. 

While I was doing this estimate, I thought "I really have no basis for what I'm putting down here.... I'd like to know how to make better estimates" 

Also at the time, I was reading several psychology related books, and the overwhelming consistent information coming from these books, that unless you are an expert at a given task (10K applied hours) that your intuition can easily lead you astray. I was certainly not an expert at estimating and I was completely guessing at what the effort was. 

Therefore, I declared that "Estimating" was going to be my topic, and threw my hat into the ring, so to speak. Immediately I started researching the topic, just in case I got picked. 

The field of estimating is WAAAAAAAY overwhelming. PhD disserations, $100K dollar software packages, complex statistical math, and simple tricks were un-navigably abundant  in my initial research. I decided to narrow my talk to the psychology of estimating; simple things that can to to trick your brain to get you to better estimates. 

I was reading "Thinking Fast and Slow" at the time, and it seemed to fit perfectly with this decision. Half of this mighty book was about professional forecasters and how our environment influences our estimates. It seemed like the perfect match for this presentation.

I started out with an outline based on that book, then I was going to fill it in with details from "Demystifying the Black Art". Once I started reading "Demystifying", that was it... the book was perfectly suited to my presentation. The new outline literally flowed out in an half hour sitting.

It was also fortuitous, because as I was talking to people about the cognitive biases associated with estimating, I got frequent warnings about informing people how poorly their brain works. With these warnings, I was pretty sure that presenting about cognitive biases, it would likely be at best, completely ignored, and at worst, a pretty sizable flame war.

So after several practice presentations to my wife, my mom, the most experienced estimator on the Twin Technologies team, and finally all of  the Twin Technologies team, I have the presentation that is presented to you at 360Stack.

---

One nice discovery that came about this from this presentation is the "How good of an estimator are you" slide, with the 10 questions. When I was in college, my professor gave us these 10 questions as a quiz. He counted the score as part of the grade, which I wasn't pleased with, as I was average for this test, which was 30%. It was comforting to discover  that 30% is average, even if he didn't mention that fact, and even 20 years later.

---

Cross links to this blog / presentation

• http://www.twintechs.com/2013/07/sharing-our-software-estimation-expertise-with-the-tech-community/
• http://aplnhouston.org/index.php/chapter-meetings/upcoming-meetings/299-february-chapter-meeting-february-20-2014

Feedback and comments

• Fantastic volume of content crammed into a 50 minutes session."
• Very practical advice on how to estimate projects. One of my favorite sessions."
• best preso ever!"
• All around really great! Slides, topic, presentation, speed, q&a, all good
• The content was well thought out! You did the #1 rule in presenting (tell 'em, tell 'em, tell 'em).
• Getting access to the slides, opening my eyes to the importance of spending extra time on estimates