Start with ChangeYourOrganizationDiary. See also ChangeYourOrganizationTactics. ---- My organization is like a lot of organizations. It has good points and bad points. I think the good points outweigh the bad, which is one of the reasons I'm working so hard at changing things: I'd like to stay here for a while. The main reason, though, is that I get grumpy easily. I don't like working in environments which make it hard to produce quality software. On the good side, this organization is willing to change. There's quiet interest in agile development, but not a lot of knowledge of what that means. There's acknowledgement of problems and desire to improve. The people in positions of authority are generally easy to talk to and are open-minded. On the not-so-good side, there are a lot of barriers to creating quality software. People specialize in technologies and are composed into project teams. (This is called MatrixManagement.) WaterfallProcess is the primary methodology. There's some bullpens, but quite a few cubicles. It's nearly impossible to PairProgram at the desks. The worst problem, though, is communication. I think it may be a result of MatrixManagement: the UI people work on the UI stuff, and the database people work on the database stuff. They go to separate meetings and have separate managers. It gets worse: project managers are a separate group ''that sits on the opposite side of the building.'' QA is on a whole 'nother floor and typically doesn't get involved until a project is coded. There's no real concept of team: a project brings together a UI person, a database person, a project manager, and a QA person. They work away, emailing back and forth and going to lots of meetings, until the project is done. Then each person goes his separate way. Worse, everybody's assigned to multiple projects simultaneously. There's really obnoxious time tracking requirements to go along with this. As I look back on this list, I'm encouraged. All the good things are personality related: it would be impossible for me to change them if they weren't good. All the not-so-good things are culturally related: they're aspects of the way the company's used to doing business. Hard to change, but not impossible. There's hope. ---- I started this diary on Nov 27th (2002), so the following is from memory. October 1st. I get hired. I talk with the person I'm replacing for half a day, and then he leaves. Today was his last day. I have a desk and a computer. Time to get to work. October 4th. One of the other people here who's into XP is working on an automated build script. There's hope. ~ October 7th. It doesn't take me long to identify the communication problems. I don't feel like I understand what I'm working on or how it relates to anything else. I address the immediate problem by picking up my computer and moving into the bullpen with the other programmers on my project. Fortunately, I'm only assigned to one project right now. With the benefit of hindsight, I can say that this was a great move. I've picked up a lot of information about the project just by being in place to overhear conversations. ~ October 21st. The project I'm on is in its final stages. We're trying to fix defects so we can release to production. We have an hour-long meeting to discuss defects every day. I try to resolve the problem by writing my defects on cards and taping them to a whiteboard. I say, "If you want to know my status, just look at the whiteboard. Everything I'm working on is on there." I estimate costs for my defects as well. ~ October 24th. No one else is writing their tasks on cards. I stop estimating. ~ November 4th. I keep talking about process improvement and agile methodologies. I'm trying not to be a ChronicComplainer but I fear that I slip into that role occasionally. On the other hand, whenever I do complain about something, it's in the context of how we could fix it. Still, I think I'm probably being at least a little obnoxious. It's hard to curb my tendencies, though. Maybe writing this diary will allow me to get it out without irritating my coworkers. On the other hand, my comments are being noticed. Or maybe it's because I was able to start fixing defects within the first week. Either way, I'm being invited to more architectural-level meetings. ~ November 7th. I take down my cards. The cards got noticed by other people, in a sort of "What's that odd collection of cards on the white board" sense, but they weren't really used by the project manager. The other two people on the team weren't following my example, so there wasn't really a point. I keep using the cards, but I just keep them on my desk. ~ November 12th. This organization, like many out-source organizations, wants to increase reuse by building a framework. I had a discussion about process with the architect of the framework yesterday and we had a meeting about it today. My points were well received, and when I suggested that we make them official policy, I got permission. So as to not lose momentum, I wrote up a one-page "official framework policy" document! We now have an agile process for framework development. The process is very, very simple. It goes like this: Anybody can work with the framework, but * you have to work in pairs * you have to develop test-first. Before my input, we were going to have a hierarchy of people who were trusted to permitted to modify the framework and people who were not. I convinced them that making it difficult to modify the framework would result in a StovepipeSystem. That opened the door for an agile approach, and the all-important introduction of ''required'' PairProgramming and TestDrivenDevelopment. The not-so-good part is that the "official policy" is not so official, as yet. I got permission that this be the official policy, but then I drafted it and sent it out. Nobody has officially anointed it as official. So to speak. ~ November 18th. I sent an email to the director a few weeks ago saying that I have experience with process improvement and describing some risks I saw with specialization. Today we had a meeting. It was a great meeting in that we got along well and discussed a huge variety of topics. I think I established my credibility. It was not such a great meeting in that there was no call to action. There's an interest in change, but in cautious change. I can't really blame him. I left feeling that I had implicit permission to make minor changes. That's not what he said, but now I feel like I'll be given some slack if I do. I don't have any real authority, but I've always been good at talking people into trying new techniques. ~ November 20th. The task-tracking whiteboard has spontaneously come back to life! No index cards, just markers. That's okay. The cool thing was that I didn't do anything. ~ November 22nd. I've converted the whiteboard from markers to index cards. Not much resistance... erasing and moving around marked notes was getting messy. Today, Nov 26th, the board looks great. November 25th. The other XP guy has installed a Wiki. It's being adopted enthusiastically by several people. It looks like it will fly. November 26th. I've introduced the idea of prioritization on the tracking whiteboard. I keep asking the project manager to order my cards by priority... and I think it's starting to catch on. I've also gotten the QA guy to submit tasks through the project manager for prioritization rather than just emailing me all the time. ---- This brings us to the present. Future diary entries should be a little more detailed, and will talk about the problems I face in more depth. The brief entries up to now have made things look straightforward and easy, I think. '''27 November.''' Today was a very good day. For the last two weeks, we've been having a growing problem with FireFighting. We're in the very last stages of releasing a product, and QA keeps discovering defects. Defects are scheduled according to what the customer feels is critical. Everything critical has to be included in the next drop. Drops take two hours because our builds aren't automated. We drop twice a week. As we've gotten closer and closer to the 'final' drop, the pressure has mounted. Yesterday , we got stuff that "absolutely has to go in!" the morning of the drop. Today, my last day before the next drop, I got a few defects that "absolutely has to go in!" an '''hour''' before the end of the day. Up until now, I haven't been reacting to this insanity well. The project manager's approach to the problem has been to ask us (actually, to tell us in the ''guise'' of asking us) to stay later, come in earlier, work on Thanksgiving, etc. Yesterday I snapped and refused. I told the project manager that we needed to solve the firefighting problem, not to ask programmers to work extra hours. No, I wouldn't be coming in early. True, yes. Smart, no. I realized my mistake right away and called the PM to ApologizeUnconditionally. I'm glad I did: things were friendly today. So why was today a very good day? Because the same situation came up today, and I handled it much better. I think my approach got us one step closer to an agile process. As I mentioned earlier, an hour before the end of the day, the PM brought in two new defects. I dutifully wrote them on cards, got him to prioritize them, and taped them to the whiteboard. For the first time, I also estimated them, at two hours each, and told the PM that I could probably only get one done before I left. The PM went back to his cube (on the other side of the building). A few minutes later, I got a call: : PM: Those two new issues are absolutely critical. They have to go in to the next drop. : Me: I only have time to do one before I leave. Which one's most important? : PM: Both are critical. They won't accept the build if both aren't in. They have to be in. : Me: Well, I don't see how that's possible. I've estimated them at two hours each, and I don't have that much time. : PM: Can you come in on Friday? (Friday is traditionally a holiday.) : Me: No, I'll be out of town. : PM: Okay, well, they have to be in. Can't you just do them tonight? : Me: I have to meet someone at the airport. The latest I can leave is 6:30, which doesn't give me enough time to do both. : PM: It's critical that these two things be in the drop. Can't you just come in on Friday? : Me: No, I'll be out of town. : PM: Who else is qualified to do work on these issues? : Me: Joe or John. : PM: John's wife just had a baby, and Joe's going to be at the beach on Friday. If you can't do these tonight, I'll have to call John or Joe. : Me: My estimate for these two tasks is four hours. I've estimated them in the best way I know how. I don't see how I can get them done tonight. : PM: Can you come in for just a few hours Friday morning? : Me: I'll be out of town. : PM: Where? : Me: (My home town, two hours away.) At this point, the breakthrough occurs. The PM accepts that the schedule is fixed. He starts negotiating scope! : PM: Okay, is there any way you can get these done more quickly? : Me: Hmm. Maybe. : PM: Can you just solve the problem in this limited case, and not the general case? : Me: Yeah, I think I can do that. I'm still not sure I can get everything done in time. : PM: Okay, let me know how it goes. It turns out that my estimates were high and I was able to get everything done with an hour to spare. That's not the thrilling part, though. The thrilling part is that: * I didn't lose my temper as I had yesterday when faced with the PM's pressuring. * I was able to use my estimates as a foundation for a calm, non-confrontational approach to the problem. * The PM stopped trying to fix schedule, scope, and resources! He accepted (eventually) that the scope had to be changed, and worked with me to find a win-win solution. Best of all, we had a brief discussion just before I left. In that meeting, the PM said he wanted to have a meeting after we delivered this version to estimate and plan all of the tasks for the next version. Perfect! It shouldn't be hard to turn that exercise into a PlanningGame. ''Question: If your build system is such a nightmare (two hours for a "drop", with a complicated list of instructions), why don't you fix it? On my previous assignment, we had build instructions that involved hopping from this machine to that machine to run the various commands necessary to build the system -- clumsy, fragile, and annoying. You also can run into problems where building a version of the system out of version control doesn't give you the same system, because the build rules have changed. I changed it into "push the button and it goes" system -- still a bit clumsy, perhaps, but now it's revision controlled and, most importantly, doesn't eat up a programmer hour just to compile the system.'' My answer to this question is clear in my own mind, but I'm having trouble expressing it. Everybody in my organization already knows that fixing the build system is a good idea, but they're not making it a priority. I hate to drag out a hoary old cliche, but think of it this way: fixing the build system would be like giving the organization a fish. Instead, I'd like to teach the organization to ''want'' to fish. So I'll be successful when the organization says, "We're wasting too much time on manual builds. Hey, programmers, how much time would it take to automate this?" (Programmers answer.) "Okay, do it. We should recognize return on that investment by the end of this month, so let's tentatively add these two items into the release plan for the next month." I'd even be happy if they said, "No, that's too much time. Keep doing it manually." Any answer would be fine -- I just want conscious consideration of issues like these. To achieve this goal, I'm doing several things: * When a drop is planned, I write it on a card and put it on the whiteboard * I measure the time it takes to do a drop, and use that for estimates * I frequently comment about the amount of time a drop takes * I had this conversation in earshot of several people on the project: * Programmer: Thanks for putting the build checklist on the wiki. * Me: No problem. * P: I've wanted it for a while, but the other guy never had time to do it. * Me: Well, I didn't have time '''not''' to do it! * Laughter all around, further comments (not by me) about not having time to ''not'' do things But none of this was really planned. I'm not cooly calculating everything in the back room, manipulating people and events with precision. I'm just following my instincts and experience, reacting to situations as they arise. Until I answered your question, I didn't think about why I wasn't automating the build. I just knew that it wasn't a priority for me. ''Well said. Note the use of the term ''priority'': The build system may be horrific, but other things may be even more important. This is an extremely important principle, and is often forgotten in the rush to do the RightThing. -- BrentNewhall'' ---- '''2 December.''' Every day, I make a tiny bit of progress, but I'm also reminded of how far I have to go. It gets depressing. If I were offered a coaching job, I'd take it in a heartbeat. On the other hand, I'm not actively looking for other opportunities. That either says that the situation is hopeful, or that I'm lazy. (Or stupid. My efforts could well get me fired.) What gets depressing is not that the organization is backwards. That's pretty common. What's depressing is that I have so little control. I know that, given the authority, I could turn this project around. In six months, I could have the whole organization dancing to a happy and productive little two-week beat. I just don't have the authority. There were a few things that have been working in my favor. The last two weeks have been a debacle for my project. Quality problems up the yin-yang. People working until dawn over the Thanksgiving holiday. Stuff like that. It's too bad for the project, but it's good for me, and it's good for the project's long-term health. This was bound to happen with the way things were being run. I'm glad it happened sooner rather than later. People are a little more willing to change, and I can make my suggestions with a little more authority. When I made suggestions today, there was more understanding and no disagreement. I didn't try to implement any radical changes, but all the suggestions I made were implemented. Starting tomorrow: * The project manager and the QA lead will be sitting in our bullpen * On-site programmers (all both of us) will be estimating tasks Yay! This actually gets us pretty close to where I want us to be. I want: * Collective task ownership * XP-style iteration planning * Fixed-length iterations * Process introspection and improvement * Various code quality things, like TestDrivenDesign and MercilessRefactoring * Customer communication improvement We're really close to getting collective task ownership. The idea of iteration planning is well planted, but not quite germinated yet. Fixed-length iterations will take a while. I think iteration planning needs to be well-established first. Active process and code improvement won't happen until the project gets out of FireFighting mode. Customer communication is a big barrier with its own set of problems. I don't think I can do anything about that until we have our own house in order. Some agile ideas are in place, but they're very fragile. I've gotten them to sprout, but if I were to leave, they would wither away: * Central, visible planning whiteboard * Task-based prioritization with index cards * Task-based, small-grained estimation * Colocated team, including "customers:" PM and QA lead Tomorrow will be an interesting day. I made some big strides today, relatively speaking, but they were in the aftershock of a really hellish week. Will the changes stick? They might, but it's far from certain. ---- '''3 December.''' Today was a surprisingly boring day. It shouldn't have been -- we had a major issue crop up, and today was the last day we could do a drop before releasing to production. But everything proceeded calmly. I don't think I can take all of the credit... but maybe I deserve it. The same scenario in the past has been a nightmare. All of the initiatives I've pushed for played a role in our success: When I arrived today, a number of issues were sketched out on the whiteboard. I talked to the PM, wrote the issues down on tasks cards, got him to prioritize, and estimated with the help of the other on-site developer. There was a major issue, so I paired with another developer that we occasionally work with, and we worked through the day. At three o'clock, I overheard the PM say something about doing a drop. This was the first I'd heard of it. When he got off the phone, I asked him whether we were doing a drop today. When he said yes, I told him that I wanted to get home at a reasonable hour tonight, which meant that we needed to start the drop by 5:00. Did the new issue on the board (previously prioritized as "least important") need to go in? He answered yes, so I told him that since I had estimated it at two hours, I would have to stop working on the major issue that I had been working on all day. Fortunately, my pairing partner was completely up to date on all the issues and could continue working on it without me. Because we had calibrated our estimates by that point, the PM knew my logic was sound. He didn't argue. I started working on the last-minute task, but did so with a relaxed feeling, confident that I had enough time. My pairing partner continued working on the major issue alone. I finished the last-minute task around 5:00. I confirmed with everybody that I was going to do a build, and then printed out the build checklist I had put on Wiki. The PM had updated it earlier to correct a problem with the previous night's build. The build went smoothly and I left right on time! *** Looking back at what I've written, it reads like one of those oh-so-annoying made-up examples of a new process in action. But it's all true. I know there's going to be lots of problems ahead, but I think we may have turned the corner. As I said in my introduction, everything I've put into place was used today: * Central planning whiteboard * Task-based planning * Task-based estimating * Colocated team * Reliable estimates, leading to respect for developer estimates * And even a smidgen of active process improvement, in the form of updating a checklist on the Wiki There's lots more to do, but I think I may have found the minimum process necessary to stem the chaos. ---- '''4 December.''' Code freeze today. Nothing interesting happening. This sort of feast/famine cycle has been common. We go into production in a few days, and we'll all be standing by, prepared to support that effort. ---- '''11 December.''' Code finally went live. Yep, it took a week. I spent most of the time fighting fires, but I did have time to write an essay about improving communication by collocating programmers, QA, and project manager. I was careful to propose a very small change: just collocating people. I didn't mention unit testing, pair programming, or iterations. I see collocation as a necessary first step, so that's what I focused on. And, if I could only choose one thing, collocation would be it. Communication is a big problem here. The other XP guy here thinks I'm being too cautious. We'll see. Previous attempts at top-down change have resulted in disinterested responses and comments about changes being too big. The XP guy thinks my timing was just off... now that we're finishing a product cycle, now is the time to change the process. Also, the fires have been so bad, they're impossible to ignore. He called a meeting yesterday (that I couldn't go to, darn it) and process issues were discussed for an hour and a half. No one else has given me any feedback on the essay yet, but that's because there hasn't been time. ---- '''12 December.''' Reaction to the essay has been universally positive, but only mildly. Everybody has said they liked it, but I've had no real feedback. It must have made an impression on someone, though, because I saw one of the managers that I ''didn't'' give a copy to carrying it. ---- '''15 December.''' Lots of good things happened today, but I'm short on time and can't discuss them in depth. But basically, I'm starting to see my labors bear some fruit. On two separate occasions today, I had the opportunity to describe my ideal process to managers. The whole nine yards: bullpen, pairing, test-driven development, me coaching, my past coaching experience. Why haven't I done this before? '''I was never asked.''' The last thing I want to do is shove big ideas down someone's throat. I've been waiting for the right moment ever since I started working there. It's interesting that it arrived twice on the same day. Oh, and both times, the response was active interest and talk about needing a new project to come along so we can do it. The other good thing that happened today was on the bottom-up front. My current team had a planning game! It took nearly four hours, which is typical in my experience. It started out a little rocky, as we were figuring out which tasks to include, but then went very smoothly once people started volunteering for and estimating tasks. I also had everybody estimate their own velocity, for use over the next week. Next week, we plan on having another planning game to check velocities and update estimates. I got the planning game to occur by talking about it over the last week or so. I kept saying to anyone who would listen that I wanted to have everyone sit down before we started the next version to look at the tasks and estimate them. Then today, my comments resulted in an architect here asking the team lead to schedule a meeting to discuss features to be included in the next version. Before that meeting, I talked to everybody I could, saying that I wanted to do something called a planning game, and here's what it was. I got buy-in from the team lead to do so and lead the meeting. As people were gathering for the meeting, I told the project manager and architect that I had been talking to people about doing a planning game, here's what it was, did they mind. It was hard work setting it all up, but it paid off. I feel like things are building up to a head here. I wish I could say why. I've effected some change on my local project, but it's been very sporadic and labor intensive: the traditional "two steps forward, one step back." But I'm also seeing wide interest in process improvement that isn't directly related to what I've been doing. I'm not even certain that it's a result of my efforts. If it is, it's because I've been continually talking about process improvement to anyone who will listen. Maybe I've raised people's awareness and interest without them directly realizing it. I don't think anyone would point to me if asked why the organization is suddenly so interested in process improvement. ---- '''16 December.''' Remember how I said it's been "two steps forward, one step back?" Today was a step back. The project manager and QA lead are no longer sitting in the bullpen. The PM is ignoring the cards we prepared Friday and has gone back to handing out spreadsheets with "commitment dates" on them. I'm pretty sure he's added tasks to the list as well, or at least shuffled owners around. I asked the PM if would mind sitting in the bullpen with us again, since it was so helpful last time... he was clearly reluctant, and said something about needing to make sure space was available. Days like today are depressing. But now that I've been off work for a while, I can apply a bit of perspective. Why is the PM resisting? I think he knows, deep down, that the cards and bullpen are effective. So what's going on? Well... * Sitting in the bullpen is probably less comfortable than sitting in his cube. There's less space and he has to use his laptop rather than his desktop. Furthermore, he misses phone calls. Also, there really isn't room. He has to stake out a narrow stretch of desk between two other people. * The cards a bit harder, but perhaps their informality turns him off. Plus, I might be acting like a threat: scheduling has been his bailiwick, and here I am taking that over with my cards. Also, the cards are not electronic, something which has invariably made people uncomfortable. And since he's not sitting in the bullpen, the cards are also not visible to him. So what do I do? (I wish I could just apply a RolledUpNewspaper. I feel like I'm trying to prop up a soggy mattress: the part I'm holding on to stays up, but everything else droops.) ---- '''17 December.''' Struggling back up... today I got the PM to prioritize the task cards we created Friday. It wasn't easy... I had to gently prod several times. All the programmers were very happy to have them prioritized, which was gratifying. I would say that the programmers are very happy with the index card approach. The PM is not actively against the idea, but is being passive aggressive about it: basically, he ignores them and does his own thing. I think he's definitely uncomfortable with the task cards, but I'm not sure why or what to do about it. If you're reading this and have an idea, I would like your input. ''...what are your personal goals in this situation? (long term, medium term, short term?) -- AnonymousDonor'' : Short term, I just want to improve quality of life for myself and other programmers here. : Medium term, I want to be in a leadership role. Preferably coach to a team of eight or so people. (I'm particularly eager to get here because I've been leading teams for several years. This job has been a step backwards for me, career-wise.) : Long term, we'll see what opportunities come up. I wouldn't mind winning the lottery. : ''What do you see as the three or four most-bang-for-the-buck potential programmer-quality-of-life improvements? (Sounds like co-location is one of them...). The reason I am asking is that maybe one way (and time) to get the PM to change is to make the easier changes first, and let him jump on the bandwagon himself once it is in motion. Just a thought, not from personal experience...'' : ''Does the PM benefit if you achieve your short and medium term goals?'' ''Have you considered DigitalStoryCards? Failing that, how about talking to the PM about it? You could just say nicely, "You seem a little reluctant to use the StoryCard''''''s...any particular reason?" -- BrentNewhall'' Hmm... just ask. How novel! ;) I'm reluctant to do that, though, and I'm not sure why. Maybe because I want to avoid confrontation? Maybe because I'm afraid of what the answer would be? Something for me to ponder. DigitalStoryCards: I've never liked them, to be honest. My answer to "IsAnythingBetterThanPaper?" is "Almost never!" Obviously I'm biased. But my rational answer is that I want to avoid technologies that give people an excuse not to sit together. I'll keep them in mind for the future. ---- '''18 December.''' Today I learned that the organization boss has acceded to a real bullpen! The details aren't settled yet, but it's probably going to combine several small, related teams into one, and include QA and the project manager. Just what I've been wanting! Now I can emphasize the next change more: a bullpen doing mostly-XP with me as coach. The boss is going to want a detailed plan for migrating to the bullpen environment, so my next step will be to create that plan. It's going to be a big project, so I'll probably do it over my Christmas vacation. Hopefully they won't proceed while I'm gone. So, why did this happen? I asked my champion the same question. (Uh... champion? It's a word I just sort-of invented: somebody I work with who has more clout than me and thinks along generally the same lines I do.) He said that he wasn't sure, but that the boss was pretty open to the idea. He thought it was just because lots of people were talking about the idea. I can't prove it, and it's oh-so-unhumble of me to claim this, but I'm pretty sure that was my doing. I've been talking about bullpens a ''lot'' recently. Hmm... a side note. When I say I've been talking about something a lot, I mean a few times a day. I actually don't talk about it that much, relative to the other stuff I do. The majority of my day is spent doing my actual work. Every so often, a legitimate opportunity to say something change-related comes up. By "talking about ''X'' a lot," I actually mean that the majority of my change-related comments are about ''X''. What does a legitimate opportunity to say something change-related look like? Here's some. I've put the ones that I think result in the most change at the top. * Something goes wrong. I say, "Let's prevent this in the future by doing ''X''." ("Next time, let's try ''X'' so that doesn't happen again."; "''X'' is a good way to solve this problem; "We should do ''X'' so this problem doesn't come up next time"; ...) * Something irritating or tedious happens. I say, "If we do ''X'', we won't have so much tedious ''Y''." * Something vaguely related to process comes up. I say, "''X'' is something related that we should try some time." * I feel like ranting. I've tried to stop doing this, because I think it tends to hurt more often than it helps. I like to rant, which can be a problem. After my intro, I say a few sentences about ''X'' and then let it rest. Every so often, I'm asked a direct question about process. These occasions are rare but wonderful. I get to spew all kinds of stuff. Impact-wise, it's probably about the same level as "something vaguely related comes up." But since I get to explain my viewpoint in more detail, it probably raises my credibility a bit more than the few-sentence approach. I also write papers. So far, I think I've averaged one every few weeks. The papers are targeted at people who are higher in the organization and generally have a direct call to action. So far, I've written them after I've already seeded the change pretty thoroughly via one of the above methods. No one has ever responded to one of the papers and said "Yes, let's do what's in this paper." But so far, the main actions recommended in all of my papers have been followed. I don't know if that's because of the word-of-mouth, the paper, a combination of the two, or just blind luck. So far I've written three papers: * A process and rationale for evolutionary architecture on the organizational framework (12 Nov). * A recommendation that we use agile development and collocation instead of building an up-front architecture (~22 Nov). * A call to collocate developers, QA, and project manager in a bullpen (11 Dec). Finally, I should emphasize that everything I do is based on moment-to-moment instincts. What I've written above isn't a set of rules I follow; it's an after-the-fact analysis of what I'm doing naturally. I think I'd come across as artificial and false if I tried to follow the above paragraphs by rote. ---- '''19 December.''' Feeling particularly burned-out today. No real reason. ---- ''(vacation)'' ---- '''7 January.''' I'm approaching a critical moment in my time here. Since I've started, I've been uncertain about what form of "ChangeYourOrganization" this diary would end up demonstrating. I think the next few weeks will tell me the answer. Coincidentally, other contract options are starting to open up. I returned from my vacation yesterday. My normal duties have been light, so I've been working on a document that describes a specific XP-like process and reasons for switching. I've hinted at this before, but this is the first time I've been this blunt. The timing is perfect: we're about to start several new projects that need this process and the project I'm currently on is winding down. I've been here long enough to earn respect and to gain an understanding of the company and its customers. If I can't make a significant change now, it's unlikely I can do so. The above sounds a little strident. That's not the attitude I have at work, or at least I try not to. I think this is the best possible time for me to make a change, so I'm doing everything I can to make sure it's successful. I finished the first draft of my paper today. I passed out a copy to a few people I trust and tomorrow I'll go over it with my champion. I plan to be clear that I consider this paper to be the culmination of my efforts and that it's important to make sure everything's correct. I'm hoping that, with his help, we'll fine-tune the paper and then (somehow) present it to management. I'm not really sure what will happen or how I expect this paper to help. Previous papers have been well received, particularly the last one, so I'm continuing in that vein. I did a pretty good job with this paper, I think, so hopefully it will have an effect. One thing I know is that there is a genuine need for the changes I'm suggesting. If the organization doesn't change, it will have difficulty dealing with the projects coming up. So... what happens if this paper doesn't succeed? I have a few options left. First, the most likely result is that there's a lukewarm to positive reaction, but no actual change. I've never gotten a genuinely negative reaction. (Well, once. But that was a reaction to the way I did something, not the basic suggestion.) The reactions are generally of the "good idea in theory, but you don't really understand what we're facing here" nature. Not that anyone's actually said that. I have never actually gotten into a serious discussion with anyone with authority about what kinds of change are necessary, and what is feasible given the needs of the organization. As I was saying, if there's no real response to this paper, then I have a few options left. I could talk to my main manager about my lack of responsibility and influence. I could tell him that I feel I'm being underutilized. We've had a brief discussion along these lines already, but I could be more pointed about it. Another option would be to talk to the director, the person with the power to make these sorts of changes, and take my case directly to him. I've talked with him once before (18 November) and it was positive. I don't really want to take either of these steps. I'm not sure why, but I feel like these are last-ditch efforts. If I make my case and the answer is "no," then that's it. I've taken it as far as I can go. Eventually, that conversation may need to happen. But when it does, I'd like the decision to already be clear... and in my favor. ---- '''8 January.''' Went over my paper with my champion today. Got the reaction I expected: agreement with the basic concepts, disagreement that it was possible in this particular organization. Ironically, he talked about the difficulty of change and how some people resisted change without knowing it. :) I kept at it, though, and eventually he agreed to run it up the proverbial flagpole with management. I could have done that myself, and maybe I should have, but I don't have a day-to-day relationship with the other members of management yet. Maybe I should start working on that. I did learn one interesting thing: my champion, along with his counterpart in project management, has the authority to make the changes I'm recommending. I'm going to let it sit and come back to it next week. In the meantime, I still have a lot of spare time. Maybe I'll start establishing a unit testing beachhead. ---- '''14 January.''' I've been going through a lot of ups and downs lately. I alternate between extreme burn-out and calm patience. The paper is making some rounds... I'll be talking to someone with influence about it tomorrow. This is actually the first person who has questioned the things I've said, which I take to be a good sign: it means he's actually reading it, not just smiling and nodding. ---- '''16 January.''' As I mentioned in my entry from the 14th, I talked to someone with influence over process about my paper yesterday. The upshot was that he and I have the same basic perspective on process, although I'm a bit more radical when it comes to agile development. We agree on what the major problems facing the organization are. We even agree, more or less, on the first step. Unfortunately, that's the point where I hit a stone wall. I don't know what it is about this company: everybody happily agrees with me, but that's as far as it goes. Nothing happens. I pushed and pushed for action... gently... and the response was essentially this: "There's nothing you can do to help at this time. Please wait an indefinite time for an indeterminate result. At some point we'll let you know, and you'll have 'the important conversation' with the guy in charge." At first, I was very disappointed with the lack of results. I still go through times every day where I feel completely helpless and burned-out. But now I think I'm just going to sit, wait, and see what happens. Communication failures are already starting to occur in a big way. We had some problems today with people accidently overwriting each others' files... exactly the kind of thing I predicted in my paper. (Unfortunately, I just predicted communication problems, not this specific issue. I wish I had!) I hope people will connect my predictions with what's happened, but I get the impression that people think I'm writing these papers from some sort of idealistic viewpoint, not a practical one. I might say "I told you so!" eventually, but not just yet. I've bruised enough egos with my recent push. Now it's time to just put my head down and observe. I'm going to focus on low-level, tech-oriented change for a while. I've started introducing unit tests for our UI components. I may be petty, but I'm going to enjoy watching things fall down for a while. I did everything I could to stop it, but I was ignored. I don't feel the slightest bit guilty about my enjoyment. Maybe I do feel a bit bitter. Any advice, O Readers? (Assuming there's any left! Add your thoughts to the bottom of this entry, please!) ''A familiar frustration. See e.g. http://www.moebius.nl/inch/index.php/TheDoomedProject. '''(BrokenLink 2004-06-28)''' Feel free to get in touch privately if you want to compare notes. -- LaurentBossavit'' ''It might help to document the failures as they occur, noting why each failure would not have happened if your suggestions were adopted. It would give you something concrete you could point to when people ask "how would your changes help us?" (Disclaimer: I haven't been in this situation.)'' : Interesting idea. I think I'll do that, but I'll keep it to myself for a while. I can see how that list would backfire. -- Diary author ---- '''17 January.''' A couple of notable successes today. First, and most enjoyable: I successfully created some unit tests for the UI portion of our app. Although this may not have much impact on the organization, it felt good to write some quality code for a change. Second, management has just done a re-org and is moving people around. This involves shuffling cube walls. I got word today that the new floor plan includes the bullpen I've been pushing for. I'm not as thrilled as I could be, though, because the project managers also moved... to a different floor. And the new bullpen won't have room for all of the programmers on the project. And QA won't be included. Sigh. It's actually not much different from what we already have. But I can keep writing quality code. ---- '''22 January.''' The bullpen idea is forging ahead. Today, I talked with the person in charge of the new floorplan, who is also the manager for the DB guys here. (We're split into UI programmers and DB programmers here.) It was a great conversation: constructive, to the point, and resulting in action. It was wonderful. I've never had a conversation like that with management here. The overall floorplan is excellent: DB and UI guys are going to be put together into little pods scattered across a single floor. Although this might not sound like a good idea, it's actually perfect: the isolation will encourage the formation of teams. Each pod has about ten people in it, which is just the right size. You see, one of our biggest problems here is MatrixManagement: part of that is the idea that when you were Created, you were Created as a UI Programmer or a Database Programmer... and Thou Shalt Not Fraternize with the Other Side. This is a wonderfully subtle attack on that problem. I wish I had been responsible for it. So I went to talk with the guy in charge of this today about PairProgrammingFacilities. First thing I did was tell him how much I liked the new layout. I did that partly because I figured that most people would be reacting with complaints about how their new area wasn't as good as the old one. Then I told him that with the current situation, it was hard to get two people in front of a computer for collaboration or pair programming. I gave a couple of examples of when I had been trying to work with someone but couldn't do so comfortably. He agreed with me, and asked for some alternatives. So I sketched out some ideas that focused on getting rid of the interior walls in the bullpen and using big rectangular tables. He raised some objections, I agreed, we batted around a few more ideas, and finally came to a conclusion that should work. Meanwhile, we talked about the advantages of pair programming, ConvectionCurrentsOfInformation, and how people would react to the reduction of personal space. I came out of that meeting feeling like I had actually made a difference. That's a first for this organization. I hope this conversation results in actual action. I'm going to follow up on it next week. ---- '''23 January.''' : Q: How many psychologists does it take to change a light bulb? : A: Just one, but the light bulb has to ''want'' to change. Today I was having a conversation with my champion. You know, the guy who has the same basic viewpoint as me but has more pull with management. He was talking about one of our clients. "They want us to reduce development cycles and respond more quickly to changes," he said. I smiled, knowing what was coming next. That's a textbook reason for instituting an agile process. He continued. "That's why it's important that we get the framework right." I was floored. I've been talking about agile software development for three or four months now. Along comes a classic customer request -- the exact request agile development was invented for -- and this is the conclusion? I've clearly screwed up. My approach has been fundamentally flawed. Since he got here (which was before me), my "champion" has been advocating a framework as the solution to the company's problems. The framework's been developed. (It's not solving the problems.) I come along and advocate agile development and people-centric processes. Of course I'm not getting anywhere. I'm not saying anything that can be heard. ''Frameworks are making my life more difficult too - I'm also trying to introduce XP (see OptimisticProgrammingSkunkworks), and the biggest problems we have are with complex software like WebSphere, WSAD (which puts proprietary patches over the open source Eclipse IDE) and STRUTS. I'm lucky we're not getting pressured to use EJB. These are SilverBullet''''''s as conceived in the image of a bureaucracy. -- TomRossen'' ---- '''29 January.''' Remember the manager I had lunch with on the 15th? He's the one who told me to sit tight and be patient. Today, I ran into him in the hall and we talked for a few minutes. One of the things I asked him about was whether my efforts were making any difference. He said that they were, but at a level I can't see. He said it was "aligning the conversation." Come to think of it, I'm not sure what that means. His statement didn't really make me feel better or worse. I have seen things change, and in the direction I've proposed. They've changed glacially slowly, and only partly in the ways I've suggested. But they're changing. What's less clear is that it's as a result of my efforts. Probably, yeah... I can see that. But the manager may have just been sweet-talking me. He knows I've been unhappy, and he's recently described his job as "talking people off the ledge." Actually, I'm not sure what that means, either. ---- '''31 January.''' From conversations with a fewer higher-ups today, I got the strong impression that people think my ideas are not evolutionary enough. They think I'm trying to do too much at once. I had two people today convey basically the same opinion, so I think it's something that's going around. It's true. I have been advocating fairly significant changes. Maybe it's impatience. I see so many problems that could be fixed if I were just handed the reins of a team and told, "go." (Judging from that sentence, there's probably some hubris to go along with the impatience. Now all I need to be great is a healthy dose of laziness.) On the other hand, it's hard for me to be evolutionary. I've suggested evolutionary changes, and some of them have gotten implemented. For example, on November 12th, I successfully put a very minimalistic agile-ish process in place for development of the framework. Problem is, it was too minimal. There was no coach. People didn't know how to do test-driven development. It didn't work. Another evolutionary change that I rammed through by sheer force of will is the process for the project I'm currently working on. The project is winding down, so a lot of the process has faded away. That made a difference for me personally, which was nice, but it didn't have any lasting effect on the organization. Back to my point: evolutionary change is difficult because there's so many interdependencies among agile practices. I think if I were more patient, I'd be satisfied with evolutionary change. I ''am'' making a difference, but it's imperceptibly slow. There's some sort of weird political shit going on. People keep hinting at it: it's gotten to the point where everybody I talk to agrees that my ideas are the way to go, but that even the director doesn't have the ability to make the changes required. Maybe that's a just a way of getting me to shut up, but if it's true, it's a serious problem. The director isn't doing his job. Okay, okay, that's harsh... I don't understand all the intricacies of the situation, etc. But let me compare this organization to another I know of. At this other organization, the director saw that there were problems with development. He considered it and concluded that agile development was the solution, but he got resistance from executives. He worked at it for a while, but eventually declared, "I'm the director of development. We need to switch if we're to be successful. It's my decision to make, and if you won't let me do my job, I need to step down." Guess what? That company is now several iterations into an agile development process. The pilot project has expanded into a normal project due to overwhelming success. Everybody is happy. My current company has an organizational disfunction which is preventing it from making decisions. Everybody, from the director on down, recognizes that there are problems. The director himself told me there were problems the day I got there. Decisive action is needed, but nothing's happening. I don't just mean in terms of agile development -- I mean ''anything.'' People in positions of authority need to stand up and say, "We need to do this, this, and this, or we're going to fail. Trust my judgement and let me do my job, or accept my resignation." It may be getting close to time for me to do the same thing. Today I was formally put in charge of a small project. It's due in May (but requires a full waterfall lifecycle), and is the first of seven related projects that require close communication. I'm worried about the risks on these projects. They require close communication, but don't have it. I formally raised the risk this evening. I said, "These projects have a real chance of failure with our current development approach." We talked about the difficulty of making change. I continued, "Failure wouldn't necessarily be a bad thing. It would serve as a wake-up call. The worst thing that could happen is that these projects succeed, but at great personal sacrifice." Too extreme? Too close to an ultimatum? Maybe. Last time I got in this mood (7 Jan), I wrote my comprehensive process proposal. That document was a failure: too aggressive. ---- '''3 February.''' Whew. Where to begin? Last Friday (previous entry), I formally raised the risk of failure on seven linked projects due this year. This morning, I had my bi-weekly meeting with my boss, and I raised the issue with him as well. I told him that the projects needed a single leader: someone who could co-ordinate all of the different people involved. He asked me if I wanted to do that. I told him I was concerned about the organization not being set up for the projects to succeed, and that I didn't want to lead a project unless it could be a successful one. So... this afternoon, the architect in charge of those projects approached me. He told me that had re-read my process proposal over the weekend, and thought it could be done. (The one I just recently dismissed as "too aggressive.") He asked me to lead the seven linked projects. I said I was interested, but I was concerned that I wouldn't have the organizational support I needed. I asked him some questions: would we have a bullpen; could I do iterative development; would developers be assigned to me full-time; what kind of leeway would I have; and then told him I would think about it and let him know more tomorrow. But before he left, I told him how much I appreciated his considering my ideas. Later that day, he introduced me as the tech lead for the seven projects. Well, okay then. :) I should be thrilled about this -- I have the ability to make a much bigger impact than I could before. Now I can demonstrate my ideas rather than just talk about them. But I really am concerned about whether this project can succeed. I'm concerned about organizational support. On the other hand, it's a golden opportunity, so I'm going to embrace it whole-heartedly. I'll start moving in an agile direction immediately. I'm not going to be aggressive about it, but I will be firm. Why? Because I think these projects will succeed or fail based on communication, refactoring, and schedule. I need to establish practices that enable these three things right away... not because we need to get started with them right away (although we do), but because I need to know if those practices can even be put into place. Basically, I'm trying to find out what the resistance is going to be to the practices I consider critical. One thing's for sure. Work just got a lot more interesting. Some notes about why I think I got offered this job: * I was calm. I laid out my point of view about failure risks in a calm, serious, but concerned manner. * I went through channels. First I told the architect. Then I told my boss. That was it. * I was clearly right. Everybody could see the problems that were occurring, even before any serious development had started. * I didn't push my point of view. I purposefully said that I wasn't pushing an agile approach, but just raising serious concerns. * I've got credibility. I've been establishing my credibility for the last four months. * I didn't ask for the job. Actually, I'm still not sure I want it. An even more interesting question might be: Why was this job created? I'm not entirely sure, but I think it was my steady concern, which was received and perhaps transmitted by my boss, combined to create a sense of real and short-term likelihood of failure, leading to growing worry on the part of the architect, leading to a feeling that something must be done, leading to... this. ''One thing to say: '''''CONGRATULATIONS!''''' :) Well, it beats beating your head against a wall, don't it? Remember this moment, and what you said ("I didn't ask for the job"), if (and when?) you start to feel that you're in over your head; you didn't ask for this, they asked you. You didn't misjudge the situation and get yourself in too deep, they judged the situation and determined that they want you in charge. So take charge :) --WilliamUnderwood'' : Thanks. :) ''Congratulations from me, too! What you have accomplished is no small feat, to set an organization in motion, and influence the culture "from below" to the point where change becomes an option. Wow! -- FalkBruegmann'' ''Sounds incredibly familiar to the situation that we have been facing over the past few months. We are a bit ahead of your timeline here, in the midst of our second XP project. Unfortunately, we have started to see the walls shrinking in on us.....our lack of Executive support has begun forcing us back into the Waterfall planning approach and status reporting structure. Old habits are hard to break, and due to the cross-functional nature of our project, we are being slowed by external stakeholders/resources necessary for our need to "Go Fast". Best of luck moving forward! -- ScottFerrell'' ---- '''5 February.''' I've done it. I've changed my organization. I've been looking forward to saying those words ever since I got back from vacation. The funny thing is, I was planning on following up with, "and I like my new job much better!" But I haven't found another job. It looks like I may have actually succeeded in changing my current organization. Okay, I have to admit: This declaration of victory may be premature. But things have never been so good. I spent most of the day talking to management and going over my comprehensive process proposal item by item, getting buy-in on each point. Yes, we'll have a bullpen. Yes, we'll have two-week iterations. Planning game. Test-driven development. Collective code ownership. Pair programming. Continuous design. Yes. Yes. Yes. Yes. Yes. Wow. I was this >< close to leaving. I had gotten all the way through the interview process with one company, and was getting overtures from two others. I've been privately committed to leaving ever since January 15th. I was fed up, burned out, pissed off. Some of that came out. I let it slip to a few people that I was on my way out the door. Maybe that's part of why things changed. I'm not really sure. I'll ask about that once things settle down. Today, I formally agreed to run the linked projects. I called the other companies and told them I was no longer available. Tomorrow, I'll go find all the people I'll be working with, explain the new process, and find a place for us to sit. It's going to be hard: the projects I'm running are the highest-visibility in the company. The client is difficult to work with. The programmers have a lot to absorb. The development environment is bizarre. This is a huge challenge. I can't wait. ''Yeah!'' ---- Continued in ChangeYourOrganizationDiaryPartTwoFebruary.