Now called, in XP: AcceptanceTest ----- FunctionalTest''''''s are programs or scripts configured to test that packages (groups of clusters of classes) meet external requirements and achieve goals, such as performance. They include screen-driving programs that test GUIs from without. FunctionalTest''''''s are a key component of software engineering, and ExtremeProgramming leverages them to permit the OnsiteCustomer to understand and track a project's growth. * Unlike UnitTest''''''s, which test the behavior of a single class, FunctionalTest''''''s test the entire system from end to end. * Like UnitTest''''''s, FunctionalTest''''''s help development go faster as well as ensuring quality. TestingLetsYouGoFast. A ChryslerComprehensiveCompensation functional test reads input from real system input files, pays one or more employees, exports results to real result files, then reads the result files back in and compares individual results to expected values. C3 has over 400 FunctionalTest''''''s. Specifying FunctionalTest''''''s is the responsibility of the customer, while UnitTest''''''s belong to the developers. On some teams the tests are written by the programmers, based on stories from the customer. Other teams use a separate team extracting functional test info from the customer and creating and running the actual tests. In either case, publish the results for all to see. A useful measurement of ProjectVelocity is the increase in number of functional tests, and the improvement of the scores. -- RonJeffries Errors caught during functional testing should first be treated to a SupportingUnitTest, then fixed, then verified with a FunctionalTest. ---- '''Benefits of functional tests''' AnnAnderson just told me of a project where her brother is a tester. They were supposed to have final code over two weeks ago, and just got it. They now have seven calendar days to "test the system" before shipping. (No, he doesn't work for MicrosoftCorporation.) In ExtremeProgramming, you build a continually growing collection of FunctionalTest''''''s. You run them all the time. You graph them ''daily'', showing total tests, green for correct, red for incorrect, yellow for not yet validated if you have that case (on C3 we do). The graph, on a larger time scale, should look like this over time: FunctionalTestGraph. More and more tests, more and more correct, leading to great certainty that the program is working. To make this work, the tests need to run automatically, and in finite time, such as overnight. To make them meaningful, you need a clean SystemIntegration every time you run the tests, such as daily. We call this process ContinuousIntegrationRelentlessTesting. -- RonJeffries It is very important for the customer, that your application does all the things he wants. So I tried several ways to check it. But every time I had to sit in front of the computer and think about the application. FunctionalTest''''''s gave me the possibility to check the application automatically. Pressing the start button, going home, and on the next day, I know if I am on the right way or not. ---- '''Functional tests and user stories''' How closely are FunctionalTest''''''s bound to UserStories? Do FunctionalTest''''''s accumulate completely independently, or is there some direct correspondence with UserStories? (e.g. "These three FunctionalTest''''''s correspond with this UserStory...") Is it the case that there's no need to list which FunctionalTest''''''s relate back to a particular UserStory? FunctionalTest''''''s are bound intimately to UserStories. They are the means whereby the users convince themselves that the stories are correctly implemented. Typically you will have several [to many] tests per story. We have the FTs in groups (suites) relating back to stories: some existing FT suites are: Third Pay Deductions; Field Car; Vacation Payoff; Reduced Hours; Foreign Nationals. There aren't any public C3 FTs, but they look just like one would think: "An employee earning $1500 per biweek requests 3 periods' vacation pay. He is issued a check for $6000 gross in the first period, then nothing for the next three." This is all put down in some cryptic testing language, or even in code, of course. Okay, so what have folk done about traceability between the originally agreed UserStories and the ultimate FTs that resulted out of them? Surely the functional tests are a little _more_ structured than the example above? -- ColinDavidMiller ---- '''Questions about functional tests''' * How do people do functional tests of GUI based systems? Presumably with some sort of automated tool. Does anyone have experience with such things? Yes, in Java. There are a number of Java GUI testing tools: see http://www.superlinksoftware.com/cgi-bin/jugwiki.pl?TestingGUIs and discuss them in http://groups.yahoo.com/group/java-gui-testing/ -- TomRoche Setting up a Rails project for outside-in development you can use Cucumber, RSpec-rails, and Webrat > sudo gem install cucumber rspec-rails webrat You will then have to run the following generators that ship with the gems: >script/generate cucumber >script/generate rspec http://github.com/dchelimsky/rspec/tree/master http://github.com/dchelimsky/rspec-rails/tree/master http://github.com/aslakhellesoy/cucumber/tree/master http://wiki.github.com/brynary/webrat --Wayne Simacek * Does the TestingFramework for UnitTest''''''s have any applicability here? I'm just starting to play with the TestingFramework, and want to get max value from it. -- BillSeitz You mean any applicability to Java GUI FVT? (FVT == IBMish for "functional verification testing." I dunno why we always say "verification" -- what good is a test that doesn't "verify"?) I believe so: I''''''m not familiar with TestingFramework, but it sounds like xUnit plus TestResult. Several of the Java GUI testing tools listed above just enable JUnit testcases to "reach out and touch" (and interrogate) a GUI. My group at IBM is working with Abbot http://sourceforge.net/projects/abbot and extending it to work with JavaSwt (it already does JavaAwt and JavaSwing). One then has "straight" JUnit testcases, compatible with all the usual tools (BTW Eclipse has great JUnit support), that can interact with one''''''s UIs. TFD, here I come :-) -- TomRoche * I'd like to know more about these FunctionalTest''''''s - are there public examples that interested people can read? ---- '''Name discussion: functional ''vs.'' acceptance test''' I tried "acceptance test" as an alternative name for FunctionalTest''''''s for a couple of days and really liked it. It emphasizes the role of the tests over the way they are written, is more approachable for customers, and aligns with existing vocab better. Thoughts? -- KentBeck I emphasize the role of functional tests as a medium of communication and supplement to stories in the dialog between development and customer. I once thought the name Story a bit dorky but now appreciate its simplicity. I wonder if functional tests might be replaced by the equally simple word, Facts. Someone might say, "I'll understand the story better once I see some ''facts''.", or "how can I help you check these ''facts''?" If you like this idea you may want to change the name of UnitTest''''''s also. Unit test, you will recall, supplement code the same way facts supplement stories. Since UnitTest''''''s are made before the code, I suggests we call them Claims. One might hear, "Now that that is done, what will we ''claim'' next?" or "I see that the refactoring violates a few ''claims''.". -- WardCunningham But, isn't a functional test a claim also? -- MichaelFeathers ''Could unit tests be a subset of functional tests? (see RenamingUnitTests)'' How about ''constraints?'' -- AnonymousDonor If I ask "what's a unit test?", most people will quickly say something about a programmer testing a smallish chunk of her own code. That's pretty close; the only XP wrinkle that isn't immediately obvious is that the tests come first. If I ask "what's a functional test?", I doubt I'd very often get an answer that included "end-to-end" or something about being based on user tasks or user stories. It seems like what people call functional tests are not, indeed, end-to-end. Rather they're tests run by an independent tester beating up on some localized feature from the user interface. A set of functional tests might explore the options in a given dialog box, for example. So, "functional test" probably does not convey the right impression to someone just hearing about XP. "Acceptance test" probably comes closer, because the connotation is that some direct user representative writes them. May have a greater chance of conveying end-to-endness. Maybe "user story tests"? Though that's missing the seemingly essential adjective-noun form. -- BrianMarick I believe the nomenclature has indeed been changed to AcceptanceTest''''''s. -- PhilGoodwin ''The way I see unit tests, is that they are a little testbench for a piece of a program. I often compare this with electronic devices. I once got an old tape recorder (dating from 1967) with a single '''huge '''elecric scheme. Devices like this are not built anymore. Modern devices are built up from sub-assemblies, like "pre-amplifier", "motor-control", etc. These are the coarse parts the device is made of. They are separately testable and therefore able to be replaced by something better without the need to adapt the rest (which allows you to produce a complete line of tape recorders with all different specifications, but with a lot of the same parts and also allows you to repair a device by just replacing a part by a new, better one).'' ''The same applies to software. The coarse parts can be tested separately, '''before''' they are built into the main program, if they are "Units". That is, modular. So if you want an alternative name for unit tests, I suggest "Part tests" or "module tests". However, "Unit tests" is fine by me.'' -- WillemBogaerts ---- CategoryTesting CategoryExtremeProgramming