Continued from SummaryOfHofExamples.

Regarding nested functions as "syntactically and visually sloppy" is either personal preference or due to lack of familiarity with such constructs.

''Re: "lack of familiarity". The fact that it's unfamiliar should count against it. I suggest the industry focus on making OOP syntax and constructs more flexible and/or powerful so that developers can leverage what they already know rather than have to deal with 2 different kinds of syntax/idioms. -t''

If we'd treated "lack of familiarity" as a justification for avoiding constructs throughout the history of computing, we'd have no ObjectOriented programming, no StructuredProgramming, no HighLevel languages, no AssemblyLanguage, no MachineCode, no computers, and probably no electricity.

''Typically research labs, pilot projects, or intentionally high-risk/high-reward start-ups test new ideas and tools, not mainstream production shops. That's not their job nor their forte. This topic is ''not'' about research or new-tool testing, at least that's my working assumption. -t''

Of course, but every "new idea" I listed above is a mainstream tool used by mainstream production shops.  New ideas become mainstream ideas, or at least generally-recognised or generally-recognisable ideas.  Given that HigherOrderFunction''''''s have been available in at least one programming language since the late 1950s, and are found in a lot of languages since the 1990s, and are now available in all the top mainstream general-purpose programming languages like Python, PHP, C#, Java and C++, they can hardly be called a ''new'' idea.

''I think we're getting off track here. The "lack of familiarity" was considered from a local scope: what a given typical shop can expect, at least in typical shops I have encountered over the years. From a training/hiring economic perspective of a given shop, it's my judgement that leveraging and maximizing OOP would often be the better option than introducing lambda's and HOF's. (Unfortunately, OO-limited languages may not provide enough options such that L/H's may indeed appear to be the least evil of short-term choices.) Whether typical developers "should" know X is a different scope and different debate.''

It's not a question of what developers "should" know, but what they will ''inevitably'' know -- if not now, certainly in a few years when the latest crop of tech school programming and university ComputerScience graduates hit the job market in force.  Programming practice evolves.  Not that long ago, OO was unknown in "typical shops"; now it's ubiquitous.  Before that, it was StructuredProgramming.  Of course, OO (and StructuredProgramming, to a lesser degree) was a whole new paradigm, requiring significant changes to programming practice.  For many, it meant re-learning programming.  HigherOrderFunction''''''s and LambdaExpression''''''s are just variants on already-familiar functions.  They require no paradigm shift, no significant change in programming practice, no re-learning programming.  For programmers unfamiliar with them, an hour or two of minimal effort -- at most -- makes them familiar.

''My crystal ball gives a different answer than yours. OO wasn't easy for the industry to swallow (and it's still being being evaluated and absorbed), but since it's "installed" already, the industry may just figure out it's better to improve implementations of OOP in order to leverage it for "code-block indirection" more than by throwing FP into the mix. Yes, I know you disagree about the grokking benefits of one over the other by typical developers. AnecdoteImpasse. ''

''And again, because SOME new ideas/trends stick around does not mean that all do. You prematurely are putting FP in the camp of those that stick around rather than fads that were not able to become mainstream such as CASE-tools, mass use of ExpertSystems, LogicProgramming, APL, GeneticAlgorithm''''''s, etc. Stop using that leaky argument. I keep debunking it and you keep giving it. FP already had a chance to go mainstream in the 80's.''

[Which is it? Is our judgement premature, or has functional programming already been around for a long, ''long'' time? It's gone mainstream. There isn't a single mainstream programming language with no support for first-class functions. That's happened. -DavidMcLean]

Almost every mainstream dynamic language also has support for Eval. Does that mean we should use Eval more?

[Not necessarily, but it does mean the feature is mainstream. -DavidMcLean]

Yes, the ''feature''.

[Do you have a point?]

Inclusion of a feature in languages and the usage volume, suggested or actual, can be two different issues.

[Certainly. For instance I'd hope eval features are used minimally, despite their ubiquity, given their well-known technical downsides. Nonetheless, eval is a mainstream feature in dynamic languages, just as first-class functions are a mainstream feature in programming languages both static and dynamic. -DavidMcLean]

So we both agree that feature inclusion and feature usage and/or recommendation are generally independent, or at least loosely-related issues?

[Sure, I'll agree to that point, but I don't think it's really affording much weight to your argument; feature recommendation, as an independent issue, still leans strongly toward the use of first-class functions. Eval is strongly advised against in language documentation, almost always in favour of first-class functions, for example. ( http://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval#Don.27t_use_eval.21 for instance.) Meanwhile, frameworks and libraries such as those considered on HofsAndEmbeddedDslDiscussion are widespread, and the standard libraries for Python, Ruby, JS, and C# all make significant use of the functionality of first-class functions. -DavidMcLean]

We were talking about what inclusion itself of a feature means in a language(s). You are bringing other issues and factors into this. I'm not here to debate Eval versus HOFs, only looking at the reasons for feature inclusion at this point. You brought up widespread existence of a given feature in languages as evidence of WHAT exactly? 

* And note I already agreed that HOF's may be fairly useful in SystemsSoftware, such as "base" libraries. I don't challenge HOF's in SystemsSoftware because I don't have experience coding in that domain.  The language designers probably wanted to also target SystemsSoftware and SystemsSoftware-like usage, and that's a possible reason why they included those FP features.

[You seem to want to argue that functional programming is a short-lived fad. It's been around since the '50s, and major aspects of the paradigm have been adopted by all mainstream languages. How is it a short-lived fad, exactly? -DavidMcLean]

Existence alone is a measure of mainstream-ness?

[Yes, especially when it's ubiquity. By the way, standard libraries are only SystemsSoftware when they're being implemented; all four mentioned languages' standard libraries provide higher-order function ''interfaces'', which would make themselves apparent as part of application software development. -DavidMcLean]

Well, let's see some that are applicable to custom app uses, and and look at possible OOP alternatives. We already looked at "timer" operations in JS at NodeJsAndHofDiscussionTwo.

[You've seen it plenty of times by now, I'm sure, including at CustomCollationOrder, but the sort methods in all four languages accept custom comparison functions:]

    # python
    sorted(someList, key=lambda x: x.someProperty)
    # ruby
    someList.sort_by {|x| x.someProperty}
    // javascript
    someList.sort(function(x, y) { return y.someProperty - x.someProperty; });
    // c#
    S''''''omeList.O''''''rderBy(x => x.S''''''omeProperty);
[So that's one example. I'm not sure how this'll help us determine whether functional programming is a short-lived fad or not, though. -DavidMcLean]

An OOP interface is possible, but I'm not going to entertain it here because the need for such is rare for custom biz apps for reasons given at CustomCollationOrder. I'm not going to repeat those here.

[Does it matter whether the need is "rare for custom biz apps"? There are domains where it's needed. A higher-order function is the way languages have, practically universally, chosen to address this need. -DavidMcLean]

I don't give a fuck about other domains and don't dispute HOF's may shine happy rainbow sparks in other domains. But us Eskimos don't need your damned refrigerators.

[Is there any point in discussing programming at all if you aren't willing to consider requirements others might have? -DavidMcLean]

''Top, why not tell us a requirement that '''you''' have?  If nothing else, it might help us to understand why our examples -- which certainly resonate with us (and unquestionably resonate with me as a developer of custom business applications) -- seem never to appeal to you.''

You guys seem to work in odd places, very different from my world. In CustomCollationOrder I believe one of you said something like "often we cannot use a database". Databases are usually ubiquitous just about everywhere I worked. It appears because you work at least part time in academia and/or are highly educated with Masters or PhD's, you are usually hired for projects that require squeezing the most out of the machine such that machine concerns are considered much more heavily than hiring and training and WetWare concerns. It's roughly comparable to the "old days" of computing when the hardware was so expensive that only the most educated and skilled were allowed to touch the machines because each wasted bit was costly. They don't usually hire Masters where I typically work because it's expected the projects are not challenging enough for them and/or believed they'd expect more money but not be better at the non-coding parts of the job. Instead, they'd be hired for specialized purposes or SystemsSoftware.

''That doesn't answer my question, and this isn't an issue of databases vs no-databases, though a good example of a non-database custom business application requirement is presenting a report with a custom sort order on a mobile device.  You could use SqLite, but it requires parsing and processing overhead that may be unacceptable when you're trying to stretch battery life and reduce power consumption costs -- which are not insignificant when you have a fleet of mobile devices.''

* That mobile angle is perhaps true; I haven't done much in the on-device mobile world so I couldn't really comment on that domain. When you have to use app-code to do database-ish things, one may have to rely on parts of a language that they didn't have to much before. It's like being put into a time machine and transported back to 1968.

* ''Sure.  Of course, there are many things that are not even remotely database-ish.  Finding an optimal path, for example, or drawing a map, or calculating the volume of fluid in an irregular tank given its depth, or an infinite number of other things.  If you see an example of HOFs/lambdas/whatever that does a database-ish thing, you can be pretty certain it's just an example intended to illustrate HOFs/lambdas/whatever, and its database-ishness is irrelevant.  It's definitely '''not''' an opportunity to decry HOFs/lambdas/whatever in general merely because that ''particular'' example happens to be implementable using a DBMS.''

* See below per "in general".

''Back on topic:  Could you tell us a requirement that '''you''' have, and then we could discuss the applicability (or not) of HOFs/lambdas/etc. to it, rather than have us create yet more examples that we think will appeal to you, but don't?''

[Once again I just have to ask. When the answers to all the interesting questions you post in genuine pursuit of knowledge are eventually dismissed, over and over, as relating to irrelevant fancy academic stuff, why persist when you know you will hear again that Top does not need to know what you know - anything he does not know does not need to be known because he writes systems for real people with real people in the real world. He wants you to shutup and listen. God knows to what. -- ChaunceyGardiner (But by all means, carry on. It is fascinating.)]

The given SKU scenario in CustomCollationOrder ''is'' relatively realistic, but like I mentioned a specific application in the domain would typically have one or few "collating sequences" such that we wouldn't need to worry about mass swapping in and out of the comparing algorithm. (The existing sorting API's may only offer HOF's or lambda's in their interface instead of say methods, but that's an API designer's choice, not an app coder's choice. The same issue came up with the "timer" example.) Thus a simple function call and/or CASE/IF statement would be perfectly fine for the one or few collating sequences.  And the author of that scenario has not identified a realistic scenario where we would need mass swapping of the comparing algorithm; they '''only identified one''' on that page. These HOF examples often have an "outer" layer, typically a loop, and an "inner" portion that needs mass or easy swappability in terms of swapping in and out a block and/or supplying mass multiple different different inner blocks to the ''same'' outer portion. A need for '''mass inner block swapping/reuse''' (MIBSR) is simply a '''pattern I just plain do not see''' and nobody has found and described a realistic custom biz situation/scenario where MISR is necessary (and that decent OOP couldn't handle). It's a unicorn need pattern so far. 

''All that demonstrates is that you don't need it.  That's fine.  Others have different requirements and do benefit from HigherOrderFunction''''''s and LambdaExpression''''''s.''

It's not my fault that YOU guys have not found and documented frequent scenarios with that pattern, and thus I don't see why you are trying to dump the ball back in my court. The ball is in your court identify such MIBSR scenarios '''in the wild''', not the lab. I'm looking for something along the lines of "Consider a trucking company with 30 types of trucks from 50 different venders. The vendors make that many variations of trucks because..." where the reason for mass X or mass combos of X and Y etc. is plausible from a business/org reason perspective. Generally grouping patterns will appear in a biz so that rather than having mass independent X's per mass independent Y's, there are situations that can be parameter-ized such that it ends up being mass combinations of parameters '''rather than mass combinations of behavior'''. Complexity doesn't just grow vertical, it also grows horizontal.

''We've tried coming up with scenarios that resonate with us, and that we hope will resonate with you.  They don't.  Now I'm trying something different: I'm asking you to describe a requirement of yours so that we can better understand your domain requirements, because obviously we're not meeting them.''

 // HOF-centric assumption:
 function looper(algorithm) {
   where (involvedStuff()) {
     do(algorithm);
   }
 }
 looper({algorithm A});
 looper({algorithm B});
 looper({algorithm C});
 looper({algorithm D});
 looper({algorithm E});
 looper({algorithm F});
 looper({algorithm G});
 // etc...
 //
 // REALITY as observed by Mr. Top
 function looper(a, b, c) {
   where (involvedStuff()) {
      case (a) {
        '111':
           if (b > c) {
             blerf(b);
           } else if (b > 6) {
             nib(c);
           } else {
             pleb(b, c);
           }   
        '222':
           nobb(b - c);
        '333':
           if (c > 9) {
             furf(b);
           } else if (c > 6 && mupp(b)) {
             plemp(c);
           } else {
             moof();
           }
        otherwise:
          blarp(a, c);  
      } // end-case
   }
 }
 looper(76, fiz, zemp);
 looper(fas, zemp-2, glerp);
 looper(4, pog + fump(4), 3);
 etc...
In short, the pattern of variations-on-a-theme is typically smaller than the granularity of "algorithm", and better managed as parameters. -t

''That provides further evidence of my hypothesis that your requirements don't need HigherOrderFunction''''''s/LambdaExpression''''''s.  That's entirely reasonable.  My typical requirements look a lot like your "// HOF-centric assumption", above.  I suspect others have similar requirements, otherwise there wouldn't be any such thing as HigherOrderFunction''''''s/LambdaExpression''''''s.''

That's why I keep suggesting we LetTheReaderDecide which pattern best matches their staff and/or domain. We don't have to provide one-size-fits-all recommendations ourselves.

''No one claimed that HigherOrderFunction''''''s/LambdaExpression''''''s are one-size-fits-all.  You're the one who suggests, effectively, that HigherOrderFunction''''''s/LambdaExpression''''''s are one-size-fits-none.''

No I didn't. My only "general" suggestion is to avoid them in shop code '''IF''' the shop doesn't need them often enough, in order to avoid confusing staff. If your domain/shop needs lots of HOF's, then it essentially becomes a REQUIRED job function and staff is typically more closely filtered for such knowledge. Similarly, somebody who is hired to do hardware/RAM-centric development will probably not be expected to know or asked about SQL, but a custom biz developer will be expected to know at least mid-level SQL (if the shop doesn't partition out DBA's or DB specialists for such). If you are in a shop full of hardware-centric developers, then introducing SQL into code should probably be avoided if reasonably possible because it may stump a future maintainer from that shop. I wish Java would have focused on improving their OOP rather than create API's (such as GUI's) that expect or nearly "force" lambda's/HOFs because many shops don't have staff prepared for such.

Parsimony shouldn't be the primary factor in determining whether to introduce relatively rarely-used idioms into the shop code base.

''How should Java "have focused on improving their OOP" in a manner that would obviate HigherOrderFunction''''''s/LambdaExpression''''''s?  Please be specific.  I suggest modifying one of the three Java examples toward the bottom of CustomCollationOrder (immediately under "If we're assuming a conventional OO language of the C#/Java variety, it could look like this") to create your illustration.''

This seems to be a contradiction. Paraphrased interpretation: "How would you improve the OOP if you are not allowed to improve the OOP because you are assuming C#/Java-style?" I'd probably have to deviate too much from their existing OO approach that it would generate questions similar to those already generated by the pseudo-code examples.

''You claimed that Java should "have focused on improving their OOP" in order to obviate HigherOrderFunction''''''s/LambdaExpression''''''s.  If you can't defend that claim, how do we know it isn't spurious?  Perhaps there's no way that Java '''could''' have improved their OOP that would have obviated HigherOrderFunction''''''s/LambdaExpression''''''s.''

I am not sure Java/C# can be sufficiently "fixed" without breaking backward compatibility. That would take a detailed analysis of the syntax and rules. I don't use Java often enough to claim to be anywhere near ready to fill that role. Let me restate the original suggestion: "I wish Java would have instead focused on improving their OOP in order to reduce the need for HOFs/Lambda's, or at least publicly documented the reasons they cannot change the language and the cost/benefits of the trade-offs they explored."

''Assume backward compatibility is irrelevant.  How could Java have improved their OOP in order to reduce the need for HOFs/Lambdas?''

For one, the prototype OO model or something similar so far is potentially more flexible than the class/object-separation model. (Or at least I haven't seen a way to extend the separation model to make it more flexible. It may exist out there in Possibility Land; I don't know everything. I'll ponder it, though.)

''I agree that prototype-based OO offers some advantages over class-based OO, but I'm not sure how it makes any difference compared to HigherOrderFunction''''''s/LambdaExpression''''''s.  Your code examples, so far, raise questions which you've declined to answer.''

On the flip side, can you point to an '''inherent''' feature of OOP that outright prevents OOP from doing things HOF's can (typically in the sense of defining the meat outside of the proverbial sandwich), and/or results in significantly more code? If not, then we'd probably just end up quibbling over language- or API-specific design and syntax, which is likely boring and pointless. "See, doing X takes more code in Java" doesn't necessarily extrapolate to ALL of OOP. -t

''I imagine a specific language could be made that would support inheritance, polymorphism and encapsulation whilst not having HigherOrderFunction''''''s and LambdaExpression''''''s, yet would provide exactly the same functionality as HigherOrderFunction''''''s and LambdaExpression''''''s but with the same number of, or fewer, tokens.  However, I don't know what it would look like.  If you don't know what it would look like either, then it's mere speculation.  Until it's more than mere speculation, we'll have to look at what real languages do.''

But we both seem to agree that in the common existing languages, HOF's result in noticeably less code for the scenarios looked at. Why debate something we don't disagree on? 

We both seem to agree that:

* 1. In common existing languages, HOF's result in noticeably less code (for scenarios studied)
* 2. It's quite possible that some (hypothetical) OOP language could be competitive with HOF's code-volume-wise. (Or an unidentified obscure existing language.)

So what's there to debate? Yes, it would be nice to analyze or define in more detail what such a language would look like rather than live with sketchy pseudo-code. I don't dispute that would be nice. I just don't feel like doing it at the moment. Maybe someday later, or if somebody else wants to fiddle with the idea...

''Perhaps it's time to shelve this debate, at least for now.''