Abstraction Versus Marketplace
-------
Lisp (and related dialects) is perhaps the most '''flexible''' language known. You can make it be just about any paradigm or abstraction you can envision. Or at least its flexibility-to-syntactical-complexity ratio is near the top (power without clutter). It's a marvel to behold; a "language design formula" that is perhaps the E=mc^2 of programming. Lisp has been around for about 50 years, and enjoyed a renascence of popularity in the 1980's during the AiBubble, which spilled over to non-AI projects. 

However, despite plenty of opportunities in 50 years, it '''failed to catch on for rank and file development'''. Its ideas keep getting introduced to various languages over the years, but often fail to "stick" for common usage, partly because such languages or add-ons don't do it as well as Lisp (meta + complex existing syntax = mess), and partly because it can encourage MentalMasturbation, job-security "creativity" in programming, and increases the caliber of skill needed to read code using such techniques, which '''increases employee and recruiting costs''' beyond what companies want to bear.

One can argue that Lisp-like techniques are the way it "should" be done, but the industry keeps saying "no" for good or bad. The industry wants tools that cater to semi-transient PlugCompatibleInterchangeableEngineers. This often results in tools that do some amount of '''"herding" to provide consistency''' with pre-packaged semi-domain-specific idioms, and less catering to "abstraction gurus". It's not quite "lowest common denominator", but something closer to the 25 percentile, meaning that roughly 75% of those hired need to be able to grok the code in a timely manner. (The economics of the caliber of hiring is debated later).

I've seen organizations complain even about "excessive function usage" with old-fashioned functions because some developers in the past were confused by the level of factoring used and they didn't want to fire them because they otherwise did decent work under lower abstraction, and often had better people skills than the "high brow" developers. (Perhaps they were partying in college while the gurus were writing compilers.) I've seen this phenomenon in multiple organizations and have at times been '''ordered to "dumb-down" my code'''. I even limit my favorite abstraction, TableOrientedProgramming, because other maintainers are not used to it.

Duplication rarely gets somebody outright stuck; it's just more busy work and perhaps more little bugs caused by missing one of the repeating segments accidentally. However, heavy abstraction can get developers outright stuck. '''Businesses often value consistency and predictability more than average higher productivity''', and hum-drum code gives them that. Being able to make accurate future predictions in terms of scheduling and resources is often highly valued in business and management. Owners and managers have repeatedly decided that this is often more important than higher average long-term productivity. (Start-ups and R&D-style projects may be an exception because "rock-star" developers can rocket you past the competition in the shorter term, getting your company or product on the map.)

* ''"[H]eavy abstraction can get developers outright stuck"???  Really?  Misuse of '''anything''' can slow developers down.  There's nothing quite like a tangled web if IF statements to ruin an afternoon.  Bad programming is bad, regardless of the level of abstraction, though it's low level code -- "three-star" C code and the like -- that's particularly notorious for being obtuse and brittle.  HOFs in, say, Javascript are notably easy to read.''

** I mostly meant reading ''other'' people's code, not a programmer tangling themselves with HOF's. And everybody knows IF's, so such is easier to detect and notice. Misuse of a wrench is more obvious because everybody has experience using a wrench. However, misuse of a Sonic Screwdriver may go undetected because so few have experience with it.

** ''Every programmer knows IFs once he or she has learned them.  Before that -- like every language feature or concept before it is fully understood -- they are mysterious.  Beginners can misuse any feature -- I'm sure you've made mistakes, as have I, even with IF statements -- but once learned, mistakes are relatively unlikely.  Perhaps if you find HOFs "tangling", it's because you haven't practised them enough to fully understand them.''

** An org usually keeps a closer eye on new programmers' code. The higher you go up the abstraction ladder, fewer "mentors" are available. And I'm looking at it from an org's perspective, not mine right now. Remember, they complained about MY function factoring.

** [New programmers ''should'' have learnt about higher-order functions. They're not a mystical hyperadvanced feature that requires years of meditation to comprehend; they can be learnt in a couple of hours, assuming a programmer has already learnt about functions in general and perhaps loops (both of which new programmers learn quite early on). -DavidMcLean]

** I have pages of "should have's" for work-world behavior and human endeavors. Your personal should-have idealism doesn't scale. Don't forget, HumansSuck. (If you work in academia, you may be biased toward a "the solution to everything is more education" approach because it would increase the demand for your services, making you wealthier. See AreWeBiasedTowardLaborIntensive.)

*** ''I've never met an academic for whom "wealthier" was the slightest motivation.  If wealth is at all a goal, academia and education are last places you want to be.  DavidMcLean is right -- new programmers learn about higher-order functions.  They're not mystically complex and understood only by a rare few, they're currently part of the average programmer's education.''

*** I have seen such goals. But, perhaps I should have also included prestige or feeling needed, etc. It's human nature to want to feel "needed". As far as HOF knowledge, other than fallowing a code pattern in the doc examples, I just don't see it widespread in the field. We'll probably have to AgreeToDisagree on that point.

** ''I'd hardly call learning about HOFs an "education".  My experience with programmers being shown HOFs is that the majority find passing functions as values to be logical and intuitive, especially as they're used to passing around objects and primitives as values.  Understanding HOFs typically requires considerably less effort than learning the quirks of the latest version of MS Office.  Resistance -- to the extent that I've seen any, which is negligible -- is invariably from older programmers imbued with the essence of languages where functions are strictly static, structural, and second-class.''

** FP is kind of in "fad mode" right now, partly because of being stuck with the HtmlStack client "standard", and partly due to exploration of parallel programming. If the typical "problems with abstraction" play out, the future may be different and settle back to the usual pattern: stiffer DomainSpecificLanguages to herd programmers. The skepticism of "older programmers" perhaps has some worth in terms of forcing fad-chasers to justify "why". Newbie programmers are not very good at "why", other than $$$. Addressing "why" is often a good exercise for everybody involved. See also NaturalEventSyntax. -t

* ''Organizations that complain about "excessive function usage" are clearly dysfunctional.  In terms of language design, why cater to dysfunctional companies at the expensive of functional ones?  Dysfunctional companies are uncompetitive, and guaranteed to eventually fail when other companies -- equal in every way, perhaps, but with a more capable IT department -- will successfully compete with them.  Isn't it better (again, in terms of language design) to help the strong and capable companies get stronger, rather than let the weak ones stay weak?''

* Continued at AbstractionAndOrganizations


In short, '''Lisp Lost''', and this includes Lisp-like techniques. You can bitch about the industry being "stupid" or "ill informed", but the reality is what it is and that's how it acts to the high abstraction attempts.

''I'd hardly describe being #13 on the TiobeIndex (as of February 2013) as "lost", especially compared to what LISP beats.''

It's not a good index of "production" language usage, as described in that topic. I don't question that it's personally popular. It's also a great teaching language. I've seen no evidence of common usage for custom business applications (unless maybe AutoCad is involved).  Look how many WikiWiki topics there are on it: CategoryLisp; it's a grand nerd conversation piece, cranking up its "mention" score. It's kind of like Lamborghini's: talked about far more often than purchased. '''TiobeIndex would be like estimating the number of Lamborghini's ''on the road'' by counting web pages that mention them.'''

I'm pretty confident in estimating that less than one percent of commercial app code is in Lisp, and would guestimate the actual number is something like 1 in a 1000 lines.

''What source of statistics -- outside of your inevitably-limited (and that's nothing personal -- all our personal experiences are limited) experience -- do you have for "common usage for custom business applications"?''

I don't have anything approaching an OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy, but neither do you. It's anecdote against anecdote.

''Be that as it may, the fact that we're debating LISP's significance over fifty years after its inception, '''and''' the fact that it ranks 13th on the TiobeIndex, '''and''' the fact that modern implementations like Clojure are growing in popularity, '''and''' the fact that its used as a scripting language in a variety of products, '''and''' the fact that some of its original concepts -- dynamic typing, higher-order functions, homoiconicity, functional programming -- have such an impact even on current products, means LISP lost nothing.  It won.  It didn't win by being a popular programming language like C# or Javascript.  Those will eventually diminish in popularity.  It won by being profoundly influential on '''every''' modern programming language, including C# and Javascript, and LISP will certainly influence the languages that follow them.''

''In short, if there was a GreatLispWar -- and I'm not sure there was -- LISP won.  Not by being a popular language, mind you, but by being influential on '''every''' popular language.''

I wont deny it's had some influence on a lot of languages. However, that's not the same as taking over production languages and/or production code design (language features actually used, not necessarily what's included in the language). I've already described the problems with your reference to TiobeIndex above. Lisp fans have been saying the mainstream breakthrough is '''"just around the corner"''' for several decades. I heard the same song and dance in the 1980's. It's like how religious groups keep predicting the immediate end of Earth, but we are still here. Semi-DomainSpecificLanguage''''''s are still in charge and still pre-package common abstractions and idioms of the domain/focus to make them '''more digestible to fungible staff'''. Whether that's good or bad, it's what the industry likes and keeps backing despite 50 years of Lisp's mega-meta capabilities.

''There are two simple reasons why LISP won't replace C# and Javascript: the perception that a festival of parentheses hurt readability, and the perception that prefix notation hurts readability.  The concepts it demonstrates, however, will endure far longer than the languages that adopt them.''

I do agree Lisp has what could be described "syntax issues", as partly described in MaspBrainstorming. But that's not the entire problem. Over-meta-tizing mainstream languages or trying to introduce such a language has repeatedly failed to catch on and I see no reason why that would change. The forces causing it are still in place. JS is in a unique position in that it's kind of forced on the industry due to historical issues akin to QWERTY keyboards.

''Actually, the "syntax issues" are the entire problem.  The "over-meta-tizing" you deprecate is being adopted by other languages.  This will, no doubt, continue.''

Until it becomes an expensive-specialist and convoluted product/stack, and then something like Html or VB comes along that's easy to for average developers to pick up quickly, and steal the market share from the it, and the cycle continues. The basic AlgolFamily idioms continue to dominate in terms of actual code written even if a particular language has ''additional'' meta capabilities. (Somewhere on this wiki is a lost topic about how languages "past their prime" add on features to try to stay competitive in the feature brochure check-mark race, but that's not the same as actual feature usage.) {Text added after original dialog.} -t

''HTML is a text markup language, closer to the old-skool wordprocessor Wordstar than a programming language, so it doesn't count.  You can't write programs in it.  It's true that "easy entry" languages do pick up a market share from professional languages, especially when a field is developing, but that's because beginners need an easy starting point and new fields attract a lot of beginners.  Such languages inevitably gain features -- i.e., more "meta-tizing" -- as their users grow more proficient.''

I don't believe they become more meta-tized. If anything they become less meta-tized because the common idioms are included in a declarative way (like common GUI idioms SHOULD be in HTML or its offshoot). Making common activities be paint-by-numbers (like "Wizard" dialogs) is how they get people to pay for upgrades to the next version.

''C++ did not provide templates at first. Generics were added to Java. C# recently gained lambdas. "Meta-tizing" gets added because it adds power.''

Often its a feature-bullet-point pissing match, perhaps unnecessary. MS wants to make C# more of a SystemsSoftware language to compete with C/C++, not just an app language, and that's partly why they added it. Languages do tend to pick up clutter over time, and I'm not sure it's a good thing. C++ is known for its steep learning curve and piles of gotcha's. It has a few too many +'s.

* [Actually, one of the main reasons lambda expressions were added to C♯ is for use in LINQ, which is significantly more useful in application software than systems software. -DavidMcLean]
* Which is possibly a plot to sell more MS-Sql-Server licenses. Non-MS database products seem to be harder to hook to LINQ and/or have more bugs.
* [It's perhaps true that other DBMSes are harder to apply with LINQ (although it looks like it's pretty easy using something like NHibernate), but how is that relevant to the issue of lambdas and systems software? -DavidMcLean]
* There's a lot of different linquistical ways to stick verbs on things.
* [True, but the only way to provide verbs as general and useful as LINQ's set is by allowing them to accept functions as arguments.]
* Depends how you define "useful". And again, the industry does not want stuff too "general".
* [The very fact that LINQ exists would suggest that a substantial section of the industry does want stuff to be general. -DavidMcLean]
* FP is kind of up in fad ranking right now, partly fueled by interest in parallel programming. Hopping on the bandwagon does happen. Besides most application developers ''use'' the LINQ libraries but do not write them themselves. Usage of a feature/API that is built on top of meta abilities is not quite the same as developers writing new code with meta capabilities. (Material added since original dialog.)
* [It's true that functional programming is gaining mainstream representation, mostly due to Microsoft also releasing F♯. LINQ isn't (outwardly, at least) an example of functional programming, though; it just uses anonymous functions. -DavidMcLean]
* LINQ-like technology may be better with strings/eval because then we can send the strings to other DB vendors without waiting for MS to write or fix their drivers. Strings are more sharable cross-platform/language than HOF's.
* ''There is already a technology for using strings/eval -- ODBC.  LINQ with strings/eval would engender all the problems of strings/eval with no benefits.  A major reason for using LINQ is to dispense with strings/eval.''
* So that MS can fuck non-MS vendors.
** [Microsoft only writing LINQ drivers for their own database platforms is hardly a means to screw over other vendors. Think about it from their perspective. Is it reasonable to expect them to write drivers for every single third-party database out there? Hardly. They'll write drivers for their own platforms, and publish the LINQ driver specification such that third parties can write LINQ drivers for ''their'' own platforms. (This is what they actually did. There's actually very in-depth tutorials on how to write your own LINQ queryables up on MSDN.) -DavidMcLean]
** And just when other vendors perfect their drivers, MS will change standards and produce Mink or Bink or Stink as the "new collections standard".
** [Why would they do that? If MS really made LINQ to spite other vendors, they wouldn't have documented it so well; they wouldn't have tutorials on MSDN for connecting one's own stuff with LINQ. -DavidMcLean]
** They have to give the semi-appearance of being open, otherwise users of other brand DB's will shun MS tools.
** [Semi-appearance of being open? .NET, including LINQ, ''is'' open. All Microsoft haven't done is implement LINQ for every single database platform in the world, and who can blame them when databases vary so much? -DavidMcLean]
** And when the other vendors finish, MS will probably change the "standards" again.
** ''That would be commercially unwise.  It is far more likely that Microsoft -- or any other major vendor -- would introduce new technologies whilst retaining existing ones.  Note that ODBC still exists and is well supported, despite the existence of LINQ.''
** I've found ODBC more difficult to configure in 64-bit Windows.
** ''How so?''
** I don't remember the details at the moment. It had something to do with backward compatibility.

* ''No, because strings/eval are fraught with problems.  Ever heard of SQL injection?  And that's just '''one''' of the problems caused by using strings/eval with SQL DBMSes.''
* There are other topics on that. HOF's won't solve injection.
* [Higher-order functions alone won't solve SQL injection, because there's absolutely zero relation between the two. LINQ, however, ''does'' solve SQL injection. -DavidMcLean]
* Not. If you pre-parse, you can pre-parse strings also. If you are just passing the expressions on to other vendor DB's, then it's what we had before. Parsing would have to match multiple vendors' SQL syntax.  There's no free lunch. We are getting off topic here.
* [LINQ-to-SQL handles parameters using prepared statements, which you've already been told prevent SQL injection by ensuring that values and commands are incapable of mixing. You can't do the same with strings, because if you use strings alone you won't be using prepared statements. -DavidMcLean]
* Prepared statements have their security flaws. Anyhow, we are revisiting old battles, such as TopOnPreparedStatements, and I see no logical reason to duplicate those here. This topic is NOT about SQL injection.
* [I moved the rest of this discussion over to TopOnPreparedStatements as requested. -DavidMcLean]
* Thank you. -t
* ''Top, you '''could''' have moved it yourself, rather than getting angry and swearing.''
* I tried that recently with something else, but somebody kept putting it back. (I'm not saying it was an intentional EditWar, only that doing such an action failed to work.)
* [It was me doing that with the other thing, 'cause I didn't immediately realise it'd been moved elsewhere. Sorry about that. (That's why I left a pointer on this page to "where all the other discussion went".) -DavidMcLean]
* Not a problem. We'll learn from our experiences and move on.

''For those of us who use "clutter" and find it of value, we don't mind if you don't use it. We object if you try to take it away. What's clutter to you is a toolbox for us.''

See the "assault weapons" analogy below.
------

Note that AutoCad added VBA scripting support, in addition to Lisp. If Lisp is so palatable, why would they feel compelled to do that?

''There's frequently a perception that a festival of parentheses hurt readability, and the perception that prefix notation hurts readability -- especially among naive or substandard developers.  Unfortunately, naive and substandard developers sometimes control the purse strings.  (Note that the VBA support is now deprecated.  There's a shift to .NET support, but that obviously is due to forces that have nothing to do with LISP vs VBA.)''

It is true that "average" developers wag the industry's dog, but that's just the way it is. You need to see it from the economics and staffing perspective of org decision makers. Being frustrated that no support is given to higher-brow developers isn't productive. (I addressed the parenthesis issue above already.)

''Us "high brow" developers aren't frustrated at all.  We typically make many times the salaries of mediocre developers and retire wealthy.  Nothing stops us from using the tools we prefer, and we succeed because we excel at using them.''

Regardless, Mitt, there appear to not be enough of you to change the industry patterns. I've learned to avoid excess meta-tizing and indirection because of complaints about confusing follow-on developers.

''I don't want to change industry patterns.  It's '''you''' who seems to want languages that cater '''only''' to the lowest common denominator, developer-wise.  The rest of us are content to allow you to use as few features as you like, as long as that doesn't impinge on our ability to use features that are as powerful as we like.  Your concern for follow-on developers is laudable, but why do you assume they'll be incompetent?''

They are not "incompetent", just often not skilled in meta techniques. You seem to be trying to paint a grey issue as black-or-white. There are plenty of developers who are quite productive under "regular" levels of abstraction. Sometimes they type more than necessary because of it, but that's their prerogative. Their code may be fairly tedious to maintain because of the duplication, but at least it's strait-forward: no drama, no weirdness, no MentalMasturbation experiments.

''Fine. Just don't advocate taking away our tools to cater to them.''

That kind of reminds me of the assault weapon debate. One side argues that "assault weapons" are too dangerous to allow regular citizens to own, while assault weapon proponents argue they are useful more often than they are misused or cause problems. Lisp is the ultimate assault weapon of languages: compact, accurate, reconfigurable, few parts, and rapid-volume fire.

''That's a bizarre analogy. Does it have a point?''

Lisp kills :-)

''FacePalm''

It's a balance act between power and misuse or poor training.

''LISP is a "balance act between power and misuse or poor training"?  Huh?  I'm still not clear on your point.''

The industry choices, not Lisp itself.
----------------
'''Part Time "Jack" Programmers'''

A general trend in the industry is away from centralized specialists and toward "departmental generalists". In other words, instead of dividing up by skill, organizations are deciding that dividing up by organization structure (departmental) may be better, and to have a '''multi-hatter generalist IT person'''. An in-between division is "hardware" versus "software" where the hardware person does general trouble-shooting of hardware, OS, and basic installations; while the "software" person focuses on learning the department's purchased applications as a power-user-level trouble-shooter, a DBA, and a custom software developer.

This means that the person doing the programming will likely only be able to devote a fraction of his/her time to programming and the skill of programming.

It's not just about programmers being "dumb", it's that they are asked to be a '''jack of all IT trades''', which makes them master of none.

''Perfectly reasonable.  Any skilled programmer who can't fix a PC, change printer toner and other gubbins, sort out network issues, make coffee, work with users, design a system, install a package, create a dynamic Web site, clean a keyboard, construct a database, and learn -- and quickly be productive in -- new languages, tools, and paradigms, should quit and take up knitting.  I don't know about you, but to me that's '''exactly''' what being a "skilled programmer" means.  As developers, we may be able to distinguish programming from printer repair, but the average user can't and is just looking for someone who is "good with computers."  So that's exactly what we should be.  The days are ending when all office work would stop to ring Tech Support and hope that the third or fourth call would finally get exactly the right kind of techie -- who, no matter how good he is, didn't have a clue who you were or what your office did -- to solve the (mysterious) computer problem at hand.''

Your suggestion that they master everything IT is unrealistic. Maybe a select few can, the Mozarts with photographic memories and speed-reading capabilities, but it takes experience for most to do things well. Without experience one wastes time getting stuck on gotcha's and poorly-documented corners. It's more economical to have in-house or rented specialists for the times when the generalist gets stuck. Your false Mozart-or-knitting dichotomy is silly.

''No, it's not.  It's the skill profile of those who succeed in IT and keep their users happy.  And it's not as uncommon as you think; indeed, it's a typical "computer geek" profile.''

Somebody who is an interface designer whiz is not going to want to clean printers and crawl under dusty desks to string cable for a living.

''Anyone who thinks they're too good to clean printers or string cable shouldn't be in this business, and that goes for everyone from junior developers to the company CTO.  SpecializationIsForInsects, and elitism is disgusting.''

{...and all us non-elitists are way better than the elitists, anyway.}

Hey, I'm just the messenger. I'm not smart enough to fix society and humanity, so I focus on tools instead.

[And your message is… what, exactly? That interface designers think they're too good to string cable? -DavidMcLean]

From an economic standpoint, it doesn't make sense to pay a skilled specialist to do a semi-commodity activity that could be done by somebody who earns much less per hour. An extreme case is paying a $300/hr brain surgeon to clean and scrub his tools after surgeries. Cleaning would then cost the hospital $300/hr when it could cost them only about $20/hr. They would have to pass that extra cost on to consumers or patients. Whether a given person falls into that category has to be evaluated on a case by case basis. The human ego may mentally put oneself in that category, but outsiders may see it otherwise.

''Medicine has not yet demanded, to use your phrase, "departmental generalists".  However, every brain surgeon knows how to clean and scrub his tools after surgeries.  It's an explicit part of his or her expertise, whether he or she does it regularly or not.''

Knowing how and actually performing something regularly are pretty much two different issues.

''Not in this case, or the IT case.''

Let me clarify: should the surgeon know how to clean surgical tools? Yes. Should an established surgeon ''regularly'' clean his/her surgical tools? No. One could argue that he/she should still do it on occasion to keep in practice.

''We're talking about IT, not surgery.  Specialisation in IT is baffling to non-IT people, who find it mysterious that the person who will write a program won't fix a printer and vice versa.  From a non-IT point of view, it's all computers, isn't it?''

There's no specialties within "computers"? I find that a rather curious claim. Anyhow, if it takes the programmer 3 times longer than the experienced shop "hardware guy" to fix the printer, then it may not be '''economical''' to the org for the programmer to fiddle with it. It's probably good to keep your skills fresh by trying variety, but that can be taken too far.

[There indeed are no specialities within the domain of "computers", ''from a non-IT point of view''. In general, non-IT people don't recognise a distinction between "hardware guys", "software guys", etc., generally collapsing them under umbrella terms like a simple "computer guy" moniker. It's obvious to us that there's a difference between skill with hardware and skill with software, but it's not obvious to the rest of the world. -DavidMcLean]

--------------

'''Why So Long?'''

HOF's have been around for about 50 years (longer if include the fact that in machine language you can "point to" a subroutine.) So why would they be mostly excluded from mainstream "production" languages for 50 years and suddenly mainstream programmers are allegedly ready for them '''now and only now'''? Did the Internet change something specific? What? What Great Switch suddenly flipped? And why didn't the Lisp fad spike of the 80's trigger it?

''Much of the industry was diverted into achieving language productivity through IDE evolution, rather than language evolution, until Javascript and Ruby -- both heavily influenced by LISP -- demonstrated that higher-order capabilities were worthwhile outside of functional languages.''

''By the way, the fact that machine language can "point to" a subroutine is not the same as a higher-order function.  A higher-order function captures its calling scope.  A subroutine generally does not.''

I still believe their semi-prominence is a temporary blip until IDE-centric or a GUI markup language standard pre-packages GUI idioms. It's a blip comparable to C++ for Windows GUI's heyday until VB, Delphi, MS-Access, etc. came along. The VB desktop approach to low-budget in-house apps was very popular and appeared to be solid until the Web's deployment ease bit into it. Shops didn't go web because web was easier to develop and maintain with existing or fungible staff, but because it reduced deployment costs and DllHell, which really ate up help-desk time. But, we'll see. (It's not that I love VB, but shops in generally sure did, minus deployment issues.) What the industry really wants is something like VB without the deployment issues. '''Shops are spending waaaay to much time on UI fiddling'''. Something has to give.  -t

''Of course, VB and Delphi are still there for the shops that love them.  If developers are frustrated with Web UI fiddling, they appear to be missing an opportunity to fill a market void with a profitable product.  But then, why aren't Web application builders catching on?  Google "web application builder" to see acres of software that you've probably never seen before.  Maybe it's because developers are realising that drag-n-drop interface builders with limited languages aren't as productive as powerful languages on their own.''

I will agree with that last sentence generally. However, it doesn't matter enough for a combo of staffing and behavioral reasons already discussed. The industry repeatedly votes for pre-packaged idioms (be it IDE's or markup or DSL's). "Meta-friendly" languages often lead the bleeding edge, but pre-packaged idioms usually follow closely behind when the patterns become less murky. The web side has just been later than normal to finish the cycle. I suspect it's partly due to a misguided obsession with MVC. I have plenty of gripes about certain drag-and-drop IDE's and limited meta ability also, I would note. (Deployment of desktop apps is still a problem.)

''Is the lateness of the Web side to finish the cycle really because it's later than normal, or because there's a permanent shift away from pre-packaged idioms to "meta-friendly" languages?  Are meta-friendly facilities, in fact, the new pre-packaged idioms?  Time will tell.  As for MVC, I'm surprised you're concerned with the apparent obsession thereof.  After all, aren't your favourite technologies the ultimate expression of MVC?  I.e., model = database, view = UI, controller = code?''

Well, that's one possibility. However, many programmers I know gripe about the HtmlStack. '''I see no love''', other than from highly-paid Web UI specialists for trendy public front-ends. Many turned to Flash to get better control and consistency, but Apple, MS, & Google are trying to kill it now. As far as MVC, it perhaps has multiple interpretations.

''Indeed, I see no particular love for the raw HtmlStack either. Love starts with languages and tools that wrap it, like Javascript and jQuery.''

Most of the features/idioms ''could be'' packaged into something easier to digest. VB's timer case in point: drag the timer object with the stop-watch icon onto the form, right-click to set properties (cycle duration), double click to open the event code editor, and type in the code that repeats. (The right-click and double-click are standard IDE behaviors in VB. There's other shortcuts also not mentioned here.) No "naked" HOF's or other implementation guts sticking out and no need to dig in manuals for setup and parameter syntax etc. Even a drunk programmer can do it. The markup version shown in BradyBunchDiscussion(?) is almost as easy.  -t

''That's fine when the "packaged" materials precisely suit requirements.  Unfortunately, they rarely do, and that's why we have higher-order functions and other mechanisms to help define what isn't pre-packaged.  Drag'n'drop forms and click-able property lists encourage human interaction, which is an inefficient way to implement customisation and (especially) automate repetitive tasks.  Language encourages automation; drag'n'drop interfaces do not.''

HOF's don't provide any new functionality that cannot be done some other way (unless the root or base system is built around them.) A code "design" improvement is perhaps debatable, but do versus not-able-to-do is another thing. At the very least, we have TuringEquivalency. Generally the "idiom machine" will handle roughly 80% of our needs, and the rest needs to be custom coded. Ideally primitives or lower-level building blocks are available such that one can diddle individual bits or pixels if need be for special cases. You have not presented a CBA case where they offer a significant improvement (outside of client and SystemsSoftware-specific idiosyncrasies).

''That's true -- HOFs don't provide functionality that cannot be done some other way.  But, by the same token, 'for' loops don't provide functionality that cannot be done with 'while' loops, and we don't really need 'while' loops when we've got gotos. Functions & procedures aren't necessary -- they don't provide functionality that cannot be done with masses of in-line code.  All you ''really'' need are gotos, 'if' statements -- no 'else' required -- I/O, variables and simple expressions.  We don't even need multiply and divide; we can construct them from repeated addition or subtraction.  Sounds like 6502 assembly language, doesn't it?''

''Obviously, 6502 assembly is no longer a reasonable tool for developing business applications.  For the same reason, "primitives or lower-level building blocks [that] are available such that one can diddle individual bits or pixels if need be for special cases" aren't reasonable either.  Lower-level facilities are more complex, more awkward, and generally riskier in terms of reliability than higher-level abstractions.  High-level abstractions are preferred, precisely because they make it easier to achieve reliable functionality than lower-level constructs.''

''Even if HOFs only offered a significant improvement in client-specific and SystemsSoftware-specific cases, that alone would be a reasonable justification for HOFs. Client-specific and SystemsSoftware-specific cases '''do''' appear when developing business software, and it's better to have facilities to easily and reliably handle them than not.''

Again again, the "advantage" is like saying SQL works better on SQL databases than Rel does, and Rel works better on Rel databases than SQL does. It's a UselessTruth. In a parallel universe, the designers of JS or its equivalent could have just as well made the native timer API be Eval-based or object-based INSTEAD OF hof-based. If it was truly powerful in a general way, you should be able to '''find a more universal example'''. The fact you can't is quite telling. You are grasping at a straw with that example, trying to milk it for every speck, which is about all it delivers as far as evidence.

''The universal example is HofPattern.''

* You haven't shown it universal in a practical sense. It appears to be mostly a lab-toy.

* ''It's not a lab toy, because it's not a program or even a pseudo-program.  It's a pattern.''

* with little known demonstrated utility.

* ''...Except for almost every algorithm that requires customisation, as described below. Like A* and its heuristic function, quicksort and its comparison function, neural networks and their output functions, genetic algorithms and their fitness functions, Djikstra's algorithm and its costing function.  And so on.''

* Well, I rarely see those in my niche other than in packaged software, as we've already been over in [I forgot topic].

* [Have we been over the possibility that, due to your unfamiliarity with techniques like HofPattern for injecting customisation into algorithms, you generally mightn't ''recognise'' when you write an algorithm that fits this pattern? -DavidMcLean]

* I've asked for example scenarios from the domain before, and nobody has really produced any realistic ones, outside of math-heavy traveling-salesman-optimization-like examples that I've almost never seen in the field accept perhaps as packaged statistical software.

* [There's still an employee scheduling application in the works to serve as example (as mentioned on SwitchCaseListVersusHof and NodeJsAndHofDiscussion). Generally, however, ''all'' non-trivial applications employ algorithms of some form or another. The ability to inject customisation into an algorithm is obviously valuable in terms of enabling reuse; as noted above, it's possible that the reusability potential of the algorithms you implement is unclear since you're unfamiliar with certain ways of customising algorithms. (Even if there's only one "primary" use for some algorithm, customisation is valuable to enable things like UnitTesting.) Of course, none of that is a ''specific'' example; we're not intimately familiar with your codebases, so identifying particular could-be-reusable algorithms from such is somewhat complicated. -DavidMcLean]

* But I found and linked 13 pre-packaged employee scheduling applications, and as I remember it, you agreed your experience on that is rather dated in that you did employee scheduling programming before most of those existed, and the example is now moot in terms of ''custom'' business software. The pattern you seem to keep leaning toward in presentations is ''not'' one I see in the field often. And your ranking method of triviality of application was also something I questioned. But it appears we have very different experience and probably will never agree on what is likely to be encountered in the field. Let each side state their observations, and then LetTheReaderDecide rather than bicker over it yet some more. Readers hate that.

* ''You remember it differently than I do.  I recall you finding some list of scheduling applications, but that's completely irrelevant.  The point wasn't whether or not scheduling is "moot in terms of ''custom'' business software", but whether or not it's an effective illustration of the value of HOFs in business applications.  Using a HOF to inject customisation into scheduling, after all, is not conceptually different from injecting a HOF into costing, route planning, "big data" analysis, social network sentiment evaluation, or any of a thousand other custom business application algorithms.  As mentioned before, it's a pattern you probably don't see in the field very often because the field you work in, by every description you've given of it, is data entry and reporting.  That doesn't benefit, at least not very often, from injecting HOFs into customisable algorithms.  By the way, my workplace's IT department currently has a project in progress to develop a custom scheduling/timetabling application.  Apparently none of the off-the-shelf ones were suitable, and indeed the off-the-shelf one we're currently using is dire.''

* 13 products (at least) and no matches suggests your company is an outlier. But like I said elsewhere, OOP is good enough in most cases for such. It's slightly more code, but also more familiar, AND more flexible because one can later attach attributes to it, unlike HOF's (without changing all the function calls). You are over-selling HOF's over OOP. (Damn, we are inadvertently reinventing an existing debate. We should probably move these paragraphs.) -t

* [Indeed, we've been over much of this before. To summarise: The use of objects to inject executable customisations into an algorithm, i.e., DependencyInjection, is semantically equivalent to the use of first-class functions to inject executable customisations into an algorithm, but syntactically more verbose. The syntactic overhead makes many common higher-order function patterns a lot less convenient, and the semantic equivalence means that passing the customisation in an object ''isn't'' going to be any more familiar. The developer must equally understand the concept, utility, and implications of passing blocks of code through the application whether the functionality is a first-class language feature or is "faked" through DependencyInjection techniques. Finally, there is no greater flexibility in the ability to attach attributes to an object, because functions have the same capability (as described in TopOnAbstraction) and in any case a large subset of the attributes you'd ''want'' on a FunctorObject can simply be captured in its closure without explicitly declaring them. -DavidMcLean]

* There are not very many "common higher-order function patterns" in custom biz apps. You are solving a problem that is not really a problem. I'm sure you will disagree about the frequency, but we'll LetTheReaderDecide if they are common. (Whisper: they're not. That's why he ''had'' to reinvent off-the-shelf-software to force an example.)

* ''"Custom biz apps" is a broad church.  A big part of it is data entry and reporting, which has little call for higher-order functions.  Another big part of it involves implementing and using algorithms for routing, planning, costing, optimisation, financial calculations, scheduling, timetabling, visualisation, "big data" analysis, sentiment analysis, social media monitoring, and so on.  For these, higher-order functions may well be appropriate.  BTW, I didn't have to "reinvent off-the-shelf-software to force an example".  The reality is that off-the-shelf software is frequently not suited to business-specific needs -- which is just as it always has been, since the dawn of off-the-shelf software.  I gave an example where that is the case.''

* Other than "statistical analysis", which is usually staffed by math and statistics majors who use statistical packages such R and SPSS, I very rarely see such activities among or involving the usual "developer pool" at the roughly 30 organizations that I've contracted with or worked for, big and small. I suspect they usually buy/rent off-the-shelf (OTS) software for production "optimization" problems, and that your particular specialty is custom versions of such optimization software. But that '''specialty would give you a biased view''' since if the org used OTS for such, you wouldn't be there to witness that fact because they wouldn't need you. But again, we'll LetTheReaderDecide. If the reader encounters or needs lots of custom optimization software, they can go ahead and HOF it up! No use continuing this AnecdoteImpasse. Anecdotally, I'm pretty confident my time at the 30 or so organizations is a sufficient sample size to know you are exaggerating the frequency. I'd bet money on it. If there are many, the orgs '''hide them in closet''' -- which is understandable if they are as willful and stubborn as you. (Yes, that ''was'' an AdHominem.) -t

* ''Why do you feel the need for an AdHominem attack?''

** Venting of frustration

** ''Why are you frustrated?''

** Because I'm dealing with an arrogant willfully-stubborn reality-hallucinating field-naive asshole. You asked.

* ''It's hardly surprising that you'd seek employment in organisations that reflect your inclinations, and be hired by organisations that seek your abilities.  As a result, the 30 organisations you've worked in almost certainly share commonalities that are not necessarily reflective of the IT industry as a whole.  My professional background and continuing professional connections are with software companies and forward-thinking (as opposed to those who grudgingly endure IT as a cost of doing business) IT departments, which are often eager to create their own software to sell, or to gain competitive advantage over companies who use off-the-shelf software.  Developers in such organisations are not afraid of HOFs.''

* Re: "Seek your abilities": having optimization projects going on in them (separate from what I do) would not prevent my hire. The same filtering mechanism that applies to you (above) would not apply to me since a mutually-exclusive build-versus-buy choice would affect their evaluation of me because I don't specialize in optimization. If they were using OTS for such, the programming pool(s) I encounter probably would not be involved and thus I wouldn't see such usage. If on the other hand, many org's did roll-their-own optmizers, I'd see those pools working on it even IF I wasn't directly involved in such projects. If my special is custom X's, then that directly affects how often I'll see off-the-shelf X's. However, if my specialty is Y, then that will have little if any impact on the ratio of custom X's versus off-shelf X's. Make sense? If they want a competitive advantage by rolling everything themselves, why not also roll their own word processors, OS's, spreadsheet software, etc? Why only roll optimization tools?  Most developers are looking for a job, not stock options anyhow. I'm focusing on what '''typical''' companies actually want and do, not some GoldenHammer pie-in-the-sky-profit-machine.

* ''Word processors, OS's, and spreadsheet software are highly generic, often called "horizontal market" software for that reason.  Vertical market software is far less generic, and far more coupled to business-specific infrastructure, data models, tools, and so on.  It's hard to obtain a competitive advantage with a new spreadsheet package, because the new package will be just like the old one and just as integrated (or not) into the corporate infrastructure.  A new scheduling and timetabling package that ties directly into staff and student record systems, and handles a university-specific way in which rooms and timeslots are rotated, could definitely represent either a significant competitive advantage or a significant efficiency gain.  As for HOFs, even if you never need them to inject customisations into business algorithms, the trend in UIs and UI frameworks is toward greater customisation and greater higher-order capability.  Unless your apps don't have user-interfaces, you're going to have to get used to using HOFs and other higher-order functionality.''

* There are a lot of ways to improve on spreadsheets. Excel is hardly the pinnacle. But, again, we'll let the reader decide the needs market for custom optimizers. As far as your UI statements...

* Rest of discussion moved to PopularityOfJavaScript.

''And you've been invited, several times, to pretend Javascript/DOM's native timer API is not based on a higher-order function, and demonstrate how you could implement the "Brady Bunch example" without HOFs.  Show us how it would be better.  Our car is in the race; it's time for '''you''' to RaceTheDamnedCar.''

* The top of BradyBunchGridDiscussion shows what "could be". If you want to limit what parts of the web I can theoretically overhaul and which parts I can't, please state the limits and the reason for the limits. I don't dispute that your HOF wrench is better at fixing '78 Fords; you've won the '78 Ford RaceTheDamnedCar award, so congratulate yourself on that one and pat yourself on the butt. BUT, I am not interested in specific brand and model-year fits.

* ''The limit is that your code should be executable.  To simulate a non-HOF Window.setInterval, either don't use it at all, or pass a named function that accepts arbitrary code in a string and invokes eval().  We'll pretend it's a setInterval that accepts code in a string rather than a HOF.''

* Please explain. Why mirror JS's current approach and only their current approach? It's an artificial limit. Even with current HTML using the HEADER refresh approach with frames, any JS in the body is "executable".

* ''It makes for a level playing field, because it compares viewable code against viewable code.  HTML using the HEADER refresh approach with frames is comparing viewable code against pre-written code -- the implementation of frames, etc. -- that we can't view.''

* But this is not a battle over SystemsSoftware design (GUI implementation). That's a silly distinction. SovietShoeFactoryPrinciple. Reason number 46 why we should avoid GUI-related comparisons. We can't see the implementation code for JS's setTimeout function either.

* ''Of course this is not a battle of SystemsSoftware design -- it's a battle over programming with HOFs vs programming without HOFs.  Therefore, any reasonable comparison must be about the code, and nothing but the code.''

* Good! Then the code at the beginning of BradyBunchGridDiscussion is better code for the marketplace than HOF's--No ugly nested-bracket crap: foo(function bar{....}). Only a mother could love that syntax.

** ''See above -- the code should be executable: "The limit is that your code should be executable."  Are you prepared to write an interpreter for the code at the beginning of BradyBunchGridDiscussion?''

** No, I'm not. But if you can point out a reason why it cannot be implemented, please do so.

** ''I'm sure it can, but why?  It's awful.  It's inflexible and difficult to learn, being constructed from an undifferentiated soup of arbitrary syntax with no underlying consistency or conceptual principle.  It's the reason we have higher-order functions and the like -- so we can avoid a proliferation of clumsy and confusing special-purpose languages in favour of composing arbitrary solutions from simple, general-purpose constructs.''

** "Difficult to learn"? That's your opinion. You are not a representative specimen. And, again the market-place prefers special-purposes languages over the longer run. We've been over this already.

** ''The market-place prefers special-purposes languages?  You mean, like C#, Java, PHP, Python, Perl, Ruby, and before them, C & C++?  I guess your definition of "special-purposes" differs from mine.  And everyone else's, it seems.''

** Php was originally created for the web, the C family tends to be for SystemsSoftware and packaged software, not CBA, and Perl was created for making system reports. When they become popular enough, they participate in the FeatureCheckboxBrochureGame.

** ''Yet, the majority of custom business software over the past two decades has been created using general-purpose application development languages like C#, Java, Visual Basic and C++, not "special-purposes languages".  I'm not even sure what you're referring to by "special-purposes languages".''

** They each generally have their niche. The C family tends to be used in cases where performance and resources are at a premium, for example. No sane person would write an OS in Visual Basic, and no sane person would write lots of small-usage departmental CRUD apps in C++.

** [Yes, for any particular problem some languages are much better choices than others. However, C, VB, and C++ are all GeneralPurposeProgrammingLanguage''''''s, in that they're not tied to any ''particular'' domain (as that would make them DomainSpecificLanguage''''''s instead). For instance, while C is indeed very good for SystemsSoftware, it's also been used to develop a huge range of applications spanning practically every applications-software domain, so it'd be absurd to claim it's not general-purpose. The majority of custom business software---indeed, the majority of software---created over the past two decades has been written in GeneralPurposeProgrammingLanguage''''''s; for biz software, the languages used are often C#, Java, VB, and C++, with occasional uses of PHP, Perl, or Ruby. -DavidMcLean]

** I think we are entering the LaynesLaw zone. Any TuringComplete language is technically "general purpose". However, they each are optimized for certain tasks or programming styles. It's impossible to optimize for everything per WaterbedTheory.


** C is lousy at string handling, I should point out, such that it's avoided for string-heavy projects UNLESS performance is more important than code writing and maintenance costs. It's one of the reasons Perl became popular early in the web. I don't see C used very often for "typical" in-house custom biz apps, other than because  of "that rogue programmer who dragged it in 5 years ago".

** [GeneralPurposeProgrammingLanguage essentially boils down to "not a DomainSpecificLanguage". The vast majority of languages used for business (and for programming as a whole, really) fall under this category. C might need a library of functions for string handling added-on, but a truly DomainSpecificLanguage might just be totally incapable of string manipulation. -DavidMcLean]

** Maybe not specific/narrow domains, but they are domain-leaning at least. Any TuringComplete language can be made to handle strings.

** [Sure, but definitionally a language which isn't a DomainSpecificLanguage is still a GeneralPurposeProgrammingLanguage. -DavidMcLean]

** That may be a FalseDichotomy. Languages can lean certain ways without being "all in".

** [Absolutely languages can lean certain ways, but definitionally a language which isn't a DomainSpecificLanguage is still a GeneralPurposeProgrammingLanguage. It's a perfectly correct dichotomy. -DavidMcLean]

** I disagree. They can be DLL: Domain-leaning languages. It ain't Boolean.

** [Even if a programming language is domain-leaning, if it is not a DomainSpecificLanguage it is a GeneralPurposeProgrammingLanguage, by definition. Sure, you mightn't ''want'' to write a typical Web app in C, but you absolutely could, because it's a GeneralPurposeProgrammingLanguage. You can't write a Web app in (exclusively) PostScript, TeX, make, or SQL. -DavidMcLean]

** Any TuringComplete language would be "general purpose" based on that description. Anyhow, this is quibbling over artificial vocab lines drawn in the sand. I'm not in the mood for a LaynesLaw fight over this. The main point is that some languages make coding certain activities or styles easier than others.

** [Nope, TeX is TuringComplete and is not general-purpose based on the description given. In any case, while it's trivially true that some languages make doing certain things easier than other languages do, the industry preference is towards languages focused on generic abstractions rather than on hard-coded support for particular tasks; for one obvious example, all the major programming languages used in industry provide the notion of ObjectOrientedProgramming, which is a generic organisation concept, and outside of third-party extensions none of them provide a task-specific feature like (for instance) XML literals. -DavidMcLean]

** Re: "Industry preference is towards..." -- the industry bounces around preferences and swings back and forth. I'm not putting any stock in a current trend/fad, even if there is a short-term move there.

*** [Over the history of programming, there's been a consistent continuing movement towards languages affording stronger abstraction potential. -DavidMcLean]

*** In part because we have more horse-power now, but generally new languages come along with some BigIdea or hard-wired with some new UI flavor and gradually replace the existing version-added-feature-heavy languages, starting the cycle over. (We've had this life-cycle discussion somewhere already.) Thus, I don't entirely agree.

*** [New languages absolutely do come along, but they don't come along without higher-order functions any more. A new language will have proper support for strings, because C isn't the pinnacle of programming any more. Similarly, a new language will, barring some kind of environmental constraint, have proper support for functions, regardless of the BigIdea it might be presenting. -DavidMcLean]

*** In my opinion, HOF's have too much overlap with OOP for what app programmers would typically be doing with it. But we'll see what the future holds. (Clipper, an ExBase compiler, had HOF's back in the 80's. Nobody I knew used them, though. COBOL has OOP now, but I doubt it's used heavily. Maybe time and open-source has made it easier to throw in features for the hell of it, whether used or not in the field.)

*** [The conceptual overlap is strong, but as noted the FunctorObject equivalent of many higher-order function patterns tends to bloat these patterns to the point of sacrificing their utility, especially when closures are employed (with manual FunctorObject equivalents, all the closed-over variables must be explicitly passed into the constructor). The equivalence seems less a case for eliminating first-class functions, however, and more for unifying the two concepts. JavaScript actually does a pretty good job of this, representing all methods simply as functions that are stored in particular places. Throwing in features "for the hell of it" really isn't the best way to justify first-class functions, by the way: SchemeLanguage is about the furthest you can get from feature-itis in a language, and it's built mostly upon first-class functions. -DavidMcLean]

*** I think we've been over this already somewhere. Deja Vu.

*** [We've been over ''some'' of it before, in TopOnAbstraction, hence my "as noted" qualifier. However, I hadn't previously noted the ''additional'' boost in FunctorObject verbosity when closures become necessary, just the initial overhead involved in creating FunctorObject''''''s at all, nor that the appropriate design decision in response to the FunctorObject/lambda equivalence is to unify the concepts, nor that JS and Scheme do this exactly this and fairly well. That stuff's new. -DavidMcLean]

*** The only "practical" use you've shown is inventing new block structures (for a database-like reinvention), which is just the kind of technique of '''cowboy abstractionists''' that has ruined lisp's reputation in the marketplace.

*** [There are two things that historically ruined Lisp's reputation in the marketplace: "the perception that a festival of parentheses hurt readability, and the perception that prefix notation hurts readability." Both of those seem to be lessening, however, as ClojureLanguage is growing in acceptance. The creation of new block structures has never been a big issue, and it hasn't hurt RubyLanguage (BlocksInRuby), JavaScript (anonymous functions), or PythonLanguage (decorators/with). -DavidMcLean]

*** I too have pondered syntax-related issues, such as MaspBrainstorming to kind of mix TOP and FP. But so far nobody's offered what I consider a viable "fix". Ruby and JS don't produce anything as simple as EssExpressions, because they can mix/use different delimiters, nesting characters, etc. in rather arbitrary ways, hard-wiring say square brackets at nesting level 3, parenthesis at level 4, curly braces at level 5, etc. It's ugly and confusing, offering huge potential for write-only code abuse.

*** [Ruby and JS certainly aren't as simple as EssExpressions, but your notion that they mix delimiters in arbitrary ways, as well as that this offers potential for abuse, is entirely fictional. Both languages associate delimiters with specific built-in constructs: square brackets for arrays, parentheses for grouping (and invocation, in JS), and curly braces for blocks (and object literals, in JS). About the only complaint you could make is that JS reuses the same delimiters for two purposes, but parentheses for invocation/grouping are an accepted convention of all C-style languages, and it's obvious from context whether braces are used to indicate a literal or a block. (In addition, there's CoffeeScript's rule that braces ''always'' mean object literals, if JS's conflation is really that big an issue.) It's impossible to abuse these delimiters for write-only code, because developers have no control or customisation of the delimiters required at any "level", because they're strictly associated with language constructs. -DavidMcLean]

*** But the fact that "grouping" and "arrays" and "blocks" are different things is part of the problem. Too many idioms that half-way do the same thing. The beauty of lisp is that it says: here's a structure: you can do array things with it, grouping things with it, and block things with it etc. as you see fit. You don't "have" arrays and don't "have" blocks, but rather DO block-ish things or DO array-ish things to any structure. By trying to semi-Algol-tize those languages, they lost some of the cleanness of Lisp. (I still believe there may be a better way to get such, but nobody's found the right mix yet.)

*** [… um, you just totally reversed your position in this argument. If you're arguing that the assortment of varying syntax for varying concepts (which don't do the same thing, by the way) is the problem with languages like Ruby and JS, then surely Lisp, which doesn't have an assortment of varying syntax, would be the most-used language out there. What are you trying to prove? -DavidMcLean]

*** The issue at hand was why OOP is not good enough to do what HOF's do. You pointed out some examples, and basically I'm saying those are trying to do Lisp-like advanced abstractions, but not as well as Lisp. Thus, they satisfy neither audience: "corporate" language users and "abstraction cowboys". Corporate language users/shop-owners don't care about such features/code-patterns, only abstraction cowboys do, and abstraction cowboys would rather use Lisp because it doesn't have "too many idioms" like Ruby etc. does.

*** [Lisp doesn't have a monopoly on abstraction: Being an abstraction technique does not make something "Lisp-like", and in particular first-class functions are sufficiently widespread that I think it disingenuous to attribute them solely to Lisp. (HomoiconicLanguages, by contrast, are an example of a feature I'd find reasonable to describe as Lisp-like.) Nor is Lisp the be-all and end-all of abstraction! In any case, you again purport an unwillingness to abstract in corporate software development that simply doesn't mesh with the proliferation of progressively more advanced abstraction tools and features in corporate languages (even Java and C#) and the subsequent widespread adoption of these tools. -DavidMcLean]

*** Lisp is better suited for wider abstraction than the current AlgolFamily attempts. Like I said in the intro, Lisp's "flexibility-to-syntactical-complexity ratio is near the top". AlgolFamily abstraction add-on's do abstraction "ugly", in comparison, and would be even uglier if more abstraction was squeezed out of them. If we only look at HOF's and only look at them for semi-occasional usage, it may not matter much (and in such cases, OO is good enough anyhow in my opinion). As far as the inclusion of certain abstraction tools in languages, just because they are available does not mean they should be used in all or most cases. Just because it's possible to give every foot soldier a nuke does not mean the military SHOULD give every foot soldier a nuke (or approve their frequent use). '''High abstraction gives bad or evil programmers more room to blow up the base camp''' (such as jamming up a project because current staff cannot grok the code to fix or add a key feature.) Low-brow repetition is often the lessor of two evils. I'm just the observant messenger. It's my opinion that too many WikiZens look at such issues from an engineer's standpoint instead of from an owner's or manager's standpoint. -t

** Re: "ObjectOrientedProgramming, which is a generic organisation concept...[that uses] third party extensions..." -- That's largely why we really don't need HOF's. QED. -t

*** [That's completely misquoting what I said. -DavidMcLean]

*** I guess I misunderstood your point, then.

*** [I'll restate with more specifics, then. The languages primarily used in industry (C#, Java, PHP, etc.) all support ObjectOrientedProgramming, a generic organisation concept applicable to many domains; most of them also support higher-order functions, another generic technique. ''None'' of these languages, by contrast, have particularly domain-specific features baked into them, XML literals being one example. I mention "third-party extensions" because there is an extended version of PHP that adds exactly that feature ( https://github.com/facebook/xhp ), but "vanilla" PHP remains primarily generic. -DavidMcLean]

*** PHP has the dollar-sign-in-strings feature that facilitates building HTML/XML strings. That's not directly domain-specific, but domain-helping. I expect more domain-specific-helping features on languages in the future. And I still suspect something like ColdFusionLanguage but that's open-source will come of age. Incidentally, its roll-your-own-nested-tag feature is in some ways HOF-like, although I'd take a different approach myself. It has hints of MASP-ness (MaspBrainstorming). The C-style may just shrink and be replaced with XML-syntax-based programming. If most of your work is assigning values to attributes, then an XML-style may work better than C-style, which is either crappy or dialect-inconsistent at mass attribute supplying, especially if we want named slots. But trying to predict the future is a messy profession. We'll see...

*** [PHP's string interpolation isn't domain-helping for any specific domain, because strings are a general-purpose datatype. If interpolating into strings in PHP performed HTML escaping, it could be construed as an HTML-specific utility. XML is rarely suitable for general-purpose programming due to its sheer verbosity (although XML-minus-verbosity is a very capable language syntax also known as EssExpressions), and it optimises for rather different requirements than a typical programming language would. Assigning literal values to attributes is of course part of most applications, but generally not one so important that it be made the fundamental operation in your language's syntax. -DavidMcLean]

*** Yes, strings are a general purpose data type, but some domains/projects use them far more than others. There's been very little pressure to add better string handling to Fortran revisions, for example, because it's mostly used for high-volume number crunching, such as weather forecasting; its usual niche does not heavily rely on strings. As far as XML versus EssExpressions, there are already long topics on that. Note that in my opinion XML is usually easier to read even though it's more verbose. Compact-ness is not necessarily the most important feature to a given language user/reader.

*** [The main reason XML is inappliable as a programming-language isn't its verbosity. It's that it can only express one type: trees of strings. You ''can'' encode a more useful basic set of types on top of that, but it's simply not practical to have to write things like <num>53</num>, <var>x</var>, <string>blah</string>, and so on all the time. EssExpressions, at least as implemented in programming languages, are significantly more usable because they define syntax for these important kinds of literals. -DavidMcLean]

*** The job of "enforcing" types would typically be the DTD, or an enhancement too. I see no reason to repeat such info per instance under normal circumstances.

*** [It's impossible to use a DTD to encode sufficient type information into an XML-syntaxed programming language. If nothing else, there is absolutely no difference between <someOperation>a literal string</someOperation> and <someOperation>name of a variable with a string in it</someOperation> unless you wrap each in <string> or <var> tags to indicate the distinction; no matter how complex your DTD is, there simply isn't enough information in the XML itself to express this difference. EssExpressions, critically, distinguish between literal strings (which have quotes) and symbols (which don't). -DavidMcLean]

*** I guess I'd have to see a specific UseCase. I don't know what you are ''trying'' to achieve. But this is not a good topic to continue on XML versus EssExpressions.

*** [The intent is to have general-purpose programming that can be expressed in XML and/or EssExpressions. While both formats express trees, XML is unsuited for the purpose since fundamentally it requires you to annotate your tree with whether a given identifier is to be treated as an identifier or as a literal string. This is true of all languages, of course, and most, including EssExpressions, solve it by using quotes "" to indicate literal strings. (ToolCommandLanguage takes the interesting approach of using $ and [] to indicate things which are ''not'' literal strings.) XML could perhaps be extended with quotes, symbols, and other EssExpressions-style syntax rules, but then it's not really XML any more, is it? -DavidMcLean]

*** By UseCase I meant a programming example. XML can represent anything EssExpressions can. The issue is the notational and language tradeoffs involved.

*** [Any code ever would serve as an example. For the sake of demonstration, here's FizzBuzz in Scheme:]


	(define (divisible x n)
	  (= 0 (remainder x n)))


	(define (fb n)
	  (if (= 0 n)
	    #t
	    (begin
	      (fb (- n 1))
	      (cond
	        [(divisible n 15) (printf "FizzBuzz")]
	        [(divisible n 5 ) (printf "Buzz")]
	        [(divisible n 3 ) (printf "Fizz")]
	        [else (print n)])
	      (printf "\n"))))

	(fb 100)

* That deserves a "Like"

*** [In XML without some basic syntactic extensions, it'd need to look something like this:]

	<define name="divisible" args="x n">
	  <equals>
	    <num>0</num>
	    <remainder>
	      <var>x</var>
	      <var>n</var>
	    </remainder>
	  </equals>
	</define>

	<define name="fb" args="n">
	  <if>
	    <equals><num>0</num> <var>n</var></equals>
	    <then><bool>true</bool></then>
	    <else>
	      <fb><minus><var>n</var> <num>1</num></minus></fb>
	      <cond>
	        <case>
	          <divisible><var>n</var> <num>15</num></divisible>
	          <printf><string>FizzBuzz</string></printf>
	        </case>
	        <case>
	          <divisible><var>n</var> <num>5</num></divisible>
	          <printf><string>Buzz</string></printf>
	        </case>
	        <case>
	          <divisible><var>n</var> <num>3</num></divisible>
	          <printf><string>Fizz</string></printf>
	        </case>
	        <else><print><var>n</var></print></else>
	      </cond>
	      <printf><string>\n</string></printf>
	    </else>
	  </if>
	</define>

	<fb><num>100</num></fb>
.
*** [I even cheated a little: Support for escape sequences like \n is an extension of XML. Is the issue apparent now? -DavidMcLean]

*** It seems you are trying to match it 1-for-1 with Scheme syntax instead of an imperative language better suited for XML.

*** [Since it doesn't have Scheme's syntax any more, that's not really the case. I have mirrored the ''semantics'' of the Scheme version, however, which I assume is what you meant. This isn't particularly relevant, though, because the problems being illustrated are totally unrelated to how like or unlike Scheme an XML-syntaxed language might be; switching to something else (say, Python) isn't going to eliminate the big problems with XML as a general-purpose programming syntax. -DavidMcLean]

*** If by "general purpose" you mean translating or representing existing languages one-for-one as they are, I agree XML is verbose for such. If we were going to use it as an imperative programming language, then I'd suggest one devise techniques that better fit its syntax, perhaps such as rely more on type-tag-free (type-indicator-free) literals and variables, which is what ColdFusionLanguage did, I would note. -t

*** [By "general purpose" I mean a language usable for general-purpose programming; it needn't correlate one-for-one with existing languages. Unifying the string and number types into one "scalar" type as in PerlLanguage would ''slightly'' reduce the XML tags needed, but not make a significant impact even on the above example. With unified scalars, there's still the necessity of differentiating variables from literals and of a fairly awkward prefix style. (= 0 n) is pretty awkward in Scheme too, really, but it's even less pleasant in pure XML. -DavidMcLean]

* [If you're only judging code on its syntax, and you already prefer XML to JavaScript's syntax, then you'll obviously prefer your design. The JS version is equally practical if written with different syntax, however; it could instead be CoffeeScript and remain perfectly executable, and then the code would look like this: http://gigahard.mine.nu/~david/hofajax.coffee.html Whether the syntax appeals to you individually is not the only metric for judging code's quality, of course, however. -DavidMcLean]

* Syntax matters when things go wrong because overly complex or nested syntax can make it difficult to separate out semantic from syntactic issues.

* [Certainly. JavaScript's syntax isn't actually particularly complex or difficult to understand in this case, however: The syntax used for passing anonymous function arguments, which you find so distasteful, is an entirely consistent result of combining the function invocation and function literal syntaxes. It's not exactly ''pretty'', but it's actually quite simple, and it doesn't obscure semantics to a significant extent. There's not even very much overhead, since you consistently use a single closing }); for each higher-order function call. Three characters are hardly going to obscure semantics. -DavidMcLean]

* When used sparingly, perhaps. If they get intertwined, things can get messy.

* [Not really. You just use a single }); to close each function call you opened. Aside from being more symbolic, it's pretty much the same as Ruby's do-end block syntax, including using three characters to close blocks ("});" <-> "end"). You can't avoid having ''some'' sort of delimiter to close blocks without using SyntacticallySignificantWhitespace, of course. -DavidMcLean]


As far as what tools to include in the "tool-box", it's a matter of weighing the trade-offs. I won't go into those again here because I've already discussed it in multiple topics of late. (Frankly, I wouldn't personally miss FOR loops; I don't use them that much. I'd gladly trade that for far more useful features such as optional named parameters.)

''No, as to what tools to include in the "tool-box", it should never be a matter of "weighing the trade-offs" because that's nothing but a euphemism for catering to the lowest common denominator.  It should always be a matter of providing the most powerful tools possible within the given language paradigm.''

I see no reason why we should ignore what the markets wants and real-world staffing situations. And I'd call it "mid-common-denominator" not "lowest". You are exaggerating.

[Language design should focus on what programmers need, because programmers use programming languages. Managers do not use programming languages. Thus, when designing a language the primary goal is making it as useful as possible ''for programmers''. -DavidMcLean]

I'd estimate that at least half of development managers were programmers themselves. If their experience with the multiple programmers that have come and gone in their shop showed that heavy abstraction and functional indirection made for better projects, including maintenance, it would gain acceptance. And when they do fuss about such, it's often not "I don't get it", but more along the lines of "the last guy who did things like that made code too confusing for the rest of our team after he left. Many couldn't maintain the code, putting us in a bind during vacations."

''If a shop that wants to outlaw certain language features, that's fine.  Some shops outlaw C++ multiple-inheritance and/or templates, other shops make effective use of them.  That's no reason to consider excluding multiple-inheritance and/or templates from C++.''

''As for features that "gain acceptance", note that HOFs are available in Javascript and lambdas are appearing in C# and Java.  It seems they are gaining acceptance.''

I believe it's a FeatureCheckboxBrochureGame. Time will tell. New languages and fads will come along and people will start forgetting about Java etc. And if people cannot find a decent use for them in their domain, HOF's will mostly be ignored. Outside of existing API's forcing it on you, you haven't found decent cases.

''...Except, of course, for every indivisible algorithm that requires customisation, as shown in HofPattern.''

It's great because you say it's great. The pattern is rare in the wild, or at least uncompetitive with alternative solutions.

[What about qsort(), which is part of C's standard library? What about approximately half of Ruby's standard library? Are standard libraries rare in the wild? What about any genetic algorithm ever written, DijkstrasAlgorithm, AstarSearch, and so on? -DavidMcLean]

I've never ever used qsort. I imagine it's useful for SystemsSoftware and embedded systems, but those are not my niche. And much of Rudy's library, and perhaps qsort's comparison techniques, could be implemented in objects. If you don't like Ruby's object syntax, complain to the Ruby makers, not me.

[If you use an object constructed on-the-fly with an associated callable method---the C++ functor pattern, effectively---then you're using higher-order functions. It's exactly the same thing, just with syntactic overhead. The minimal syntactic overhead on blocks in Ruby is ''vital'' to their utility in code; any syntax that required declaring an object-with-some-named-method to pass to things like Enumerable#each would be too verbose to be used as ubiquitously as Enumerable#each is and should be. JavaScript might be a better example to look at here, because its object syntax is ''extremely'' concise; despite this, one finds that the vast majority of functions-that-need-to-take-functions just take functions, not objects. -DavidMcLean]

* Moved rest of discussion to DatabaseAbstractInterfaceDiscussion.

------

Re: "Abstraction Versus Marketplace"

Or is it TheoryVersusPractice??

FalseDichotomy -- The third point is, for lack of a better term, "elegance" which is always tied to the task at hand. LISP is the capstone for functional programming, but nowadays, it's about data -- a bunch of nested parens representing data relationships isn't suited for recursive solutions.

''Recursion-centric techniques are not really "fighting" for territory with FP and "traditional Algol-style", at least not that I see. Perhaps it could become a visible competitor, but it's not there yet.''

-----------
"Intelligence" is not so clear cut. I don't consider my wife very "intellectual"; however, she has a great ''street smarts'' that I admire much.
--------
See also: IfFooIsSoGreatHowComeYouAreNotRich, WorseIsBetter, EconomicsOfAdvancedProgramming, OneMoreLevelOfIndirection, AynRandDesignPhilosophy, IndustrialSociology, StaffingEconomicsVersusTheoreticalElegance, SummaryOfHofExamples
------
CategoryLisp, CategoryIdealism
----
FebruaryThirteen JanuaryFourteen