NaturalEventSyntax continued

Here's a draft generic event and remote service protocol that is not intended to favor or lock one into any particular language, paradigm, or mark-up; yet be sufficiently flexible. (Being designed around "remote" services does not preclude local services. Also, title is a misnomer as this is not about a syntax. I'll call it "Eventop" for now if we need a working name for reference.)

* Synchronous calls - Somewhat similar to an old-fashioned HTTP GET. Typically used if response is expected within about 15 seconds.

** [Fifteen seconds. A synchronous call that might take up to fifteen seconds. Just… just think about that. -DavidMcLean]
** An asynchronous call or call set may be more appropriate then. Whether a max response time is hard-wired or configurable can be debated.
** [A single synchronous call shouldn't ever take a whole second, and I suspect even 500ms is still pushing it. This is one of the reasons HTTP GET requests are not synchronous calls. -DavidMcLean]

* Asynchronous calls - requests don't wait for a response, and responses can come at any time without notice (while the session is active).
** Send event
** Receive event - if the target system cannot process multiple received events at the same time, they queue up; i.e. the language- or system-specific driver should have the ability to queue incoming events. There may be optional "fullness handling" attributes that control what happens is the queue exceeds a threshold. Options would be ignore event without notice, ignore event with notice (which itself may fill the queue), or trigger an error. (Maybe a simple "static" flag can be specified as part of the std. to indicate full-ness to avoid filling the queue with error events.)

* Input is a tree-map and output is a tree-map (nested map). Each event/call will have one of each (in some cases they may be empty). Each app language or markup system can translate the tree-maps to their own favored structure if they want. We'll postpone defining the details of the tree-maps for now. For discussion purposes, let's assume that the two tree-map choices are XML and JSON.

* The "srvc_status" branch on tree-maps are used to define or determine if there were errors or warnings. Here are example values:

   srvc_status.error_count = "3"
   srvc_status.error_msgs.1 = "Tires are missing"
   srvc_status.error_msgs.2 = "Driver is drunk"
   srvc_status.error_msgs.3 = "No license plates"
   srvc_status.warning_count = "1"
   srvc_status.warning_msgs.1 = "Left tail-light cracked"

A prototypical event handler sample function:

  function sample_event(inputs: treemap): treemap
    var outputs: treemap
    ...
    if inputs.fooBranch.barLeaf = "cat" then
      outputs.branchX.branchY.leafX = "foo"
    end if
    ...
    if noProblems then
      outputs.srvc_status.error_count = "0"
    end if
    return outputs
  end function

This is not to imply the results must be processed with functions or function-like blocks. They could be queued or sent from tables in a database, for example, with diff apps in diff languages reading from the reception table(s) and writing to the output tables(s).

A tree-map is chosen because it's relatively simple yet relatively flexible. EssExpressions were considered, but are too confusing to many people, and don't have native or sufficiently-close counterparts in many tools.

--top

[How does this standardise ''anything''? Your proposed protocol is essentially "input and output that are isomorphic to JSON"; this doesn't enforce any of the consistency benefits you were extolling in NaturalEventSyntax, because it doesn't really enforce anything at all. Services may still use arbitrary ''keys'' in the dictionaries they're passing around, so there's no assurance that two services will be even slightly compatible. What benefit does this system afford over actually using JSON, which is in fact already available on all widely-used languages and paradigms? -DavidMcLean]

Please clarify what you mean by "keys".

[In your above example, fooBranch.barLeaf is a key. The key in general is that which references a specific value in the dictionary. -DavidMcLean]

It's local and unique to the event "call". It's like a local array. I didn't define any global (session-wide) data structures at this point (but may consider it if a common need found.)

* [Yes, I got that bit. How does making it local address my concern? -DavidMcLean]

* I'm not understanding what your concern is. How about a sample collision re "unique".

* [Ah. Indeed: My problem has nothing to do with collisions, nor to do with uniqueness. The issue is that you've advertised "swappability" between two different implementations of the same functionality as a feature of this protocol. However, the protocol as presented doesn't have any way to ensure that such a thing happens. Consider two search-as-you-type implementations, since that's the example we're on: Suppose one of them takes a search-box identifier argument and calls it inputs.searchBox. Even if the other one also takes a search-box identifier, it might be looking in inputs.queryBox instead. Thus, there's no assurance that the two implementations will be swappable. -DavidMcLean]

* The map-array (or it's equivalent) is the ''interface'', not the implementation. If you put the wrong value in the wrong attribute of an XML interface, for example, you similarly will get the wrong results. GiGo. If you mean like a DTD for better validation or documentation; yes, we can define something like that, but that's just icing on the cake.

* [I'm not saying XML won't have the same problem; it will. I'm saying your proposed interface doesn't provide the features you claim it will, specifically the ability to swap between different implementations of the same service. Why is it better than JSON or XML if it doesn't provide the swappability advantage you claim it does? -DavidMcLean]

* I'm sorry, I still don't understand what you are getting at. I didn't define implementation and made no statements about implementation. I'm defining an interface. And it's not really competing with XML and JSON because it can be XML and JSON if one wants. Basically it's passing nested data structures (an array called "inputs" and an array called "outputs") to and from 3 event "types".

* [You claimed on NaturalEventSyntax that your interface would afford swappability between implementations of the same service. It clearly does not do this. -DavidMcLean]

* Please clarify. I don't see it.

* [It's really quite simple. Implementations A and B are of the same service and both use your protocol. Implementation A has named a certain parameter inputs.A, and you write your code to put the appropriate value in inputs.A. Then you change to implementation B and find that, although it accepts the same parameters, it has called the relevant parameter inputs.B instead of inputs.A, so your code doesn't work. There's no greater swappability afforded by this protocol. -DavidMcLean]

* That's an interface change. Of course interface changes break shit. There's no magic way around that.

* [That's exactly my point. Your protocol doesn't provide any help with such issues. You claimed on NaturalEventSyntax that it could. Since it doesn't, what benefits ''does'' it have? -DavidMcLean]

* What's an example of "helping"?

* [Strictly-defined slots for particular parameters, such that different implementations of the same thing quite literally ''can't'' have different interfaces. Of course, that takes a huge toll on flexibility. -DavidMcLean]

* Like I mentioned above, we could have the equiv of DTD's to "check stuff". But you are correct in that I'll take flexibility over "rigid checking". Named parameters (maps) are generally more flexible than fixed-position parameters.

* [It's not checking I'm getting at, but a protocol that ''doesn't'' encode arbitrary data structures, as XML, JSON, and your proposal do. Such a protocol would be more like HTTP, in which there is a strict defined set of basic commands available. Without a protocol like that, it's impossible to enforce that different implementations of one service expose the same interface. (Note that if you ''do'' have a protocol of that sort, it's very easy to have different implementations exposing the same interface; consider how many different Web servers and clients there are, all of which interoperate easily.) -DavidMcLean]

* I'm still not getting the gist of your complaint. Perhaps show an example of it "done right" versus "done wrong", and the downsides of the "done wrong" approach in terms of change scenarios. Why is it "arbitrary" versus say fixed position parameters? The possible parameters are also arbitrary, such as the number of parameters. Fixed position parameters are a "list", with the arbitrary aspects inherent to a list. Same with a tree-map: the data structure has certain degrees of freedom and certain structural restrictions. A tree-map has more degrees of freedom than a (parameter) list, but it's still the same interface concept. The meaning of a given interface structure depends on the service. The service (or a human) provides documentation describing the minimum and optional elements needed to communicate with the service(s). Saying that "foo.bar.x" must have a value is no different than saying the 4th parameter of a fixed-position parameter list must be populated. Somebody can come along and change the service to expect it at the 7th parameter position instead of the 4th. Only supernatural powers can prevent such changes from braking stuff.

* [Well, yes. It's obvious that arbitrary positional parameters have the same problem. A protocol that encodes arbitrary lists is as incapable of enforcing interface consistency as one that encodes arbitrary trees; they're both encoding arbitrary data structures, after all. However, I'm not claiming that a positional-parameter list is any more resilient to interface changes than a keyword-parameter tree. You seem to have misinterpreted "[s]trictly-defined slots for particular parameters" as "parameters are in a list", rather than as "each individual parameter has a strictly-defined meaning included in the protocol's specification". To enforce that differing implementations of the same functionality expose the same interface---to be able to advertise that your protocol enables implementation swappability---you ''need'' a protocol such that, for any given command, the numbers and types of allowed parameters ''and their meanings'' are fixed and specified as part of the protocol spec, like HTTP, or IRC, or FTP. Your proposed protocol doesn't do that, therefore advertising a swappability benefit is misleading: It only enables swappability between two implementations when they both have the same interface, and everything that has ever been created allows you to swap between two things with identical interfaces (how would it stop you from doing that, after all?). -DavidMcLean]

* Http has a "rigid" portion and non-rigid portion, such as content (HTML etc.) in its definition. Same with FTP: it doesn't dictate specific file names and quantities, for example.

* [Both protocols are entirely "rigid" in their specification. FTP mightn't dictate specific filenames, but it dictates that, for example, a GET command must be in the form "GET ''filename''", where ''filename'' is the full name of some file on the remote system. Your protocol doesn't have the equivalent of "GET commands must be in the form 'GET ''filename'''"; nor does it even have the equivalent of GET commands. (Not that this necessarily is a ''bad'' thing. JSON doesn't have such a concept either, after all. It does however mean that the protocol doesn't define a particular interface rigidly enough to ensure implementations may be freely switched.) -DavidMcLean]

* That's just "movement" details.

* [What are "movement" details?]

* The protocol for moving stuff from A to B

* [Um. Yes, FTP is just a protocol for moving stuff from A to B. All protocols are just protocols for moving stuff from A to B; that's what protocols ''do''. How is that a counterpoint to my explanation of the distinction between strictly-structured protocols like FTP and more freeform protocols like JSON? -DavidMcLean]

* Many server web platforms represent HTTP interaction in terms of map arrays, with an added "content" element for the markup or results. To get the info, one may call something like GETVARS['http_referer'] and GETVARS['content'], for example. True, that's not necessarily the way it works under the hood, but much if maps nicely into maps (pun partly intended). -t

* [So? Yes, you can access data from HTTP requests and responses through a dictionary (or an object) on many Web platforms. That doesn't change the fact that HTTP, unlike XML and JSON, has a very strict, well-defined structure to which all requests must conform; it is this strictly-defined structure that enables interchange between different implementations of HTTP (both clients and servers). Because XML and JSON impose no such structure, you cannot expect to swap between two client or server implementations simply because they both happen to use JSON. The relevant distinction is between HTTP's strict format required for all requests and the "looser" formats of JSON and XML which are free to represent essentially arbitrary trees. It's because of that distinction that you cannot claim a given protocol will enable swapping between implementations unless it enforces a strict, well-defined structure as HTTP does. -DavidMcLean]

* Like I said, they have the "rigid" part and the open-ended part. HTTP has an open-ended part also (which I called "content" above). The rigid part typically manages the business of moving stuff back and forth and provide necessary or common status and communication infrastructure info. Some frameworks have 3 parts: the rigid part, the "convention" part which is things like suggested names or data patterns, and then the open-ended parts. The "srvc_status.error_count" path in the above example would be the "rigid part" because each transfer package will have it. It's kind of like shipping with UPS: they have standards for all packages they accept, but for the most part the content (insides) of the package is up to the user of their service.

* [The payload delivered in an HTTP response isn't part of the HTTP protocol. -DavidMcLean]

* Please elaborate.

* [The fact that there ''is'' a payload in an HTTP response---the "content"---is a defined part of the protocol. What that payload ''is'', however, isn't a defined part of the protocol. It's not an "open-ended part" of the protocol; it's just not in the protocol at all. -DavidMcLean]

* I think you are saying that Eventop dictates that a tree-map be used to xfer "data" while HTTP doesn't for its "payload".  Perhaps, but that's not necessarily a bad thing. I would note that one could have a path such as "data.bytestream" that is just a blob of bytes.

* [Kind of but not really? Eventop prescribes that a tree-map, probably isomorphic to XML or JSON, be the basic protocol used for communication between services; HTTP prescribes instead that services use a strictly-structured set of basic request-response formats. While the flexibility of the former is not necessarily bad, it means that you can't be assured of interface compatibility between services simply because they use this same protocol. For instance, knowing that two services communicate over XML is absolutely no guarantee that they can communicate with each other. By contrast, knowing that two services use HTTP means that they will be able to communicate, because HTTP is much more strict. -DavidMcLean]

* I don't see how it's "much more strict".

* [Do you know how HTTP requests and responses must be formatted? -DavidMcLean]

* A text file with certain things in certain places. How is that different than a tree-map that expects/puts things in certain places? Addresses are "slots" and byte positions are slots. (We already discussed possible ways to encode tree-maps.) A text file is a kind of data-structure: a one-dimensional array of bytes indexed (at least) by integer position and often by "line" position. A tree-map is just a different data structure, but this standard similarly has a fix/reserved portion and more open-ended portion(s). Why discriminate based on the data structure used?

* [It doesn't matter what kind of data structure is used. What matters is that HTTP defines what the "certain things in certain places" ''are'': For a GET request, for instance, there's a header line in the form "GET ''path'' HTTP/''version''", then there's a series of request headers---which are also defined in the spec---and then there's a message body, which is indeed permitted to be a freeform string. Note that the HTTP spec alone makes well-defined various facts about the request. It's a request of "GET" type, it's looking for ''path'', it's using a given HTTP ''version'', it might've specified an Accept-Language header so the server can choose the correct localised response, and so on. The XML and JSON specs do nothing of the sort. If you know something is "XML", all you know is that it's some kind of tree. Without a further protocol specification that exists outside of XML or JSON itself, you can't know facts like "this XML tree represents an HTTP GET request". -DavidMcLean]

* Eventop can potentially piggy-back on HTTP to avoid reinventing lower-level server-oriented details. But it still "makes well-defined various facts about the request", just like HTTP. (Okay, some are still fuzzily defined.)

* [How? Where's the specification of Eventop's protocol that makes every request sufficiently-well-defined that it's possible to build compatible services just by adhering to the spec? -DavidMcLean]

* Generally I'm starting high on the abstraction tree and working downward. Do you have a specific question that you want fleshed out? Note that we probably don't want to limit it to a particular transport mechanism in order to make it flexible enough to use and swap between say HTTP, named pipes, telnet, etc. without rewriting our app and/or API beyond changing config info.

* [The transport mechanism is irrelevant. Eventop isn't currently well-defined enough to ensure that services are compatible solely because they use it; this issue arises because it's very similar to XML, which has the same problem. You'd need to define a much more strict format than "any nested dictionary ever" to ensure compatibility. -DavidMcLean]

* No, it's NOT like XML. We've been over this already and I answered that already.

* [We did? When was this? (Also, it is like XML. What are you talking about?) -DavidMcLean]

* XML doesn't define any kind of event model.

* I thought I asked essentially the same thing before above, but here goes again: How about a demonstration of a specific "compability failure" of Eventop that the "right way" would prevent.

* [I did provide one of those. You responded with "Of course interface changes break shit". -DavidMcLean]

* It broke your alternative also, or presented a flexibility trade-off such that there's no free lunch.

* [Yes. Like I said, that's exactly my point: there's no free lunch here. You can't claim that Eventop enables swappability between implementations (at least, any more so than anything else ever created) unless you significantly restrict its flexibility such that any two implementations of the same service ''must'' be exposing exactly the same interface to be conforming to the Eventop specification. -DavidMcLean]

* If you present sufficient or common-enough realistic problem scenarios, I may choose to support a different trade-off path. As far as that particular claim, we seem to be considering different scopes of "swappability".

* [I did that already too. There are two search-as-you-type implementations for Eventop. One gets the query text from inputs.queryBox, one gets the query text from inputs.searchBox. They're not compatible, even though they implement the same thing and both use Eventop. As for different "scopes" of swappability, how are there any? Either you can swap one implementation for another or you can't; shouldn't it be binary? -DavidMcLean]

* How are the alternatives immune to such interface changes? Those are decisions (and changes) by the ''service'' designer (such as a SAYT or query service), NOT Eventop implementers.

* [Alternatives are immune to such interface changes because they define an interface strictly enough that any implementation that changes said interface it is not conformant with the specification. Eventop, XML, and JSON interfaces all don't do this, because two services may be entirely conformant with the Eventop|XML|JSON specification but still incompatible---this is because such specifications don't describe a "whole" interface, leaving aspects of interface definition to individual designs. -DavidMcLean]

* No they don't. You are making up things. HTTP does NOT define SAYT paramaters, for example. We can make up a SAYT service standard and call it "SAYTNOOF 2" or whatnot and pre-define all the parameter positions/addresses, but that works for both sides of the argument.

* [No, you're assuming that "the alternatives" means "HTTP". It doesn't. It means something like SAYTNOOF 2, which you've just presented. You can be assured that two HTTP services are able to communicate; likewise, you'd be assured that two conforming SAYTNOOF 2 implementations can communicate. You have no such assurance that Eventop, XML, or JSON services---without any further description of their respective protocols beyond "adherence to the Eventop|XML|JSON spec"---are able to communicate, because they're less strict. -DavidMcLean]

* There seems to be a major misunderstanding: I'm not defining a specific service here any more than FTP defines a specific file tree or ODBC/JDBC defines a specific table structure. A SAYT service is a specific "kind" of service. Eventop is intended as a semi-general-purpose service communication/coordination standard upon which things like SAYT are implemented on top of (minus the GUI part, perhaps).

* [Yes. Exactly. My problem is that you claimed on NaturalEventSyntax that your protocol would afford swapping between different implementations of the same service. As a general-purpose event-y communications protocol, it doesn't enable this at all. Services, including search-as-you-type, must define their own consistent interface, possibly on top of Eventop's basic protocol, and two or more implementations must share that "overlying" interface to enable swapping. Thus, the claim that your protocol affords said swapping is erroneous, which was my original problem. -DavidMcLean]

* "Sameness" of services is usually defined by the interface. If you are talking about some other way to compare implementations to decide "same", I was not aware of it.

* [Two implementations of search-as-you-type that expose different interfaces. They're implementations of the same service, no? -DavidMcLean]

* I don't want to LaynesLaw over "same" today. I think you misunderstood my original swappability point. Which mention instance are you referring to make sure we are talking the same thing. I'll revisit it and try to clarify it.

* [Your original claim went like this: "Second is swappability. Another vendor could provide a better cheaper and/or faster service/component with a same or similar interface so that we can swap it in when it comes. (Vendors tend to copy or follow similar conventions to steal customers.)" This is actually pretty reasonable as a concept; since you've clarified that both services use the same interface, they will be compatible. ''But'' you can't really claim that as an advantage of your own protocol, because it's not your protocol but the vendors' own ones that are providing the benefit; it's an empty truth that two implementations may be swapped if the designers used exactly the same interface, and choosing to use Eventop doesn't really provide any greater assurance of having used exactly the same interface as having both used XML or JSON. I point out the issue in the following forms: "Why would using markup give you any such swappability benefit, if JavaScript doesn't? There's nothing in JavaScript that requires different implementations of the same service to expose different interfaces, and there's not really anything in your markup spec that actually enforces that different implementations of the same service expose the same interface (nor can there be, in general)." and "In addition, you can't ensure that the interface to any given service is standardised simply because it's expressed as XML: As noted below, there's not actually anything in XML that can enforce that different implementations, even of the same service, expose the same interface." In response to the first of those, you suggested that I "see above", presumably in reference to the other; said other received a response along the lines of "The key is standardization of interface, not so much that it's XML.", which would be reasonable if the interface was strict and well-defined enough that standardising it did indeed solve the problem, but it doesn't. At no point in NaturalEventSyntax did you identify that describing your protocol as superior for reasons of swappability was erroneous, nor did you withdraw such a claim, hence my consternation. -DavidMcLean]

* Gosh, what a threadmess that topic is; it's hard to review. I think that statement was about how your API required exposing JS itself instead of packaging it in terms of interfaces so that it could be written in some other language besides JS without changing existing usage instances. One test of good cross-platform interface design is being able to swap the underlying language the first one is built in without breaking usage instances.

* [Oh, ''that's'' what you were getting at? Reasonable, I suppose. For what it's worth, my implementation isn't actually written in JavaScript but CoffeeScript, and it'll interoperate frictionlessly with most any language that can run in the same environment, JS included. -DavidMcLean]


How about you(s) rework SAYT as found in NaturalEventSyntax as JSON then, and show how it would work in a non-JS client. (I agree the name-spacing needs clarification, but it's only a rough draft right now).

[Hmm. Reworking it as JSON. The input to http://gigahard.mine.nu/~david/instant/docs/blended_instant.html is already a JavaScript object; I could change the request to the server and the response to be JSON objects as well, but it wouldn't be particularly useful to do so since only a single value (a simple string) must be transferred each way. What kind of non-JS client do you have in mind? -DavidMcLean]

I see a lot of embedded function calls such that it's hard to tell where the interface itself starts and ends versus the example's implementation. How about FORTRAN. Primitive languages are generally a better test of cross-system communication. You can emulate such by only using explicit functions (no objects and no HOF's) in some other common language.

* [You appear to be assuming that FORTRAN does not have support for higher-order functions. You are however mistaken in this assumption. (It does indeed have no support for closures, though.) -DavidMcLean]
* Oh shit! F77. When COBOL gets HOF's, I'll retire.
* [Fun fact: FORTRAN 77 had higher-order function support too. It's not just an F90 addition. By the way, COBOL also had similar support, called PROCEDURE-POINTER in some versions and FUNCTION-POINTER in others. -DavidMcLean]

[The implementation part of that example is everything above the Sample Usage heading; there's quite in-depth documentation associated with pretty much every line of source, so it should be extremely clear what's going on. What do you mean by "embedded function calls", exactly? -DavidMcLean]

[Why would FORTRAN be a better choice than JavaScript for demonstrating an interface to what are primarily Web services? FORTRAN is not actually usable for any Web app's client-side implementation, nor would it be of much use if it were. In addition, it doesn't have a built-in DictionaryDataStructure; this makes representing a JSON interface quite impractical, but, critically, it ''also'' makes representing your proposed protocol equally impractical. Shouldn't we be focusing on languages that are likely to be used along with this possible protocol? Those languages that make what you're proposing easy (Python, Ruby, and indeed JavaScript are the major ones) also make JSON equally easy. -DavidMcLean]

It would be better because it doesn't assume much in terms of language features. A side benefit is that non-programming and semi-programming tools can more easily hook into it. Incidentally, one can emulate nested maps in Fortran and wrap it in function API's. It's not fun to implement, but is doable. You can assume one is passing in string paths.

[Your protocol assumes ''exactly the same'' language feature that JSON does, namely a DictionaryDataStructure. How is it different? -DavidMcLean]

For one, JSON doesn't define an "event" or "service calls" protocol. Second, some may want to use XML instead of JSON. Sure, one can define XML equivalent to any JSON structure, but why choose JSON as the root reference structure? JS may die out.

''It's more likely that JSON won't die out (at least, not in any relevant period of time, i.e., the lifetime of a project) than your "generic event and remote service protocol" will succeed.  No disparagement meant by that -- it's an obvious truth.  Unless you're prepared to implement it -- such that people can actually use it -- it's not even worth discussing.  No one is going to implement it for you, and if it's not going to be implemented, why waste time considering VaporWare?''

For the sake of argument, let's study the approach assuming it uses a JSON tree-map data structure, or at least not worry about how the tree-map is encoded for now.

[Alright. Let's see your proposal in action. How do you implement, say, search-as-you-type? (We're dwelling on that example a bit, but we do already have directly comparable implementations of it so it seems a good choice.) -DavidMcLean]

Keep in mind that this is not a GUI protocol, at least not out of the box. Something else would have to take care of GUI issues and the interaction details would depend on the specific GUI system or interface used.

As far as the server interaction of SAYT, it could provide the search results. Assuming we use the asynchronous events, an On-Change GUI event (in the GUI system/app) could call our asynchronous "send" event above with the contents of the search box. The display-match-results portion of the GUI side could look in the asynchronous "receive" event queue to get the results data, format and display it.

[So far, sounds like a slightly lower-level version of NodeJs; this is reasonable, because Node's event model is a good one to be mimicking. Let's ''actually'' see it in action, though. Demonstrate something, perhaps with code samples. (Also, why should the "receive" event queue need to be accessed directly? Doesn't this proposal have event listeners?) -DavidMcLean]

That's a language- or app-specific feature. This draft protocol doesn't specify how or when the incoming events are processed, only that the client has some queuing potential if it's not able to digest them quickly.

[Why delegate the concept of "run this when an event arrives" to specific implementations? Isn't the ability to register a listener on particular events important enough to include in the standard? Actually, I'd go so far as to suggest that having some sort of event mainloop, along with listeners registered to deal with specific events, is ''essential'' to the very concept of event-driven programming. So why aren't listeners and the mainloop included as part of your event protocol's standard? -DavidMcLean]

I disagree. That's language or tool-specific. I gave an example of where the event queues could be RDBMS tables and it's up to apps to decide what they want to do with the data. (It's also possible to put On-Change DB triggers on the tables, if the RDBMS supports them.) We can suggest by convention that in-coming events run a language-specific event handler or some sort, but OOP-ish libraries may choose to use objects and FP languages choose to use HOF's, etc. Why dictate a paradigm up-front? Language-specific crap is what fucked up GUI World for an entire generation. SplitOperatingSystemIntoServices

[Is your rudeness necessary? Having an event loop and listeners isn't language or tool-specific. It ''is'' paradigm-specific---specific to the EventDrivenProgramming paradigm---but what you're designing is an event protocol. Why not build it using event-driven programming design? -DavidMcLean]

Insulting technologies (non-people) is fair game. There are too many ways to implement event handling. Why hard-wire one particular way, especially if it goes against the flavor of a language/tool/favorite-programming-style?

[What common ways to implement event handling don't involve a mainloop and event listeners? -DavidMcLean]

I don't know, I rarely dissected their underlying implementations. That's the more the business of compiler writers and such.

[You claim that there are "too many ways to implement event handling" but don't actually know what ways exist?]

Are we talking about at the machine-code level, or the language level?

[Either. You claimed there were "too many ways to implement event handling". What are some of the ways you were referring to? -DavidMcLean]

At the language level one can use at least OOP, HOF's, or polling loops.

[How are those different ways to implement event handling? Are they not all equivalent to "a mainloop with event listeners"?]

TuringEquivalancy? Perhaps, but why does it matter? Why should the standard dictate the event-handling "flavor" on app programming side? Again, that kind of over-specification is what got the world stuck with language-specific GUI's. To serve as a communication platform for a wide range of languages, tools, and devices; that's over-specific. We leave and pick up the packages on the front porch, but who and how you check and/or place the packages there is up to YOUR household: you can sit Gramps there to whistle whenever a delivery comes, or you can open the door and check every half hour, or you can only check when you get home from work, and hope your porch is big enough to hold a day's worth.

[No, not TuringEquivalency. These-things-are-the-same-thing-as-each-other equivalency. Event-driven programming through OOP is implemented using a mainloop and event listeners stored in objects' methods. Event-driven programming with higher-order functions is implemented using a mainloop and event listeners registered by being passed into a higher-order function. Polling loops quite obviously involve a mainloop, and unless there's only one event in existence you need to dispatch to different handlers from the polling loop. The details of how event listeners are registered and how the mainloop is spun differ between these things, certainly, but the actual premise of a mainloop that spins through the events and dispatches them to event listeners is unchanged. So, what ways are there to implement event handling that ''aren't'' equivalent to "a mainloop with event listeners"? If there aren't any, how can there be too many ways to implement event handling? -DavidMcLean]

Three is already too many. The fact that they may be implemented similarly on the machine-code level is irrelevant. I'm not sure mainloop is the only way to implement such, but it doesn't matter. It's a lower-level detail that the std. has to care about.

* [It's not that they're implemented similarly on the machine-code level. They're implemented similarly right up on the app-code level; there's only trivial interface details, due to differences between the paradigms, that distinguish the pure-OO method from the higher-order-function method. -DavidMcLean]

* Some developers and tools are particular about such differences. You may not be, but I'm NOT using you as the reference WetWare specimen.

* [Then those developers and tools can implement the mainloop and listeners using whichever technique they like. I'm only suggesting you include an event-spinner mainloop and the ability to register listeners in the spec. I'm not suggesting you define those things to be specifically implemented using objects, or higher-order functions, or whatever else. Eventop libraries for particular languages would be free to implement them in whichever way best fit the language idioms. -DavidMcLean]

* How do you do that ''without'' favoring a paradigm or programming style? Example?

* [By saying something like this. "Event listeners are a section of code that may be associated with a particular event, and whenever that event arises the listeners associated with it will be executed. An event listener is passed the event's inputs array as an argument. A mainloop() procedure should be provided by a conforming Eventop implementation. When executed, this procedure will loop indefinitely over the events in the queue, handling each one by executing its event listeners. Once handled, an event is removed from the queue." -DavidMcLean]

* No. Unnecessarily specific.

* [How so, and to what paradigm/language/tool/whatever is it specific? (I actually thought I was being a little too ''general'' to be useful while writing that, so huh.) -DavidMcLean]

* I would perhaps include sample implementations to encourage consistent usage, but NOT make it a mandatory part of the standard. I consider it scope creep. A simple app doesn't necessarily need to "register" anything anyhow.

* [Really? ''Are'' there any simple event-driven applications that in fact don't ever register even a single event listener? -DavidMcLean]

* I smell a LaynesLaw mess over "register". Anyhow, if they are always required to handle/process events, they'll organically appear such that we don't have to provide them. Driver/API writers will generally know the 3 common ways: explicit polling loops, objects, and HOF's.

* [No need to use the word "register"; I just went with "associate" in that example spec snippet above. If listeners aren't always required to handle events, how does one design an event-driven app without them? If they are, why not define them in the spec to increase consistency and cohesion between Eventop implementations? -DavidMcLean]

* Polling. Or checking every time some other action takes place (being it may be single-threaded). A simple script may hard-wire what it checks and have no dynamic event "registration". Why force dynamic event registration?

* [A script that continuously polls for particular events and runs code for those events ''is'' spinning a mainloop and activating listeners in response to certain events. Event listeners aren't necessarily registered "dynamically", so it's hardly forced. Still, a script that has hard-wired checks for particular events (and code to run when those events show up) does in fact have hard-wired listeners for particular events. -DavidMcLean]

* Like I said above, it's a word/vocab issue.

* [Why not make it ''not'' a vocabulary issue, by defining unambiguous meanings of "listener" and "mainloop" in your specification? -DavidMcLean]

* In a language/tool-neutral way? I don't know how.

* [Similarly to how I did above, perhaps? -DavidMcLean]

* That's not language/tool-neutral.

* [Pray tell, to what language or tool is it specific? -DavidMcLean]


Incidentally, how is dumping the incoming events to an event table(s) which are then shared and processed by multiple apps an implementation of "mainloop"?

[How do the apps process events? In order to handle each event in their respective queues (which could be some view of the event table), they've got to loop over them, no? -DavidMcLean]

It's up to the app. They may only look at the queue once or a million times.

[If an app looks once, what happens if it doesn't find the event that it's looking for? Wouldn't it need to look again until that event shows up? -DavidMcLean]

Again, that's up to the app. It may send an email to the user saying, "Nothing new found. Go back to whatever you were doing." Maybe the ''user'' is the "mainloop" because they press the "See if anything in queue" button :-)

[Why would you want that? If you have an evented system, why base it on a manual-refreshing button when it could just as easily check for new stuff automatically? -DavidMcLean]

That's the app designer's question. One possible scenario is that too many people leave a monitoring app open on their desktop even when not using it such it floods a given server with queries.  The app developer may decided to solve that by adding an explicit "refresh" button to make sure an actual human wants the info, and thus cuts down on queries being processed.

[And it would be better to add a forced manual refresh button instead of disabling the refresh after a given interval without focus/activity/whatever and automatically restarting it when the app is focused again… why? -DavidMcLean]

An eyeball detector? Newer smartphones have it, but we shouldn't assume such would be available.

[No. Focus. The normal kind that we've had on computers ever since they started having more than one window. -DavidMcLean]

Window focus and eyeball focus are not necessarily the same thing. I have dual monitors. Sometimes I drag my Outlook window to the other monitor so that I can see if new emails come in WHILE working on some other app. Technically, Outlook won't have "window" focus while I am work on the other app.

[Well yes, obviously window and eyeball focus are different. (I wasn't claiming otherwise, however.) That workflow is indeed disrupted by having a focus timeout stopping the automatic refresh, ''but'' that workflow is also broken by there being no automatic refresh and a manual button being required to initiate refreshing. If the app does indeed need to stop refreshing to reduce load on the server, the automatic method with either focus or activity shutoff is obviously preferable---users who're active in the app get refreshing as wanted, users who are inactive don't put load on the server, and no users have to be concerned with an extra button. -DavidMcLean]

I think what you are suggesting is that that the app runs a refresh cycle for a fixed period of time (multiple looping) and then expires. Suppose the queue is typically updated only once every half hour anyhow, but the "status" query is resource intensive.  The half-hour is beyond a typical refresh period such that it's not worth implementing and thus skipped. But anyhow, why are you '''micromanaging app design''' here? Let the app designers decide that. Yes, sometimes they do stupid designs, but that's their problem; I'm not trying to reboot all of humanity here.

[No, the app refreshes automatically infinity times, provided the user is interacting with it. When the user stops interacting for a given interval, the refreshing is disabled; when the user continues to interact with the app, the refreshing restarts. There's a fixed time period given, but it "resets" every time the user does something in the app; since we're trying to prevent inactive instances of the app from loading the server, the optimal method is to detect inactive app instances using such an activity timeout (possibly combined with a window-focus one) and disable those instances' accessing of the server. As for that which you've deemed "micromanaging"? Because you're suggesting that your protocol should allow certain functionality to support particular use cases, and I believe that the use case you chose as an example is a bad design. Why design your protocol specifically to support stupid app designs? Why not encourage better ones? -DavidMcLean]

The scenario I gave was a monitoring application, such as monitoring servers, news events, suspicious product orders, etc. One does not necessarily "interact" with such in order to use (read) it.

[Certainly, but if you aren't able just to have it refresh automatically all the time always, since for example that overloads the server (as was your original premise here), it makes far more sense to disable and re-enable the refresh automatically based on activity than to eliminate automatic refresh entirely. Ideally, you'd start with only disabling refresh (or even just reducing the refresh rate) when the window is unfocused (or, better, if you have something like the HTML5 Page Visibility API, when the app specifically isn't visible). If that doesn't reduce load enough, disabling refresh after an inactivity timeout too may become necessary. Disposing of automatic refresh entirely isn't likely to be the right choice. -DavidMcLean]

I covered those situations already. The window focus/visibility issue was covered in the "Outlook" scenario. The only known technology that would work would be eyeball detection. We are wondering into irrelevant trivia fights anyhow. '''I am not going to dictate app implementation decisions. If you do, then we just disagree''' and should cut it here.

* [Yes, and I covered your coverage of those situations. In the Outlook scenario, your suggested design also entirely fails. Automatic refreshing that's temporarily disabled during inactivity is significantly preferable to 100% manual refreshing in many scenarios; can you identify any scenarios in which it's ''worse''? -DavidMcLean]

* You cannot detect "inactivity" in such situations. And you said "many scenarios". Many, but not necessarily all. I'm not going to try to crystal-ball ''all'' possible app/server design scenarios.

* [Correct---you can't accurately detect inactivity in that situation, hence disabling the automatic refresh on what appears to be inactivity may cause false positives. This is still preferable to having absolutely no automatic refresh, however, since in many usage scenarios (any situation in which keyboard inactivity corresponds to user inactivity, for example) it does indeed provide a better user experience. In the edge cases where automatic refresh will be false-positively disabled, the solution becomes essentially equivalent to and hence no worse than a totally-manual refresh system; since it's better in some situations and no worse in others, it's the better solution. -DavidMcLean]

* Whatever; I choose to let such tradeoffs be the app builders' call. I don't believe in your or my ability to forecast all future needs and tradeoffs with sufficient accuracy.

* [There is no tradeoff: A mandatorily-manual refresh is absolutely worse than an automatic one in ''all scenarios''. Even if you can't just have automatic refresh running at all times, for reasons of limited resources, setting it up to limit based on inactivity will, ''at the very worst'', give you a program that is equivalent to the manual refresh one and at all other times a better program. -DavidMcLean]

* I told you already, we may not be able to detect human activity in some cases.  And manual refresh is simpler to implement and more consistent than your "sometimes" scenario. I'd rather ALWAYS manually refresh than have the machine sometimes refresh for me and sometimes not because then I don't have to think about whether a refresh happened or not. A reflex is simpler to most WetWare than check-first-then-react.

* [It ''doesn't matter'' that you can't detect human activity in some cases, because when you can't detect human activity the system degrades to be exactly the same thing as a manually-refreshed system. As for the question of inconsistency, there's nothing precluding including a refresh button in the interface, even if it's set up to refresh automatically. Many automatically-refreshed apps are already known to do this, in fact; consider popular smartphone apps with their "swipe to refresh" convention. Since your always-manual solution ''absolutely needs'' a refresh button included on the UI, the cost of adding such a button in the automatic setup is mitigated. Also, such a button could act as visual feedback: Have the "circular arrow" of refresh spin around whenever a refresh is in progress, for example, then grey it out when inactivity is detected and hence automatic refresh is disabled (ungrey it out when activity happens again, obviously). Under this design, users reap the benefit of an automated refresh process, as well as the control afforded by a manual process. It's definitely a little harder to implement than a purely-manual system, but isn't it our job as developers to implement whatever gives the best user experience (subject to technical constraints, of course)? -DavidMcLean]

* You are micro-managing the app UI design. I don't want to dictate specifics of app UI design because every designer has a different design philosophy and only their way is right and everybody else is stupid and wrong because only they "get" Zammagamma Newthing.

* [Maybe. I'll stop designing and instead point out the flawed assumption that led to the whole discussion: If the user clicks a "refresh" button, that doesn't make the user the eventing mainloop; even assuming the "master" event queue exists on a remote server, the logical result of clicking refresh is to ''mirror'' the queue or parts of it to the local app, which the local mainloop would then process. -DavidMcLean]


In general, the "looper" (checker, try-er, poller, etc.) may be a couple or more servers or layers "down the road" such that the Eventop kit has no control over it.

[Why? What use cases mandate that Eventop not include its own mainloop, other than the scenario we've already discussed that is handled much better in another way? Remember, we don't want to encourage bad app designs. -DavidMcLean]

The refreshing/recheck control mechanism may be on a different server or system. The client may be making such decisions, for example, but Eventop is being used on the server side.

* [Uh… why aren't there events on the client-side too? And if you're trying to throttle refresh of the Eventop queue from specific clients, surely those specific clients are accessing (and caching locally) the Eventop queue, no? -DavidMcLean]

* There may well be events on the client-side, but it likely would be outside of the scope/control of Eventop on the server side. I'm letting the developer or org split up the parts of their apps how they see fit and will not use a standard to try to ram event coding or partitioning styles down their throats. I agree that ''somewhere'' in the system stack there is likely something akin to a "mainloop", but it may be distant from or outside of the influence of the Eventop installation.

* [Aside from possible legacy issues like "there's no Eventop implementation for what we have to use on the client-side", why wouldn't you want to set up the eventing of your system such that it all conforms to a single event spec, assuming such a spec exists? What benefits are afforded from encoding events on one part of an application inconsistently with another part of the app, when a specification exists to standardise said events? -DavidMcLean]

* The portions could be split up for multiple reasons. I don't want the standard to concern itself with such questions. If you do, make your own Eventbloat standard.

* [Technical standards that seek to standardise something with existing and conflicting implementations ''absolutely'' should be concerned with how the standardised version handles the non-standardised version. Ideally, an eventing standard such as Eventop should present some mechanism whereby non-standard events can be "assimilated" into Eventop events and henceforth exhibit all the guarantees the standard provides. And I hardly think it's bloat for a standard to, y'know, actually ''standardise things''. -DavidMcLean]

And I've decided not to try to shape app design with the draft standard. If you want to make another standard that does try to shape app design, be my guest. I'll ignore it, though. '''I'm out to make a tool, not spank app designers.'''

* [Choosing features to include in your specification should absolutely be shaped to some extent by the potential use cases of those features. (You've argued this exact point before, actually.) If the only possible use cases of a feature are inferior software, why have the feature? It's common wisdom to start new features at "minus 100 points", only including them at all if they can overcome that starting bias. -DavidMcLean]

* What "inferior software" are you referring to? You mean existing systems and tools? I'm not working on the assumption we can reboot the Earth.

* [No, I'm referring in particular to software built under the spec that exhibits inferior design to similar software built under the same spec. The specific example I'd had in mind was the idea of requiring the user to refresh the app manually, despite automatic refresh providing a superior user experience in practically every scenario, even in the presence of resource limitations making an inactivity-based-refresh-disabling necessary. The features necessary to permit manual refreshing don't earn many points from their starting -100, unless other use cases can be presented in which there are not far better ways to solve the problem. -DavidMcLean]

* What's this point system? Like I said, sample reference client code can be provided for common languages or OS's. I don't want it to get bogged down with client-side standards. That's a whole new cave.

* [The point system would be this one: http://blogs.msdn.com/b/ericgu/archive/2004/01/12/57985.aspx Isn't your event standard supposed to be applicable on the client-side? It was originally proposed in response to a problem (search-as-you-type) that necessarily must be at least minimally addressed on the client-side; if it's server-side only, what do we use for events on the client-side? Why not be consistent? -DavidMcLean]

* I'm not against client-side stuff, I'm just not focusing on client-side standards. If you want to propose client-side standards, be my guest. But, I may not participate.

* [If you aren't focusing on client-side standards (or general standards that'll apply on both client and server), why did you propose this standard in response to a need that can only feasibly be addressed on the client-side? -DavidMcLean]

* Only? No. It's a mix of server and client, but the meat of the app would be server side.

* [Yes, only. The components of a search-as-you-type system I initially presented are the components which ''must'' exist on the client-side, not the server-side, for a feasible solution. Sure, the server must do its part retrieving actual search results, but it is not possible to offload any more of the processing from client-side to server-side without sacrificing the search-as-you-type nature. -DavidMcLean]

* Your statement "why did you propose this standard in response to a need that can only feasibly be addressed on the client-side?" Is false because it can be feasibly addressed by having parts on BOTH sides. 

* [Not in the sense I meant by the original statement. The part that packages up queries from the user, sends them to the server, and receives corresponding search results can only feasibly be implemented client-side, and that's the part I presented in the original topic, NaturalEventSyntax. You responded to this by presenting this Eventop standard; unless it's applicable to the client-side, it ''cannot'' take the place of the part of a search-as-you-type system I originally presented and therefore cannot address the original need. The fact that the server does other work (fetching actual search results from a database/flatfile/in-memory store/whatever) doesn't impact the fact that the actual problem of search-as-you-type is addressed (and solely addressed, at that) by the ''client-side'' code; the search-result retrieval on the server is merely solving the problem of search, which would also be part of a "manual" search-button-style searchable system. -DavidMcLean]

* I want something that works with Fortran, Cobol, BrainFuck, BASIC, Footran, and FU++. You exclude all that if you hard-tie it to a client standard.

* [Right, you would. So don't hard-tie it to a client standard. Make it a client standard in its own right, as well as a server standard. Make your standard a general design for evented applications and infrastructure that's equally applicable to running in a local client and a remote server. -DavidMcLean]

* Make a great client standard that works in a wide variety of languages and OS's and I'll hop on board.

* [Okay. I like the way JavaScript implements events, so let's start with that. Then let's notice that JS is supported in all major Web browsers on all major operating systems, and let's subsequently notice that every language on http://altjs.org/ intrinsically works with the same event model and additionally runs on all the same platforms. There we go, standard made. -DavidMcLean]

* You are assuming we run everything in browsers? I should have also said existing apps, not just existing languages. And by the way, Fuck JavaScript!

* [I'm not assuming we run everything in browsers, merely that browsers are a place where things can run and that they work on a wide variety of operating systems. Imposing a standard over an existing app that didn't follow the standard is almost certainly going to have friction, but this is no fault of the standard. And by the way, is your rudeness necessary? -DavidMcLean]


From experience, shops are often a hodge-podge of different technologies and tools, and the organizational dynamics prevent a given developer from having control to design or change a good portion of it. Sometimes you have to use "push" when "pull" would be better, and vice verse. I'm not going to AssumeCanOpener, and will try to practice SeparationOfConcerns with regard to tool-sets.

[A specification that seeks to formalise events in the event-driven programming sense should be expected to formalise the meanings of basic eventing concepts like the "event", the "mainloop", and the "listener". It's not mixing unrelated concerns and therefore not violating SeparationOfConcerns to specify these primitives all together, because frankly they are all essentially meaningless in isolation; an event as defined by Eventop without some sort of handler is no different from any other DictionaryDataStructure, for example. -DavidMcLean]

Then you are dictating architecture of the tool that uses the event protocol, which is both unrealistic and overreach.

[No, you aren't. You're defining the architecture of your event protocol. -DavidMcLean]

------

'''More Specs'''

* As a convention/recommendation, paths that start with "sys_" are reserved for predefined uses. Typical content should start with "data."

** [Why use a "sys_" prefix when a dot denotes a nested object? Wouldn't "sys.", i.e., reserving only the single object "sys" at the top level of the structure for predefined uses, make more sense? -DavidMcLean]

** I did that for a reason, but I forgot it. Should have asked a few months ago.

* Language or tool-specific drivers should provide a way to be notified when a new event arrives, and be able to "peek" into the incoming event queue, if it's appropriate to the tool. Typically, sample/reference API's would be provided. An OOP version, a HOF version, and a poll-loop version could be provided, for example.
----
JuneThirteen