The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
I'm not sure this is true anymore even in this case - with multicore processors, cache dependency, and pervasive virtualization, the speed of your program can be significantly affected by what else is running on the box. Remember that the x86 itself has a JIT on the chip, converting x86 machine code to whatever microcode the processor uses. When I was at Google, they provided special dedicated machines for benchmarking, with custom run-times that disabled a lot of the containerization/virtualization features, and there was still a lot of noise in benchmark times.
That's certainly true and I hope I didn't give the impression that I thought this was a binary situation. It's just that code which runs in a JITed environment, particularly one like the browser JavaScript runtimes which aren't even versioned, has an even wider exposure to skew. Everything running on a multiprocess/user operating system is exposed to resource contention but e.g. your JavaScript code also has to worry about things like Chrome disabling optimizations anywhere you use try/catch.
> The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
While the release of a new AOT compiler version might not affect the performance characteristics of your application without an active choice to recompile, the release of updates to any OS components your application interacts with, may, in much the same way that a new web browser or JVM version would. The only way you get immunity from that is to bundle the whole software stack running on the bare metal as your "app", rather than relying on lower-level software that may change independently.
Right – that's why I didn't say “known compiler and standard library”. That said, it's a difficult comparison because OS updates tend to be more conservative and are definitely less frequent than browser updates, AOT apps in lower-level languages like C/C++ are going to manage a higher percentage of their core data structures because they're pinning something like Boost whereas a JavaScript developer just gets whatever version of Object/Array/etc. is available, and there's not really a concept of a pinned release: if you install Windows apps, you'll pull in major versions of things like .NET or the JVM but there's no analogous practice of pinning a particular browser version. The closest we've come was Internet Explorer's compatibility modes which they've deprecated and which never changed the core JavaScript engine's performance – e.g. IE8-running-as-IE7 still applied the IE8 JavaScript engine's validation and optimizations.
It does but the most common JITed language in existence is JavaScript, where the uncertainty is especially pronounced. For example, Python isn't [traditionally] a JITed language but most Python developers know that their code will run on CPython, which is pretty stable and you have a rough idea of which versions it'll run on, whereas something running in a browser can see huge shifts in what is an expensive or cheap operation from user to user over the course of a single day, much less release to release.
Remember when WebKit started JITing regular expressions? That changed the benchmark situation so much that some people called it cheating and not a single line of user code changed. That's not impossible in other environments – I'm sure there's been a case where a shared library update made a big difference – but it's far less common in my experience.
