Change how Meteor executes Async code #11505
Replies: 57 comments 85 replies
-
Thanks a lot for this overview @filipenevola ! |
Beta Was this translation helpful? Give feedback.
-
I'm working in a few changes to be possible to use mongo package already using the standard Node.js async at least in the API. I'll update this discussion when I have code ready to be shared. The idea of this first phase is to provide the API in a way that removing Fibers later the new API will be exactly the same so developers don't need to migrate the mongo use cases they are going to add from this point forward. I'm still trying different ideas for the final API so I can't share much at this point but my ideal API would be something like: // brainstorm :) we don't know the final API yet
const LinksCollection = await createCollection('links');
const links = await LinksCollection.find(); In this process I'm finding a few challenges that I would like to describe at least one briefly here. For example, we can't just change all the functions to async functions because in many cases the function is used more like a class, creating new instances using const collectionDefinition = new Mongo.Collection('links');
const LinksCollection = await collectionDefinition.asyncInit(); I'm not mentioning here the natural challenge of converting to That is all for now, the motivation of this message is just to provide more visibility for the community. |
Beta Was this translation helpful? Give feedback.
-
My main concern is how this will affect tracker, method stubs, and any other api's used on the client that requires knowing the context code is run within, since the current way it does that requires the code to be synchronous. Does one of the parts of the plan include looking into this?
This gave us the equivalent of top-level await where we could have async code (using fibers) in the top-level of a file, and Meteor would wait for it to finish before loading additional files. Do we have any plans to implement top-level await in Meteor's module system? |
Beta Was this translation helpful? Give feedback.
-
After hitting the wall this morning with async and Meteor 2.3 (thanks @filipenevola for the links!) this topic really catched my attention.
As this is all a lot of work, I was wondering if before building these new packages, a modern "skeleton" for these "async" packages could be defined:
I'm not sure if you actually run into these problems at all in practice, new Mongo.Collection doesn't have to do anything asynchronously at all, so this will be perfectly possible I guess?
|
Beta Was this translation helpful? Give feedback.
-
Sorry for interrupting this discussion. Is this package helpful as an alternative? |
Beta Was this translation helpful? Give feedback.
-
For my application code, I assume this means that pretty much all calls to the Mongo and Accounts package will become async calls instead. The idea above to create alternate packages with new api signatures (e.g. mongo-async) sounds promising in theory but I wonder what the ripple effects will be. Should every meteor package that interacts with mongo or other outside services create an alternate "async" package with new signatures and then piece by piece move the implementation from the "sync" (deprecated) version to the async version? Below are the obvious ones I could see that I use. Some will probably need forking since they have been abandoned.
Just trying to wrap my head around the order of execution here outside of the core meteor packages |
Beta Was this translation helpful? Give feedback.
-
I just published a draft PR for Mongo Package Async API #11605 |
Beta Was this translation helpful? Give feedback.
-
@filipenevola, I was happy to watch your Meteor Impact talk about returning to the roots and - while I understand the arguments - I feel like eventual remove of If there is no other future-proof solution except let x = 0
const inc = () => { x += 1 } into this (pseudocode) let x = 0
const inc = () => { x += 1; updateDOM() } I don't know to what extent it's technically possible, but maybe Meteor could track the usage of its exposed async API, like Mongo operations, and add Generally, I think that adopting this approach could be a huge step in the "less code, more power" direction you mentioned in your talk. |
Beta Was this translation helpful? Give feedback.
-
Any progress on this issue? We are getting more and more people asking for MeteorJS support from APM and are looking forward to support it as soon as the switch to async/await is done. |
Beta Was this translation helpful? Give feedback.
-
I’d be happy to help migrate the dependent code, esp all the code in accounts_base, but I’m unsure how to convert all methods that invoke mongo methods to be async without the IDE support that e.g. moving to Typescript would give. Or are there tools that help detect invocations that lack the needed await? As an example, take destroyToken in accounts_server. Since it invokes this.users.update it will need to become async (this.users.updateAsync ? not sure what the plan is here) and all invocations will need to adapt and also become async, which then ripples out via the call chain. What is the best way forward to convert javascript code to become async by analysing all call chains? Should there then be an intermediate sync version that just invokes Promise.await(destroyTokenAsync(...args)) and logs a warning? What is the execution plan here? |
Beta Was this translation helpful? Give feedback.
-
Note that |
Beta Was this translation helpful? Give feedback.
-
Just ran into the issue with wasm today and found #11143 in the process of debugging. I might consider putting some upwork or internal development resources towards helping make progress. |
Beta Was this translation helpful? Give feedback.
-
Hey everyone, we finally have updates here! I just want to reassure we are continuing the work to change the way Meteor executes async code. This is a gradual work with many pieces and many people involved. Please read the initial plan and comments on this discussion to understand the context of this work. Plan PhasesPhase 1 - Node.js 14 [FINISHED]Phase 1.1 - Fibers running on Node.js 16 [CANCELED]Phase 2 - Make Fibers usage in Meteor server application runtime optional [CURRENT]Steps to complete this phase:
Expected Result
Phase 3 - Make Fibers usage in Meteor tools optional
Phase 4 - Declare Meteor packages using Fibers in maintenance mode
At the moment, the great people directly involved in this work are @filipenevola (Quave), who will lead the plan and bring another dev from Quave, @radekmie from Vazco, @denihs (and myself) from Meteor, and @zodern from Monti APM. We hope to make more progress in the mentioned steps in the next weeks. We really appreciate the recent feedback about this work from people like @StorytellerCZ, @menewman, @jankapunkt, @rochdev, @perbergland, @Julusian, and many others. I think if we centralize the discussion here and in the specific PRs that will help everyone to follow the progress. Please reach out to me if you see something where you could collaborate. If you think we are not considering something important, please comment here. This work is our main focus now, and it will be in the next couple of months. |
Beta Was this translation helpful? Give feedback.
-
I think we'll need to adopt some tools for working without Fibers, but in async mode. For example, Meteor uses Maybe we can add a new label like "Remove Fibers" and add a separate issue for such things like above? |
Beta Was this translation helpful? Give feedback.
-
Hello, everyone! Some updates about the top level await support. To refresh our memory a bit: the top level await support is required to be able to wait for the To give more context, let's see a bit of how Meteor's build system works: before evaluating any module/code, we use Babel to check for syntax errors. This means that just swapping the current You can check how Meteor bundles the code by checking the
So the steps planned to support TLA are:
For the 2nd point, the 2nd approach is the most promising one, as from what we tested it works. This means that the example we gave above would be changed to:
This is our current plan. If you guys have any suggestions/recommendations, please let me know your thoughts. |
Beta Was this translation helpful? Give feedback.
-
Hello everyone, quick bi-weekly update: Here are the latest updates on the successfully completed tasks:
Currently, we are actively working on the following tasks:
You can follow the updates on our Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
Hello everyone, quick bi-weekly update: Here are the latest updates on the successfully completed tasks:
Currently, we are actively working on the following tasks:
You can follow the updates on our Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is a small update: Some tasks we've been working on it:
Our plan for the next week is to release one more Alpha, which will include changes like this one and this one. After that, we'll address the issue Unhandled Rejection when fails to bundle the app as well as more issues our community is reporting, like this one. See you soon for more updates! You can follow the updates on our Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is another update for you! In the past few weeks, we addressed issues like this, and we have a new Alpha, We now have a new document where you can see the main tasks we need to finish until we have a final release for version 3.0. You can check that out here. Also, for the next two weeks, the main tasks we'll be addressing are Unhandled Rejection when fails to bundle the app and Review Meteor.call, Meteor.callAsync, Meteor.apply, Meteor.applyAsync so we can release a final Alpha version. See you soon for more updates! You can follow the updates on our overview doc, Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is a quick update for you on what we've been doing in the past few days! Our We've also been working with HUGE thanks to our community on this one. We had a lot of feedback and participation on it. And special thanks to @zodern, who created this code. You can check their post about it here. We also started working on our Migration guide and updating our Docs for version 3. Unfortunately, we had some drawbacks in the past couple of weeks, but we're still confident that we can have a Beta version by the end of the year. So stay tuned, and see you soon for more updates! You can follow the updates on our overview doc, Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is a quick update on what has happened in the last month! We launched the first Beta version 🎉!
Meaning that we finished the work with our Something worth mentioning again is that this version is already running with Node 20! Also, we're working hard to finish migrating and updating our docs and guides. You'll hear more about it soon. We're now moving forward with the Beta, and your help with tests and feedback will be of great value! So stay tuned, and see you soon for more updates! You can follow the updates on our overview doc , Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is another update on what we've been working on in the past few weeks. We fixed an issue where Symbol.asyncIterator on Cursor was not working. We updated the Reify package inside the core where it wasn't updated yet. Also worked on issues like: Insert not working for collection created of the fly. Right now, we are working on:
We also moved forward with the docs, and it's looking really good. Here is a sneak peek: We are planning on releasing a Beta 1 pretty soon, so stay tuned! You can follow the updates on our overview doc , and on this Github PR. |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! Here is one more update. We release one more Beta, this one being beta.4. It includes:
We're already working on the tasks for the next one, so stay tuned! You can follow the updates on our overview doc , and on this Github PR. |
Beta Was this translation helpful? Give feedback.
-
I can see in the PR that |
Beta Was this translation helpful? Give feedback.
-
@denihs @Grubba27, in terms of migration, are we still expecting any of the latest async changes for 3.x branch to be backported to 2.x branch? |
Beta Was this translation helpful? Give feedback.
-
I've just started testing our applications on beta.6 and I've encountered an inconsistency regarding the documentation. Specifically, it's about the handling of the |
Beta Was this translation helpful? Give feedback.
-
Updates about the first Meteor 3.0 RC Release: Due to some roadblocks, our ETA for the RC Release has been postponed to 19th April. These are the tasks for the first RC:
Estimated Release Dates
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the first Release Candidate for Meteor 3.0 🤩🥳 The priority of the next RC will be to review and fix issues with Cordova's experience. Next Releases:
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the Release for 2.16 beta 🤩🥳 Tasks and fixes for the next Meteor 3.0 RC:
Next Releases:
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the Release for Meteor 3.0-rc.1 🤩🥳 We also have the Meteor 2.16-rc.0 🤩🥳 Tasks and fixes for the next 3.0 RC:
Next Releases:
|
Beta Was this translation helpful? Give feedback.
-
Plan to change how Meteor executes Async code
Related PRs: #11605
Goal
The goal of this discussion is to define a plan together with the community on how to proceed with
async
execution in Meteor.js.We, Meteor Software, are proposing a plan but we are open to ideas and improvements on this plan.
How the community can help?
Context
Fibers was the choice for Meteor as at the time there was no async/await standard available, you can understand more this choice by watching this talk (slides).
Meteor uses Fibers to implement async behavior when you use Fibers directly or even when you use async/await keywords or a Promise. This enabled Meteor to use new patterns very early.
In recent years some conflicts between Node.js and Fibers started to appear and then the need for a different approach.
Examples of conflicts:
Our desired approach is to follow what was adopted as the standard for async code in Node.js but this has a fundamental challenge as the new standard doesn't allow the usage of
await
keyword unless we are inside anasync
function.The challenge here is that in some cases this requirement is going to affect many calls layers in the code. This change will affect Meteor tools, Meteor core packages, and user application code as well.
The good news here is that most of the packages using Fibers don't expose its usage so it's not going to affect user's code.
So far we have not identified many cases where user application code is going to be affect and we also have plans to automatize and minimize the impact of this change, probably the common use case will be around MongoDB calls that would need to be awaited.
Next we propose a few phases to rollout these changes, the idea is to minimize as much as possible breaking changes and also to provide ways to give plenty of time for developers to adapt their code.
Plan Phases
First the goal is to make Meteor compatible with Node.js LTS for as long as possible using Fibers and later remove Fibers as a dependency but without the pressure of Meteor not working with Node.js LTS.
Phase 1 - Node.js 14 [FINISHED]
This phase was completed on Meteor 2.3
Expected Result
Meteor working with LTS Node.js using Fibers until April 2023. We do not expect to fix possible conflicts with async_hooks and WebAssembly at this point.
Phase 1.1 - Fibers running on Node.js 16 (optional)
We could extend for how long Meteor will be running with Fibers on Node.js LTS versions by supporting Node.js 16 as well, for that we have a few challenges that we need to address:
It's important to notice that this phase it not required, it could be executed in parallel with the Phase 2 or even skipped.
Expected Result
Meteor working using Fibers with Node.js 16. We do not expect to fix possible conflicts with async_hooks and WebAssembly at this point.
Phase 2 - Make Fibers usage in Meteor server application runtime optional
Meteor uses Fibers on server runtime and also in the Meteor tools. The goal of this phase is to make Meteor server application runtime possible without using any Fibers.
The plan is to not force all the applications to migrate to this new way of running async code altogether. We are planning in a way where applications could change a few packages each time.
Of course, the consequence of having a few packages still relying on Fibers is that while there is at least one package in an application relying on Fibers the entire application will need to run in the version of Meteor where Fibers is still accepted and compatible.
The good news here is that most of the packages using Fibers don't expose its usage so it's not going to affect user's code.
Here are the steps to complete this phase:
boot.js
uses a Fibers to start the server. We already have some early results removing thisboot.js
usage here. In this commit it's possible to generate the boilarplates and run the Meteor server without the global Fiber.Promise
implementation of Meteor that uses Fibers toPromiseFiber
.PromiseFiber
.bindEnvironment
andwrapAsync
in external async calls.async
andawait
are used in the application code.Packages using Fibers
section below for details.-async
in the name providing the same functionality using standard async.mongo-async
package would provide Meteor collections where is necessary to await the calls to MongoDB.-async
package when creating new projects withmeteor create
.Expected Result
Meteor apps will be compatible with
async_hooks
andWebAssembly
and any other Node.js modules as Meteor apps are not going to be using any non-standard technology anymore in application code.This phase is not going to require a full migration in a single step. Applications can migrate group of packages in incremental steps, guides are going to be provided as soon as a package is ready to be replaced.
With this phase completed the applications should start to use the new
-async
packages and adoptasync
andawait
keywords where it's necessary in the new packages.Also package authors from the community should be able to have many examples on how to migrate their packages if they are relying on Fibers directly. Application code that is using Fibers directly should also adapt to regular
async
andawait
keywords instead.At this point Meteor tools will still using Fibers but this is not going to affect application code in anyway but this would prevent Meteor to migrate to a newer version of Node.js that is not compatible with Fibers, until the next phase is completed.
Phase 3 - Make Fibers usage in Meteor tools optional
Meteor tools use Fibers to execute async code. We want to remove this dependency as well. It's important to understand that this dependency does not cause problems in application code but as Meteor tools are an important part of Meteor platform we also want to remove Fibers from it. Otherwise this could keep locking Meteor in an old version of Node.js because of Fibers' conflicts.
Here are the steps to complete this phase:
Tools using Fibers
section below for details.Expected Result
Meteor will be compatible with async_hooks and WebAssembly and any other Node.js modules not only in application code but also in Meteor tools.
Phase 4 - Declare Meteor packages using Fibers in maintenance mode
At this point Meteor is no longer dependent of Fibers so we would not need to keep adding features and updating core packages depending on Fibers as they already have a new version without Fibers.
This doesn't mean that we are not going to fix bugs in these packages but they are going to be considered
maintenance only
. If new features are implemented they are going to be added only in the-async
version.Expected Result
Meteor continue to evolve without having to provide new features to two versions of the same package.
This also is going to provide stability for applications that are no longer receiving new features and probably don't need to have latest features.
We are open to feedback on this plan and we will be glad to answer any questions.
See below two sections listing usages of
Fiber
,Future
andPromise.await
in packages and later in Meteor tools.Packages using Fibers
wrapYieldingFiberMethods
1Tools using Fibers
* maybe we can remove this usage instead of migrate
** it's probably a very important usage
Link to the previous Meteor Feature Request about this.
Beta Was this translation helpful? Give feedback.
All reactions