[RFC] Module Tools Architecture Evolution

[10Derozan]

Background

It has been half a year since the official release of the V2 version of Modern.js Module, and the landing projects have reached over 2500 in bytedance. In the long run, the module project has stabilized and the overall function has basically been able to meet the needs of different businesses. During the landing time, many problems were still exposed. At the same time, the quality of the project and overall performance were also seriously degraded. In order to ensure the overall health of the module project and the planning to reduce its investment in the future , it is necessary to carry out a reconstruction of the overall architecture., to improve the robustness of the project.

Target

• Reduce engineering maintenance costs:
  • Workflow: to reuse the Modern.js repository workflow
  • Changeset & release note: Use the same version number and release information as @modern-js scope
• Improve project quality:
  • Test: Remove duplicate tests to improve coverage
  • Bundle size: Reduce package size and dependency size
• Improve build speed: serious deterioration compared to the initial stage of the project
• Improve the quality of the warehouse: most of the modules in the warehouse are constructed through module engineering, and the quality and construction speed of the entire warehouse are closely related to the module engineering.

Core issues

Do we need the Libuild tool?

If you're unfamiliar with libuild, this is the repository.
From the development of nearly a year, we no longer need Libuild. It is positioned within the team as a tool that is not perceived by users, so there is no separate doc, no publicity, no users. When the Libuild project started, we positioned it as an independent library building tool driven by Esbuild at the bottom . We create the Libuild repository, which has its own configuration, its own CLI commands, and its own plug-in system.
Below I will break down all the features of Libuild in turn and integrate them into the Modern.js Module.

Major changes

Adjust

Plug-in system

Let's first take a look at the build hooks provided by the Modern.js Module to users. This is link
This has nothing to do with Libuild, let's take a look at Libuild's design:

By bridging with Esbuild , Libuild is involved in the entire build process of Esbuild.
So when we remove Libuild, we can add the new hooks to the Modern.js Module.
Only load + transform + processAsset (renderChunk) is expected to be retained, why resolve is no longer retained here, which will be mentioned below.
The original Libuild plugin is converted into a module engineering plugin, no longer a separate layer.

Boundary case:

What if the user only wants to add transform hooks to the ESM product?
Need to maintain a builder instance, this cost is relatively high, as done in libuild
(If passed parameters, the API design of this hooks will be strange)

Remove

enhanced-resolve

In Libuild, enhanced-resolve takes over the entire OnResolve, we plan to remove it, keep only one instance, and only use it in the plugin that handles styles. There are several main considerations here:

• Esbuild will continue to execute the OnResolve callback without setting the path in the OnResolve hook, which means that our previous construction of external and sideEffects can continue to work.
• Esbuild built-in resolve has better performance, and also supports some important configurable items, such as mainFields , alias .

CLI, Configuration, ...

Calling the Esbuild API directly means that many previous builds will be outlawed, including but not limited to:

• CLI : Remove the 'libuild ' command, removing all cli Options.
• Config: Libuild configuration is no longer maintained separately, including configuration file processing, schema verification, configuration standardization, etc.
• Env : no longer reads environment variables to enable/disable some experimental behavior.

Merge

Watch

Rollup and Tsc do not provide rebuild APIs like Esbuild , which means that after the merger, we cannot uniformly manage the incremental build state through a watcher in the watch state, and the initial plan remains the same.

Error & Log

#4325

Npm package merge

All libuild-related packages are no longer maintained, mainly the following six:

• @modern-js/libuild
• @modern-js/libuild-plugin-babel
• @modern-js/libuild-plugin-babel -preset
• @modern-js/libuild-plugin-node-polyfill
• @modern-js/libuild-plugin-swc
• @modern-js/plugin-svgr
  The plugins in the module project have covered all libuild plugins. They just need to adjust the implementation.

Configuration optimization

There are also a few external users complaining #4270

The most common problems are :
Why does the product directory structure change after I configure buildConfig ?
Most users will not look at such a warning:

Although the doc is detailed enough and easy to understand, such behavior is really not "intuitive".
Configuration optimization is mainly through the following two points:

1. Use buildConfig as a higher priority configuration to merge with buildPreset .
2. Format Support Array

Steps

I will move libuild source code, tests and workflow to this repository, and then, remove all features about libuild and add it to module tools.

[chenjiahan]

Does this mean that the libuild repo will be archived after the migration?

[10Derozan]

Yes, it will be archived.

[chenjiahan]

get~

[10Derozan]

In the first phase, I have migrate the source code and test to this repo #4368
In the next stage, I will refactor the CLI information and logger according to #4325

[10Derozan]

In #4651 , I have finished the task.
And it released in 2.36.0