Dependency Manager — An Approach to Multiple Repositories in Flutter

As a code base begins to grow, in an environment where you are developing multiple applications, it is normal to have some dependencies that are shared across projects. This means that if we don’t want to copy and paste the same functionality, we need to create individual packages that can be used by multiple projects, an approach that has be discussed in this article - Modular Flutter Apps — Design and Considerations.

In one company I worked with, we were using multiple repositories in our development pipelines, in order to create multiple applications with Flutter Web. However this approach created a new problem- if each package can have different versions, how can we version them and reuse them in all other packages and projects?

In this article we will discuss some of the possible approaches to how we can manage dependencies across multiple repositories and present one solution we use at our company. For this purpose, we will consider an application called app_a that will depend on 3 libraries - lib_a, lib_b and lib_c. Furthermore, both lib_a and lib_b depend on lib_c.

Dependencies graph

Approach 1: Path Dependencies

The easiest approach to set up a project is to create a project folder in which we place all the libraries and applications, so that we can reference them by relative path. For this to work, we will need to create a strict folder structure, so that we can reference each project:

.
├── app_a
├── lib_a
├── lib_b
└── lib_c

Now we can add the dependencies on pubspec.yaml using the path parameter:

name: app_a
version: 1.0.0+1

dependencies:
  # Other dependencies

  lib_a:
    path: ../lib_a

  lib_b:
    path: ../lib_b

  lib_c:
    path: ../lib_c

And the same would apply to the lib_a and lib_b:

name: lib_a
version: 1.0.0+1

dependencies:
  # Other dependencies
  lib_c:
    path: ../lib_c

If we run flutter pub get, we can successfully get the dependencies and if we run our app via flutter run -d chrome it will run our application. However, this approach has several problems when we use it in an environment where we are working in a team.

First, this approach means that each developer will have to update to the correct version of each library when developing. This can lead to situations where two developers are developing on app_a can be using different versions of lib_c, leading not only to conflicts but also to other development issues since each developer will be running a different version of the same application.

In summary, this solution may fail since we are not able to stipulate strict versioning of the libraries. And for this reason, we explore our next proposition - submodules.

Approach 2: Submodules

With submodules we are able to use the benefits of a path dependency alongside a strict version reference, since each submodule can point to a specific git reference.

When adding a submodule, we are adding a git repository to our parent repository, in which each have its own commit tree. We can stipulate then what is the reference - commit, branch or tag - that we are going to use for each submodule, meaning that we can have developers using the same library version.

Let’s start by adding the git submodule to lib_a and lib_b.

git submodule add https://github.com/Vanethos/dm_lib_c.git

This creates a new folder called dm_lib_c in the root of our libraries and adds a .gitmodules file. In this file, we can specify the git branch or reference that we want to use, for this example we will use the master branch for lib_c

[submodule “dm_lib_c”]
    path = dm_lib_c
    url = https://github.com/Vanethos/dm_lib_c.git
    branch = master

Since this submodule is added as a new path, we can reference it in the pubspec.yaml file as a path reference:

# ...
dependencies:
  # ...
  lib_c:
    path: ./dm_lib_c

We use the same methodology in app_a to add lib_a, lib_b and lib_c as dependencies, and for each one we specify the correct branch that we want to use in the gitmodules file:

[submodule "dm_lib_a"]
    path = dm_lib_a
    url = https://github.com/Vanethos/dm_lib_a.git
    branch = submodules
[submodule "dm_lib_b"]
    path = dm_lib_b
    url = https://github.com/Vanethos/dm_lib_b.git
    branch = submodules
[submodule "dm_lib_c"]
    path = dm_lib_c
    url = https://github.com/Vanethos/dm_lib_c.git
    branch = master

And update the pubsec.yaml file accordingly:

name: app_a
version: 1.0.0+1

dependencies:
  #...

  lib_a:
    path: ./dm_lib_a

  lib_b:
    path: ./dm_lib_b

  lib_c:
    path: ./dm_lib_c

Finally, we can run flutter pub get in our app_a to retrieve the dependencies. However, when we do so we are presented with the following error message:

Because app_a depends on lib_b from path which depends on lib_c from path, lib_c from path dm_lib_b/dm_lib_c is required.
So, because app_a depends on lib_c from path dm_lib_c, version solving failed.
Running "flutter pub get" in app_a...                                   
pub get failed (1; So, because app_a depends on lib_c from path dm_lib_c, version solving failed.)

So what is happening here?

When pubspec tries to resolve its dependencies it has a rule: all dependencies must have the same version, path or git reference. This is why when adding a new dependency to our projects, we usually add it with the ^ symbol - this allows pubspec to find an appropriate version the dependency in a strict set of rules that you can find in the official documentation.

In reality, when we are adding submodules to each library, we are specifying different paths to our dependencies. Let’s take a closer look at our tree when we add the submodules on app_a:

.
├── dm_lib_a
│   └── dm_lib_c
├── dm_lib_b
│   └── dm_lib_c
└── dm_lib_c

app_a will have a dependency on lib_c with the path /dm_lib_c, however, lib_a will have the path dependency on lib_c as /dm_lib_a/dm_lib_c, leading to conflicts on pubspec since it cannot decide which version to use.

Despite the fact that initially this solution sounded like a good approach since we could define a strict reference for each library, it falls short since it creates issues with the dependencies management of pubspec.

However, this approach gives us an idea to how we can solve our issues - using git references for each package. Let’s explore that approach.

Approach 3: Using git references

As seen in the Dart dependencies documentation, we can add a dependency to pubspec.yaml via a git reference:

dependencies:
  kittens:
    git:
      url: git@github.com:munificent/kittens.git
      ref: some-branch

The ref can be any git reference - a tag, a branch or a commit. In the case of this example, we are using branch references to identify our app, so we can use the following to identify lib_c in the pubspec.yaml of lib_a and lib_b:

name: lib_a
version: 1.0.0+1

dependencies:
  lib_c:
    git:
      url: https://github.com/Vanethos/dm_lib_c.git
      ref: git_reference_tag

And in our app_a we can reference all the other libraries:

name: app_a
version: 1.0.0+1

dependencies:
  lib_a:
    git:
      url: https://github.com/Vanethos/dm_lib_a.git
      ref: git_reference_tag_a

  lib_b:
    git:
      url: https://github.com/Vanethos/dm_lib_b.git
      ref: git_reference_tag_b

  lib_c:
    git:
      url: https://github.com/Vanethos/dm_lib_c.git
      ref: git_reference_tag

And if we try to run our app, our app can get all the dependencies, compile and run! So we overcome our last approach’s problem of not being able to compile and applied its idea of using git references to designate strict versions for each library.

With this approach each developer will be able to run and work on the same version of the app, leading to less conflicts and errors down the line.

Nonetheless there is a new problem with this method. As we said in the previous approach, all dependencies must have the same reference, which means that if we want to update the git reference on any library, we will need to create a commit in each library that references it. This can be a problem if we are working in a project with more than 20 libraries that may have dependencies between themselves and are constantly being updated. The process of tagging and updating each library can be time consuming and error-prone, which may lead us to discard this solution too.

One way we can solve this issue is by having our dependencies being set in a remote repository - introducing the Dependency Manager.

Proposed Solution - Dependency Manager

From our previous approaches, we now know two things:

If we use git references, all the developers on the team are going to use the same version of each library;
The problem with this approach is that each time that once there is a new update on a specific repository, we have to update all pubspec.yaml files with that new version.

So what if there was a way to have a remote repository where we control all our dependencies? That’s the main idea behind the dependency manager.

Dependency Manager Graph

Dependency Manager (also known as Bill of Materials), is an outside repository where we stipulate the versions for all libraries that we use. We can have as many Dependency Managers as we want, but we can use the following rule of thumb:

If we have libraries that are used in different projects we use a specific Dependency Manager for them
For each app we have a specific dependency manager.

But how can we create the dependency manager?

First, in the pubspec.yaml file we add all the dependencies that we want:

name: dependency_manager
version: 1.0.0+1

dependencies:
  lib_a:
    git:
      url: https://github.com/Vanethos/dm_lib_a.git
      ref: git_reference_tag_a

  lib_b:
    git:
      url: https://github.com/Vanethos/dm_lib_b.git
      ref: git_reference_tag_b

  lib_c:
    git:
      url: https://github.com/Vanethos/dm_lib_c.git
      ref: git_reference_tag

Then, inside the lib folder we have to create one dart file whose purpose is to export all our dependencies, making them available for all projects that depend on this one.

library dependency_manager;

export 'package:lib_a/lib_a.dart';
export 'package:lib_b/lib_b.dart';
export 'package:lib_c/lib_c.dart';

By doing this, when a file imports package:dependency_manager/dependency_manager.dart it will automatically have access to lib_a/lib_a.dart, lib_b/lib_b.dart and lib_c/lib_c.dart.

Finally, to make our dependencies available, we add the dependency_manager as a git reference for each library, for example:

name: lib_a
version: 0.0.1

dependencies:
  flutter:
    sdk: flutter

  dependency_manager:
    git:
      url: https://github.com/Vanethos/dm_dependency_manager.git
      ref: master

There is something odd in this process - we have our own library as a dependency of itself and we will have a lot of code in our dependencies that we will not use. This is where Tree Shaking comes into place - all the code that we will not use will be discarded when we compile our binaries.

For our dependency_manager to work, all our libraries need to point to the same reference of it so that there is no conflict. But what if I want to compile my app pointing to a specific version of the dependency manager? That is where the dependency_overrides come into place.

As the name implies, we can use dependency_overrides to override specific versions of libraries we are using. As an example - though we are using remote dependencies, what happens if we are working locally on a feature on lib_b and want to see how it behaves when integrated on app_a? If we adhere to the same folder structure as stipulated in the path dependencies, we can do the following:

name: app_a
version: 0.0.1

dependencies:
  flutter:
    sdk: flutter

  dependency_manager:
    git:
      url: https://github.com/Vanethos/dm_dependency_manager.git
      ref: master

dependency_overrides:
  lib_b:
    path: ./dm_lib_b

When pubspec compiles the dependencies, it’s going to retrieve all of them from the dependency_manager minus the lib_b dependency which is going to reference from path.

This has a huge advantage when working with distributed teams - when several developers are working on the same application, if one of them creates a Pull Request (PR), she can use the dependency_overrides to specify the PR branch so that’s easier for a reviewer to compile the app on their machine:

name: app_a
version: 0.0.1

dependencies:
  flutter:
    sdk: flutter

  dependency_manager:
    git:
      url: https://github.com/Vanethos/dm_dependency_manager.git
      ref: master

dependency_overrides:
  lib_b:
    git:
      url: https://github.com/Vanethos/dm_lib_b.git
      ref: feature/PR-1-adds_calculator

In this example when we use dependency_overrides, we will be able to compile the app with the new feature developed in branch feature/PR-1-adds_calculator.

If we are changing multiple libraries, we can also use dependency_overrides with the dependency_manager.

First, we create a new branch with the same feature name: feature/PR-1-adds_calculator. In it, we specify the ref for all the branches we are working on, for example we are only going change lib_a and lib_b:

name: dependency_manager
version: 1.0.0+1

dependencies:
  lib_a:
    git:
      url: https://github.com/Vanethos/dm_lib_a.git
      ref: feature/PR-1-adds_calculator

  lib_b:
    git:
      url: https://github.com/Vanethos/dm_lib_b.git
      ref: feature/PR-1-adds_calculator

  lib_c:
    git:
      url: https://github.com/Vanethos/dm_lib_c.git
      ref: dm_ref_c

Then, when creating a PR, we must tell other developers that they can use the following dependency_overrides:

dependency_overrides:
  dependency_manager:
    git:
      url: https://github.com/Vanethos/dm_dependency_manager.git
      ref: feature/PR-1-adds_calculator

And with one override, we basically stipulated that both lib_a and lib_b will target the branch feature/PR-1-adds_calculator that we have worked on. This makes it easier to build and deploy if necessary.

Downsides of using a Dependency Manager

Although we see many upsides with the dependency manager, there are a couple of issues that we need to be aware of.

The first one is that since we are using a lot of git references, when we try to get new packages via flutter pub get, it will take longer since pubspec needs to verify all the dependencies and git references and download them.

The second, is that since we are using a branch reference for the Dependency Manager, flutter pub get might not always get the latest version, which means that we need to upgrade the packages. However, this can cause an issue - maybe we want to get the latest version of the Dependency Manager but we don’t want to upgrade all of our other packages - so how can we solve this? By specifying directly which package we are upgrading:

flutter upgrade dependency_manager

With this command we will upgrade only the dependency_manager dependency.

Finally, as we have seen with the git references, each time we finish a new feature on a library we will need to release a new version of that package, which means that we need to:

Create a new tag in the relevant repository
Update the Dependency Manager with the latest tag This process can be time consuming and error-prone, hence why it is recommended that we create scripts to automate this process.

Conclusion

As we have seen, there are many ways for us to manage dependencies in pubspec, and though some might fit small applications, it is hard to create a solution that is scalable and usable in large codebases.

The solution we propose in this article is an alternative solution to manage our dependencies by creating a remote repository, and though it causes some issues such as long flutter pub get commands, we feel that it’s the solution that fits our purpose 🏇.

There was another solution that was not included in this article - creating our own pub server in which we could host our dependencies. This would cause other issues, such as the time setting up the server, and the cost of maintaining it, and it would not always give a solution to every problem - how would we update dependencies across libraries? And how could we stipulate the dependencies when someone is manually testing a PR?

As a final thought - these are the kind of problems that arise when we start creating apps that will be developed and maintainable over a long period of time. It’s hard to find or to create good solutions, so that is why I personally advise for sharing the knowledge and discussing it openly - have you ever come across a problem similar to this one? How did you solve it?

The GitHub repositories for this article can be found here: https://github.com/Vanethos/dm_dependency_manager https://github.com/Vanethos/dm_app_a https://github.com/Vanethos/dm_lib_a https://github.com/Vanethos/dm_lib_b https://github.com/Vanethos/dm_lib_c

Want to get the latest articles and news? Subscribe to the newsletter here 👇

And for other articles, check the rest of the blog! Blog - Gonçalo Palma