Using code generation to create a scalable and reliable codebase

Photo by Teng Yuhong on Unsplash

When developing multiple Flutter applications, we are doing the same tasks over and over again - we get the backend specification for our REST API or gRPC protos, we analyze it, create all the necessary classes (models, repositories and state-management classes) that conform to our architecture.

But, there’s a risk to manually do the same tasks over and over again - first, we have the time that we spend generating the code by hand, second we might induce some bugs in a specific iteration that we don’t catch before our client explodes our app, and lastly, if we change our app architecture (maybe we found an optimisation for our API response parsing), we will need to make the change in every project.

This creates an interesting challenges - if the code has the same set of constraints, with the only variables being the API it’s using and the classes name, what if we could create a tool that would generate all this code for us? One way to do it, is by creating a code template, which will allow us to generate code for us. In the present article we will instead use the name Code Generation since that is the outcome of using a code templating tool.

Flutter has already some libraries that can aid us in our quest of code generation, however, we are going to focus this article in the code_builder package, created by the Dart Tools development team that will give us a high degree of customization to fit our needs.

In this article we will use the code_builder package to replicate a simple Dart class that contains properties and methods.

Creating our Dart project

Before we can use the package, we will need to make a small adjustment - instead of working with Flutter, we will be creating a Dart project to helps us generate our code. With Dart, we can create a smaller script, that can be quickly ran via the terminal.

This Dart project can be created in different places - either we can add it to an existing Flutter application, or we can create a separate repository just for the code generation. To create it we will use the following command:

$ dart create <project_name>

To run our project we also make use of the Dart command line.

$ dart run bin/<project_name>.dart

However, we can rename our folder from bin to codegen and our <project_name>.dart to generation.dart or whatever we find is more suiting, which will change the run command to:

$ dart run codegen/generation.dart

Fundaments of the Code Builder Library

The code_builder package allows us to generate code by using different components- Classes, Methods, Parameteres, Code block, and Library that can define a different set of Directives for code imports, and plenty more, that we can see in the official GitHub Repo. Using these different elements we can start decomposing any piece of code.

Let’s take the following class as an example so that we can take it apart and create a small Dart tool to re-generate it.

import 'package:flutter/foundation';
import 'package:awesome_app/models/mechanical_builder.dart';

@immutable
class CarBuilder extends MechanicalBuilder {
  const CarBuilder(
    super.id,
    super.manufacturer,
    this.brand,
    this.color,
    this.isElectric, {
    this.parts,
  });

  final String brand;
  final String color;
  final bool isElectric;
  final List<String>? parts;

  String getOrder({required String factory}) {
    if (parts?.isEmpty ?? false) {
      return 'No Order, parts are needed';
    }

    return '[${isElectric ? 'ELECTRIC' : 'COMBUSTION'}] $factory: $brand';
  }
}

We start by identifying each component that we will need to use in order to replicate the same structure:

Library will be the top-level component, holding the properties for a file, such as imports, that are called Directives;
The Class component creates a class with all its properties, such as annotations, extends, mixins, and its name;
Fields are the properties of a class;
Methods are the functions that we can add at top-level or even inside a class and other functions;
Finally, each Class can have one or multiple Constructor that will accept different Parameters

With this set of components we are now ready to investigate each one individually and see how we can use them to create our Dart code generation tool.

Library Element

The Library component will allow us to add the following:

Directives, which are imports for the current file;
Annotations;
The body of the file, which can be Class, Method, Enum or other type of code_builder component.

Since our file does not have annotations but it has instead a couple of imports, we will start by trying to add one import to it, the file that contains the MechanicalBuilder, which we will need to create the CarBuilder class.

import 'package:awesome_app/models/mechanical_builder.dart';

To better organize our code, we will create a function called generateCarBuilder() that will be called in our main function that has the creation of the Library component. We will do the same for each other component, creating a new function, when necessary, to make our code more readable and organised.

void main() async {
  await generateCarBuilder();
}

Future<void> generateCarBuilder() async {
  final directive = Directive.import(
    'package:awesome_app/models/mechanical_builder.dart',
  );

  final output = Library(
    (lib) => lib
      ..directives.add(directive)
  );
}

Other than direct imports, the Directive allows to use export, importDeferredAs, part, partOf, and even show or hide specific classes per import.

As seen in the example, the classes in the code_builder package use the Builder Pattern, which means that instead of passing arguments via the constructor, we add them via a setter. As a result, non-List types such as the name can be assigned directly, whereas if we want to add new elements to a List parameter we need to add or addAll:

final output = Library(
  (lib) => lib
	  ..name = /* library name, to be added at the top of the file as `library some_name;` */
    ..directives.add(directive)
    ..body.addAll(/* classes, method, and parameters */)
);

Class

Here we will be adding the class name, the class it extends, mixins, and annotations.

For our example, we will want to replicate the following structure:

import 'package:flutter/foundation';

@immutable
class CarBuilder extends MechanicalBuilder {

}

Note that here we only are importing the flutter/foundation package, that is going to be needed for the @immutable annotation. The import for the MechanicalBuilder was already added in the Library component.

Using the same logic as before, we can explore the Class component of the code_builder library to see what we can customize:

Adding a name for the class;
Specifying which class we are extending;
The annotations for the class;

Some of the other fields - constructors, fields, and methods- will be explored over the next sections.

For both the annotations and the extend, we need to provide a way to specify what is the type we are importing plus any possible URL that can link to that type specification. In the case of the @immutable annotation, we will need the import for package:flutter/foundation.dart, which we specify inside the refer function.

Class getClass() => Class(
      (b) => b
        ..name = 'CarBuilder'
        ..extend = refer('MechanicalBuilder')
        ..annotations.add(
          refer(
            'immutable',
            'package:flutter/foundation.dart',
          ),
        )
        ..constructors.add(
          /* Constructors */
        )
        ..fields.addAll(
          /* Class Fields */
        )
        ..methods.add(
          /* Methods */
        ),
    );

The refer function creates a reference to another type. In the case of the MechanicalBuilder, we already have it resolved via the Library (as a comment, this is not a good practice, it was used only as an example to showcase how we can add imports via a Directive). Our URLs to other package can be added via dart:io, package:awesome_package/package.dart or even a relative import, ../../tools.dart.

Fields and Constructors

Fields can be class properties or any variable that we declare that is available globally to all files. In the case of the CarBuilder, it will be the list of the properties for the class.

  final String brand;
  final String color;
  final bool isElectric;
  final List<String>? parts;

As with the Class component, we will be able to create a new Field component by declaring its properties, namely the name, type, and modifier (if the field is final, a const or a variable). Since can have more than one Field in our class, we will create a List.

List<Field> getClassFields() => [
      Field(
        (field) => field
          ..name = 'brand'
          ..modifier = FieldModifier.final$
          ..type = refer(
            'String',
          ),
      ),
      /* Other Fields */
      Field(
        (field) => field
          ..name = 'parts'
          ..modifier = FieldModifier.final$
          ..type = refer(
            'List<String>?',
          ),
      ),
    ];

These Fields are assigned via in the class constructor, that we can create via the Constructor component. As we know, Dart classes can have multiple constructors, some of them used as a factory for external libraries, such as the case of the freezed package.

For the CarBuilder we know the following:

We will have parameters that will be passed to the parent class, the MechanicalBuilder, via a super.parameterName;
Some of our parameters will be positional, whereas one of them is going to be a named parameter, the list of parts;
Finally, we will label our constructor as const.

  const CarBuilder(
    super.id,
    super.manufacturer,
    this.brand,
    this.color,
    this.isElectric, {
    this.parts,
  });

In this case, we will have the Constructor component that holds a list of requiredParameters and optionalParameters. The Parameter component is fairly similar to the Field component, the major difference being that we can add a toThis flag so that the parameter has a this.parameterName and a toSuper flag that converts the parameter to super.parameterName.

Constructor getConstructor() => Constructor(
      (b) => b
        ..constant = true
        ..requiredParameters.addAll(
          [
            Parameter(
              (parameter) => parameter
                ..name = 'id'
                ..toSuper = true,
            ),
			   Parameter(
              (parameter) => parameter
                ..name = 'brand'
                ..toThis = true,
            ),
            /** Other Parameters **/
          ],
        )
        ..optionalParameters.add(
          Parameter(
            (parameter) => parameter
              ..name = 'parts'
              ..toThis = true
              ..named = true,
          ),
        ),
    );

Method

As the final step, we are going to generate the getOrder function that we have inside our CarBuilder class.

String getOrder({required String factory}) {
  if (parts?.isEmpty ?? false) {
    return 'No Order, parts are needed';
  }

  return '[${isElectric ? 'ELECTRIC' : 'COMBUSTION'}] $factory: $brand';
}

Since there are multiple parts to this function, we will divide it into two components - the function declaration and its body.

String getOrder({required String factory}) {}

Analysing the function declaration we know that it is a function that has a return type, a name, and a set of parameters. This can be easily translated inside the Method component, alongside the Parameter component for the factory, which we will set as required.

Method getMethod() => Method(
      (method) => method
        ..returns = refer(
          'String',
        )
        ..name = 'getOrder'
        ..optionalParameters.add(
          Parameter(
            (p) => p
              ..required = true
              ..named = true
              ..name = 'factory'
              ..type = refer('String'),
          ),
        )
        ..body = /* Code Block */,
    );

The body of the method, where we are going to add our logic and return our String, is composed of two small code blocks - the if statement and the last return statement.

if (parts?.isEmpty ?? false) {
  return 'No Order, parts are needed';
}

return '[${isElectric ? 'ELECTRIC' : 'COMBUSTION'}] $factory: $brand';

The major difference between a code block and the rest of the components that we have discussed so far, is that instead of creating the body as a composition of elements, we will be giving a String to a Code component. However, instead of creating the String for a whole block of code, we can instead use the Block component which accepts different Code components, allowing us to create smaller and reusable sets of Code components.

Block getMethodCodeBlock() => Block(
      (block) => block
        ..statements.addAll(
          [
            generateIfBlockStatement(),
            generateReturnStatement(),
          ],
        ),
    );

Code generateIfBlockStatement() => const Code(
      '''if (parts?.isEmpty ?? false) {
      return 'No Order, parts are needed';
    }''',
    );

Code generateReturnStatement() => const Code(
      "return '[\${isElectric ? 'ELECTRIC' : 'COMBUSTION'}] \$factory: \$brand';",
    );

Formatting our code and printing to the console

Now that we have all of our code generated via different code_builder components, what remains is to generate the code.

For that, we will need to find a way to convert all the Components into a String buffer so that can add it to a File or print it in the console. This is achieved by using the DartEmitter class. This class allows us to add all the directives we specified in the refer functions, by creating a new Allocator that will take care of that for us. Plus, it can organise all our imports by setting orderDirectives as true.

Future<void> generateCarBuilder() async {
  /* Library Component creation */

  final emitter = DartEmitter(
    orderDirectives: true,
    allocator: Allocator(),
  );
}

Then, to properly present our code, we will need to format it, which can be easily achieved by using the DartFormatter class from the dart_style library.

Future<void> generateCarBuilder() async {
  /* Library Component creation and emitter*/
  print(
    DartFormatter().format(
      '${output.accept(emitter)}',
    ),
  );
}

Running our Dart command we will see our CarBuilder class printed in the console!

In the following gist we can see the full code sample.

Creating a new file with the generated code

But what if we want to save it into a file instead of printing it into the console?

In this case, we can make use of the dart:io library, which allows us to create folders, with the Directory class, and files, with the File class.

For our specific case, we can create a codegen/out folder where all of our generated code will live, and create a new file called car_builder.dart.

void main() async {
  await generateCarBuilder();
}

Future<void> generateCarBuilder() async {
  await Directory('codegen/out').create();

  var file = File('codegen/out/car_builder.dart');
  
  /* remaining code */

  await file.writeAsString(
    DartFormatter().format(
      '${output.accept(emitter)}',
    ),
  );
}

If we rerun our dart tool, we will see that a new file was generated with all the expected code!

Next Steps

We were able to achieve our objective, which was to replicate a simple class, with fields, a constructor, annotations and a method via the code_builder package.

But, truth be told, as it stands it is not very useful. We are only able to generate one instance of a class without any type of customization.

What can we do about that?

Since we have all our code separated into different functions - a function to create a class, a method, and so on, we can change them to instead of generating the hardcoded properties that we have set, they could allow for some sort of input. This input could come from two different sources. On one hand, we could create a Dart class that would specify all the information we would need, or we could create a yaml file that is read on startup in which we would extract all our information. The second route could be better to more easily organize and share your code. Nonetheless, in both ways we would provide all the info we would need - the list of classes, their properties, file location, and so on, allowing us to generate all the code that we would need.

There is an edge case - the body of the methods or any other code blocks. For this case, we could ask ourselves - what type of functions do I need to constantly generate? Maybe it’s code for the state management framework that we are using or all the abstractions that we need to generate. For each case, we could then create different types of inputs - an input with information for the generation of the necessary bloc and repositories, for example.

Also, we should not stop at the generation of user-facing code. We could also go the extra mile and generate tests for our generated code! This way we are sure that all the code that is inside our repo is fully tested.

Conclusion

With code_builder we have a lot of flexibility in what we want to do. But, if we don’t organize our code in a proper way, we will quickly have a chaotic codebase that will be too difficult to handle.

This is to say that code_builder should not be the only code generation tool in our tool belt, but we could also look into friendlier alternatives, such as mason_cli that quickly generate templated code. There’s even a website that hosts all the official and user-generated templates, called BrickHub.

Or, we could even go into another completely different direction and generate a custom code-generation engine with tools such as Go Templating, that although has a steep learning curve, will give us the same, or even higher, level of flexibility as code_builder with a better code organization structure.

Nonetheless, no matter what your tool of choice for code generation, we should always follow the same set of steps:

Is this a repeatable pattern of code that we need to automate its generation?
Are these patterns of code present in different projects, or are there multiple instances of it yet to be created?
Is this a substantial amount of code that needs automation? Or can I use a library to achieve the same result?
How am I going to maintain my generated code? Is it going to be generated once, or will I continuously generate it once there are updates to the code templates?
What is the time cost of developing the code automation tool? Does it make sense looking at what needs to be generated?

With all that said, happy coding! And may code generation be a way for all of us to achieve our objectives faster and with more quality!

Originally published at Invertase - Using code generation to create a scalable and reliable codebase.

Want to get the latest articles and news? Subscribe to the newsletter here 👇

And for other articles, check the rest of the blog! Blog - Gonçalo Palma