Breaking Boundaries: A Developer's Journey through the challenges with low code tools, Web Scraping, and Code Injection for a Nonprofit (Regenerativt Sverige)

In my recent collaboration with the non-profit organization, Regenerativt Sverige, I embarked on a unique journey tackling challenges with low code tools, web scraping, and code injection. This organization, formed in 2023, is a response to the growing movement of Regenerative Agriculture, both in Sweden and globally. More about them at the end of the article.

I first encountered this inspiring organization through a course at Hanken, where a member served as a guest lecturer. Intrigued by their cause, I offered my assistance. They had a website but needed a solution for member-generated content related to the regenerative movement, as well as a map showcasing member activities. They were also intrigued by the idea of an app.

The organization has a website that they developed with Squarespace. Squarespace serves as a comprehensive all-in-one content management system (CMS). With a single subscription, you gain the ability to effortlessly create and design your website, host content, secure a custom domain name, facilitate product sales, monitor site analytics, and explore various other functionalities. Squarespace, inherently designed as a no-code platform, offers seamless website development. However, opting for a higher subscription introduces the capability for CSS-injections and code-injections, providing advanced customization options. Furthermore, a premium subscription unlocks API access, empowering users with additional functionality for specific tasks.

Navigating platforms like Squarespace presents a spectrum of advantages and drawbacks. The foremost advantage lies in its user-friendly interface, catering seamlessly to individuals with minimal or no coding expertise. A plethora of templates streamlines the process of launching your website and incorporating content. However, the drawbacks surface in the form of limitations within Squarespace. Certain functionalities remain inaccessible, and the platform offers a finite selection of integrations, restricting the augmentation of your website. Consequently, incorporating features such as user registration and content creation becomes unattainable, hindering the realization of specific desired functionalities.

After discussions, we decided on a two-pronged approach: a separate website with additional features and an app using Flutter for a unified codebase.

Leveraging Flutter for app development not only streamlines the process but also opens the door to crafting a website using the same codebase with slight adjustments for optimal browser customization. This approach significantly enhances efficiency, allowing for a unified codebase across IOS, Android, and web platforms. Remarkably, this consolidation doesn't necessitate an escalation in the subscription plan, ensuring a cost-effective solution without compromising on functionality.

Functionality Overview:

1. User Interaction:
   • Members register and sign in to the app, saving their information to a database.
   • Signed-in members access a map, view other members' details, and create events/articles.
2. Admin Workflow:
   • Admins receive notifications of new articles/events.
   • After review, admins copy content to Squarespace and press 'Publish.'
   • All members receive notifications of the new content.
3. Cross-Platform Accessibility:
   • Content is visible in both the app and on the Squarespace website.

This way combines the good parts of both technologies. You get the simplicity of Squarespace and the extra features with the app and website using Flutter, along with Firebase as the backend. It's a complete solution that's easy to use and gives you more capabilities.

Web scraping and code injections

Now, delving into the technical aspects of this blog post. Given the absence of a usable API for retrieving various events and articles, I opted for the webviewwidget in Flutter (https://pub.dev/packages/webview_flutter). This package not only facilitates the display of a website within the app but also grants the flexibility to execute JavaScript for modifying its appearance.

I began the process conventionally by simply loading the page for the articles:

controller = WebViewController()
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      ..loadRequest(Uri.parse('https://www.regenerativtsverige.se/lasning'));

I found it necessary to execute some JavaScript as there was a navbar on the page, and I wanted to prevent users from getting confused and inadvertently navigating to other pages. After inspecting the HTML element on the website, I devised the following solution:

controller = WebViewController()
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      ..setBackgroundColor(theme.backgroundColor)
      ..setNavigationDelegate(
        NavigationDelegate(
          
          onPageStarted: (String url) {
            controller.runJavaScript('''
      var element = document.querySelector('header');
      if (element) {
        element.style.display = 'none';
      }
    ''');
          },
          
          onWebResourceError: (WebResourceError error) {},
          onNavigationRequest: (NavigationRequest request) {
            if (request.url.startsWith('https://www.regenerativtsverige.se/lasning')) {
              return NavigationDecision.navigate;
            }
            return NavigationDecision.prevent;
          },
        ),
      )
      ..loadRequest(Uri.parse('https://www.regenerativtsverige.se/lasning'));
  }

Here, I incorporated JavaScript to locate the header class and set its display to none. This effectively removed the navbar. Additionally, I implemented the onNavigationRequest to restrict URLs to those starting with 'https://www.regenerativtsverige.se/lasning', ensuring users can only navigate to pages containing articles.

I also addressed an issue with the image quality, particularly on iOS devices. To rectify this, I implemented the following adjustments: 

 var images = document.querySelectorAll('img');
    images.forEach(function(image) {
      image.srcset = image.src;
    })

By modifying the image srcset to the image source, I successfully restored the high-quality resolution of the pictures.

On the event page, there was already a button facilitating the return to all events from a specific event. However, this feature was lacking for articles, and Squarespace didn't offer an option to add it without resorting to code injections (available only with a higher subscription). 

Consequently, I implemented JavaScript to dynamically include a back button on the various articles.

 if (url.length > 'https://www.regenerativtsverige.se/lasning/'.length) {
              controller.runJavaScript('''
                var elementToPushDown = document.querySelector('.blog-item-inner-wrapper');
                if (elementToPushDown) {
                  // Add a 20-pixel margin to the marginTop property
                  var currentMarginTop = parseInt(window.getComputedStyle(elementToPushDown).marginTop) || 0;
                  elementToPushDown.style.marginTop = 20 + 'px';
                }

                var textElement = document.createElement('div');
                textElement.innerHTML = '<p style="position: absolute; top: 0px; left: 22.5px; background-color: transparent; z-index: 9999;"><span style="font-size: 20px;">&larr;</span> Tillbaka till alla artiklar</p>';
                document.body.appendChild(textElement);

                textElement.addEventListener('click', function() {
                  window.location.href = 'https://www.regenerativtsverige.se/lasning';
                });

                window.addEventListener('popstate', function() {
                  textElement.parentNode.removeChild(textElement);
                });

                history.pushState({}, '');
              ''');
            }

Initially, I adjusted the layout by pushing down the container containing the article content to prevent any overlap with the back button. Subsequently, I introduced a new div element and applied styling to enhance the appearance and positioning of the added back button.

Following those adjustments, the controller code appeared as follows:

late WebViewController controller;

  @override
  void initState() {
    super.initState();
    controller = WebViewController()
      ..setJavaScriptMode(JavaScriptMode.unrestricted)
      ..setBackgroundColor(theme.backgroundColor)
      ..setNavigationDelegate(
        NavigationDelegate(
          onPageStarted: (String url) {
            controller.runJavaScript('''
      var element = document.querySelector('header');
      if (element) {
        element.style.display = 'none';
      }

      var images = document.querySelectorAll('img');
    images.forEach(function(image) {
      image.srcset = image.src;
    });

    ''');
          },
          onPageFinished: (String url) {
            if (url.length > 'https://www.regenerativtsverige.se/lasning/'.length) {
              controller.runJavaScript('''
                var elementToPushDown = document.querySelector('.blog-item-inner-wrapper');
                if (elementToPushDown) {
                  // Add a 20-pixel margin to the marginTop property
                  var currentMarginTop = parseInt(window.getComputedStyle(elementToPushDown).marginTop) || 0;
                  elementToPushDown.style.marginTop = 20 + 'px';
                }

                var textElement = document.createElement('div');
                textElement.innerHTML = '<p style="position: absolute; top: 0px; left: 22.5px; background-color: transparent; z-index: 9999;"><span style="font-size: 20px;">&larr;</span> Tillbaka till alla artiklar</p>';
                document.body.appendChild(textElement);

                textElement.addEventListener('click', function() {
                  window.location.href = 'https://www.regenerativtsverige.se/lasning';
                });

                window.addEventListener('popstate', function() {
                  textElement.parentNode.removeChild(textElement);
                });

                history.pushState({}, '');
              ''');
            }
            
          },
          onNavigationRequest: (NavigationRequest request) {
            if (request.url.startsWith('https://www.regenerativtsverige.se/lasning')) {
              return NavigationDecision.navigate;
            }
            return NavigationDecision.prevent;
          },
        ),
      )
      ..loadRequest(Uri.parse('https://www.regenerativtsverige.se/lasning'));
  }

I repeated the process but excluded the addition of the back button for the events. 

Webscraping with Flutter

Besides the changes mentioned earlier, they also wanted a page with all the links from their header. These links included information about the organization and basic details about regenerative agriculture. To make things easier and avoid adding links manually each time something new was added, I used a method called web scraping to get these links automatically.

To start, I examined the HTML elements to understand the structure. Notably, every header item with links had the class ".header-nav-folder-title," prompting me to collect all such elements:

var folderTitles = document.querySelectorAll('.header-nav-folder-title');

Next, in a for loop, I extracted the title from each class and added it to the list if it was unique:

var topicTitle = titleElement.text.trim();

      // Check for uniqueness before adding to the list
      if (!uniqueTopics.contains(topicTitle)) {
        uniqueTopics.add(topicTitle);
      }

Following that, I obtained all the classes for items inside the header titles and selected the right one using the loop index:

var folderItems = document.querySelectorAll('.header-nav-item--folder');
var linkItems = folderItems[index].querySelectorAll('.header-nav-folder-item');

Utilizing a map, I extracted the title and link for each item:

var topicLinksList = linkItems.map((element) {
        var titleElement = element.querySelector('.header-nav-folder-item-content');
        var linkElement = element.querySelector('a');

        return {
          'title': titleElement?.text.trim() ?? '',
          'href': linkElement?.attributes['href'] ?? '',
        };
      }).toList();	

topicLinks.add(topicLinksList);

The overall code structure looked like this:

 List<List<Map<String, String>>> topicLinks = [];
  List<String> uniqueTopics = [];

  @override
  void initState() {
    super.initState();
    fetchWebsiteData();
  }

  void fetchWebsiteData() async {
    final websiteUrl = Uri.parse('https://www.regenerativtsverige.se/');
    final websiteResponse = await http.get(websiteUrl);
    dom.Document document = dom.Document.html(websiteResponse.body);

    var folderTitles = document.querySelectorAll('.header-nav-folder-title');

    for (var (index, titleElement) in folderTitles.indexed) {
      var topicTitle = titleElement.text.trim();

      // Check for uniqueness before adding to the list
      if (!uniqueTopics.contains(topicTitle)) {
        uniqueTopics.add(topicTitle);
      }

      var folderItems = document.querySelectorAll('.header-nav-item--folder');
      var linkItems = folderItems[index].querySelectorAll('.header-nav-folder-item');

      var topicLinksList = linkItems.map((element) {
        var titleElement = element.querySelector('.header-nav-folder-item-content');
        var linkElement = element.querySelector('a');

        return {
          'title': titleElement?.text.trim() ?? '',
          'href': linkElement?.attributes['href'] ?? '',
        };
      }).toList();

      topicLinks.add(topicLinksList);
    }

    setState(() {});
  }

Finally, within the scaffold widget, I utilized the scrapped data to construct a user interface. I created a Column widget where, for each unique topic, I displayed the title and associated links:

for (var (index, topic) in uniqueTopics.indexed)
                    Column(
                      crossAxisAlignment: CrossAxisAlignment.start,
                      children: [
                        AutoSizeText(
                          topic,
                          style: theme.h2,
                        ),
                        for (var link in topicLinks[index])
                          MoreLink(
                            title: link['title']!,
                            titleUrl: link['href']!,
                          ),
                        const SizedBox(
                          height: 15,
                        ),
                      ],
                    ),

This arrangement allowed me to loop through the header titles, display them, and underneath each title, show the item titles. Clicking on an item opened the associated URL.

This is how it looked:

Regenerativt Sverige as an organization:

The organisation Regenerativt Sverige (Regenerative Sweden) was grounded in 2023. It is the result of conversations and discussions which have taken place over the previous years, around how best to represent and support Regenerative Agriculture which is clearly a growing movement in both Sweden and globally.

They define Regenerative Agriculture as:

“The facilitation of the highest imaginable vitality in ecosystems through the efficient provision for human need”

In practice this translates to the production of the food and raw materials, which we as humans depend on, in a way which seeks to work in harmony with natural processes as much as possible. Our actions and management should heal degenerated ecosystems and lead to more vitality and abundance. Life creates life. We focus especially on the integration of grazing animals (primarily ruminants) in an adaptive grazing system as the motor in kickstarting the regeneration of life both above and particularly below the soil. We try to maintain a landscape perspective and resist the temptation to divide land into categories of use.

Worry about our futures, climate change and global industrialisation and consolidation of resources and power are symptomatic of the growing disparity between what we choose to prioritise in our societies (especially in terms of how we produce and consume food and raw materials) and how we as individuals relate to the natural world. We believe in a fundamental paradigm shift towards an acceptance of our responsibilities as an important and fundamental part of the epic and fluid complexity which is nature. We see this as the only positive forward direction to take.

Through supporting, nurturing and facilitating the development of the Regenerative movement in Sweden we hope to see a growth in the number and frequency of people and organisations coming together to both reinforce our connections as humans and to create opportunity for food and resources to be produced and used in ways which not only regenerate the ecosystems on which we all depend but also our social and economic systems.

Regenerative Sweden is a non profit organisation whose committee members work to engage our current and future members in creating and maintaining a network, both nationally and internationally in the wider regenerative movement. We are proponents of Holistic Management as a framework for our decision making and have close ties to the Savory Institute. We are always keen to make new contacts and learn from the experience of others.

For more information go to www.regenerativtsverige.se