Mobile app architecture with FlutterFire

In my time working as a software developer, I previously had a chance to be part of small development teams building production ready apps. Hopefully, this article can guide other teams in building their apps.

Traditionally, if you wanted to build an app for mobile users, you would have to form a sizable development team to write the different parts of the app. It can be hard to find people to do this, so the question is – Can this be done with a much leaner team of 3 or less? The good news is that it can! There are technologies in the market that allow engineers to build mobile apps quickly and cost-efficiently.

Depending on the type of app you are building, development requirements may differ. Some apps need to store and retrieve large amounts of data, so those require a database; If the app needs to communicate to other users, you need a server; Finally all apps definitely need a user-interface(UI). Other things your app may need, like payments and video streaming are out of the scope of this article. To fulfil most app requirements, I recommend a FlutterFire Architecture.

Architecture overview

FlutterFire comprises of 2 parts: Flutter and Firebase. Flutter is the frontend code which can be deployed to both iOS and Android devices. Firebase will be the backend of the app, residing wholly in the cloud. If you are familiar with either Flutter or Firebase, you may skip the introductions and proceed to the FlutterFire Implementation.

Image 1: Overview diagram

Flutter – Flutter is a framework written in Dart language which is used to package apps to both Android and iOS devices. Before Flutter, developers had to build Android and iOS apps separately. Flutter allows users to maintain a single code base for both. React-Native is another cross-platform solution not discussed here. You can find instructions on how to install Flutter on your device here: https://docs.flutter.dev/get-started/install

Firebase – Firebase is Google’s mobile development platform Firebase has a range of cloud services that provide easy-to-implement solutions to common needs such as accounts and database management. AWS and Azure provide similar services, however in the case of Flutter the integration is better supported and documented at the time of writing.

The benefit of using cloud services like Firebase is that you do not have to buy your own servers and pay only what you use. You also have to added benefit of writing simple code to perform calls to Firebase to for most of your app features. The rest of this article is dedicated to explain the services and how they integrate to mobile apps.

FlutterFire Implmentation

Authentication – Use this for signups, sign ins and account management. Firebase Authentication helps you securely store and authenticate logins for free! You also have additional options like sending verification emails/sms. https://firebase.flutter.dev/docs/auth/usage/

Cloud Firestore – This is a no-SQL database. Instead, it stores data as documents and provides fast and cost-effective querying. Flutter can directly perform CRUD operations on Cloud Firestore. Docs can be found here: https://firebase.flutter.dev/docs/firestore/usage/.

I recommend Engineers check these 3 things when using Cloud Firestore:
1. Configure the security settings on the database to ensure data protection. This can be found in the console under the ‘rules’ tab.
2. There are limits to querying due to the structure of data. This is especially the case with compound queries. Check out the docs here: https://firebase.google.com/docs/firestore/query-data/queries.
3. I would advise programmers to create a class for every document type in Flutter. This class will store type information like String, double, List<>, etc. This is to ensure the data types sent and retrieved from the database are consistent. Firebase and Flutter types are not totally compatible

Cloud Functions – This is the where the main backend code will reside. Any code that performs database operations, network calls or traditional server work, will be deployed here. Again, the benefit of having a serverless architecture is cost-efficiency. There is no need to buy and maintain a server, you can pay-per-use on Firebase.

There are 3 types of functions you should be aware of:
1. On Request – As the name suggests, these functions are called Ad Hoc.
2. Scheduled – These functions are run at specified intervals.
3. Triggered – These functions are useful if you want code to run when certain events happen. Some common examples are running database checks or operations when the database is changed. Other triggers include when a user authenticates, when a file is uploaded to Cloud Storage etc.

With these functions, you do not need to provision any servers. Firebase will spin up server instances to run these functions when required! Check out the docs here: https://firebase.flutter.dev/docs/functions/overview/

Edit: In my excitement in writing all the benefits of Cloud Functions, I neglected to mention one very important drawback – Cloud Functions are written in Javascript or Typescript, NOT Dart. So you may have to learn how to code in those languages and setup the libraries.

Cloud Storage – Cloud Storage is similar to AWS S3 and Azure Cloud Storage. It is a storage for files like images, videos, audio, etc. There is nothing that make it stand out from other cloud storage solutions, you could even use others like AWS S3 if you want, but since the Firebase account is already setup, you save time and effort implementing this part of your app. Docs for Flutter’s integration with Cloud Storage can be found here: https://firebase.flutter.dev/docs/storage/overview/

Cloud Messaging – If you want your app to notify users when the app is in the background or is not running at all, you need to setup Cloud Messaging. This could be the most tricky to setup out of all the other services because different platforms control notifications differently. However, once you manage to setup Messaging correctly, Firebase and Flutter help you send and receive notifications to both iOS and Android users with no fuss! Docs for Cloud Messaging: https://firebase.flutter.dev/docs/messaging/overview

Final note about using Firebase: TRACK YOUR BILLING. This article describes how a charity racked up a 30k bill overnight from poorly written code: https://hackernoon.com/how-we-spent-30k-usd-in-firebase-in-less-than-72-hours-307490bd24d. You can track you billing easily in the Firebase console’s settings.

Conclusion

There is long term support and planned upgrades for both Flutter and Firebase. The features listed here are not extensive and one should examine the docs to see if the FlutterFire architecture is appropriate for your app. Nevertheless, it is worth stating that building an app with this architecture can be done by a solo developer (I have done it). The free tier of Firebase is very generous and if you have little traffic on your app, no need to sweat because you pay close to nothing.

Happy coding!

Beginner tips for Exploratory data analysis with Pandas

This article is written as part of Gaogao’s Advent Calendar series. This is the first year I’m participating and I wanted to write about an area of tech that I’m interested in and currently working on. Here I present eight tips for common issues that beginners would find useful.

Data can come from everywhere – web scrapping, online surveys, collection from customer records, data analytics platforms, and messy data is inevitable. Being a beginner starting your first few projects, messy data is a scary prospect. Hopefully the tips here can help you resolve generic issues while you overcome the initial panic of seeing over a million rows of haphazard, missing, strange data. I assume you are using either a notebook of some kind like Jupyter Notebook, Google colab, Kaggle etc. I also assume you are coding in Python and using the Pandas library.

Optional Parameters

A lot of inbuilt methods come with useful options that users, both experienced and beginner, sometimes forget about. Sometimes the documentation doesn’t explain it in the best way so you may have to Google some example or try it yourself to understand what each parameter does. Here are just some examples

#1 Tip: For reading and exporting ,
i) remember to use the ‘encoding’ parameter if non-Latin characters are present, e.g. Japanese /Chinese;
ii) if you do not want indexes, use ‘index=false’
Ex – df.read_csv(or read_excel)(<filepath>, encoding=”utf-8″, index=False)
Ex – df.to_csv(or to_excel)(“CleanData.csv”, encoding=”utf-8″, index=False)

#2 Tip: Check the axis. Some methods require you to specify ‘axis=1′ to apply to columns(axis=’columns’ is the same but longer to type). Else you will apply them to rows!
Ex – df.sort_values(by=1 ,axis=1)

#3 Tip: Remember to use ‘inplace=true’ to shorten your code
Ex – df.drop(<col_name>, axis=1, inplace=True)

Column Operations

If you can find a ready-made method like .groupby() or .split(), use those. They are faster and cleaner. Otherwise the next best I would suggest using is .apply + lambda function. This is highly flexible and quite readable. I use this A LOT.

#4 Tip: If you want to test a lambda function, remember to return the original variable. This is to ensure the data is not accidentally changed.

Return original in all cases when checking output

#5 Tip: To make your code readable, I suggest defining and running the function in the same code box if possible.

Lambda functions applied to df columns in the same code block

Python Types

After using Typescript and other coding languages that can type checking, Python can feel like a step backwards. The numbers your assumed were ‘int’s (integers) turned out to be ‘str’s (strings) or there was a ‘NaN’ in one of the rows which made the method throw an error… Its frustrating. Unfortunately, there is no easy way around this problem but to check and convert all data points in your columns to the one you desire.

#6 Tip: Use .dtypes() to confirm all types are consistent.

‘object’ type could either be all strings or a combination of types

#7 Tip: Get rid of ‘NaN’s in your columns. Use .isna().sum() to check for ‘NaN’s. Then use either .fillna() or dropna().

#8 Tip: Change all column data to a single type.
Example: df.loc[:, ‘col_name’] = df.loc[: ,’col_name’].astype(float).

Thank you for reading and hope you traverse your messy data with confidence!

First time Flutter

So we couldn’t travel around Japan as much as we would have liked, but I managed to channel that energy to finishing my second mobile application. This time the app is a learning and dictionary resource for users who are interested in learning business Chinese and was built with the Flutter framework. It is called QUIKA and it took about 9 months for this baby to go from concept to full launch, a journey which started around the end of September 2020. I’m writing this to journal the steps (and struggles) that were undertaken for this app-making journey, with a little technical background at the beginning.

Flutter is a framework meant to help developers build mobile applications on Android and iOS in one single code block. Traditionally, you would have needed to code in either Kotlin or Java for Android AND have a separate code in Swift for iOS. Thus, even though I didn’t have any experience in Flutter, I was willing to learn the language in order to make the app on both platforms. Also, I had no choice. Our project team only had 3 people, 2 of whom were working full time jobs and had more than their fair share of personal commitments to deal with. They also had never built an app before. That left me to squeeze out what little expertise I had with programming and try to set this thing up.

_{Initial design for splash screen, category selection and flashcards}

Learn by doing. This is probably the best phrase to sum up the entire journey. From the initial set up to the eventual launch, 95% of tasks were something that I’d never done before. Github, StackOverflow and Youtube will have a permanent dock on my Google Chrome browser. Additionally, because I was the main programmer, I had to solve most of the programming problems on my own. I couldn’t say “eh, I don’t know how to do this, can you do it?”. There was no one else. This meant that if I could not find an off-the-shelf solution, I had to homebrew my code and I can tell you, a lot of it is homebrewed.

Enjoy the process. You won’t see the end product or results when you are in the thick of it, but you will know when you are making progress (or not). There were times when I had to get off the terminal and go do some exercise because the solutions were just not coming. I would say that that is actually the best thing to do. You aren’t doing a timed exam nor being assessed daily. And when I did get something solved, or my code didn’t throw errors when running, you feel like a baker who had successfully baked a technically challenging but incredibly delicious cake (Many things can go wrong with cake. I know because my wife bakes).

We took twice the time we planned for. Since our app helps users to learn Chinese, we initially wanted to push it out for testing during Lunar New Year. That plan was slowly pushed to March, then April. Then once we got some feedback, we deliberated long and hard over what we had to do to get the best app we could possibly make. Even now, we are convinced that it can be better. It can always be better. But perfection is the enemy of progress, so rather than overthinking the tweaks and smoothen every aspect of the UI, we decided to launch it and test the waters. The app was launched on the Android Playstore on 9 June 2021, and on the Apple App Store the following day, 3 months later than when we set out to launch.

_{Old Quiz UI and list customizations that eventually did not make it into the final design}

You are not a software engineer, until you are. I don’t have any formal certification in computer programming or software engineering. I doubt that I’m using best practices in a lot of the code that I write (I’m probably not). Nevertheless, I finished a working app; one that is tested on the iPhone and Android phones; one that runs smoothly with no crashes and has 1 or 2 minor bugs. I’m also working on developing my own computer game and other programming endeavors, coded in a different language and entirely different setups. I think you have to do it once to know if you can do it and if you like it. For me it is both.

Most problems are solvable. It’s not to say they were simple. In fact some were really tough. Like when we had to trawl the internet to find 1000 example sentences in Chinese. For that I learnt how to use Beautiful Soup and Selenium. Other problems were very annoying. I knew for these problems, they were due to poor documentation or just systemic/historical issues with some of the frameworks. The process of launching on iOS was particularly cumbersome. Not only is Apple more expensive to publish on, I ran into so many issues which led to much swearing (like when they rejected my app icon because it had an alpha channel…). Despite these issues, we managed to come out the other side with more experience and knowledge than when we first started.

Document your process. Besides commenting your code (which is sacrosanct), you MUST MUST MUST record all your troubleshooting history, especially those that don’t involve code. They can be things like how to resolve merge conflicts in Git, or how to solve issues with Swift reading Dart code (these were the closest I came to physically fighting the laptop). One might ask which processes are worth recording amidst the vast number of problems you solve. The simple answer would be that 1. the solution took a ridiculous amount of time and effort only for the actual process to take minutes; and 2. its something specific that you don’t come across often so you will forget. Perhaps you may even be able to help someone in the future.

As a final note, there is one last item on the list, which is to get feedback on real world users. This app isn’t for me after all. I would like to know if people find this useful, or if there are features that they would like which will be of use to them. My goal is to scale the value and usefulness of software to a bigger and bigger group of society. The first was a personal grocery list app, now it’s a business Chinese resource app, next it would be a game. Beyond that, I’m not sure.

If you want to try the app, search “Quika” on either the Android or Apple app stores. The app icon is the one on the top of this post. You can also find us on https://www.linkedin.com/company/quika-app.

Launched v1.0

Week 1: Indie game dev of (untitled)

Currently, I’m serving a 14-day self isolation period which means I can’t go out to meet people and eat at places I have been missing for 4 months. It also means that I can’t write about our travels. Instead, I have been taking the time to work on my next project – developing an indie game.

When I first returned, my neighbour Sean proposed we collaborate on making a game together. I have no programming or development background, but he knew I had just launched my app on the Google Playstore and have some basic understanding of Unity and thought it would be a good idea to create something. I agreed and we got to work immediately.

Here’s a simple day-to-day breakdown of what I have done so far:

Day 1:
Discussed Game Design – 2D top down RPG
Story and theme – Rubbish collection and sorting adventure with fantasy-based backdrop
Game Engine – Unity
Graphical editors – Aseprite & GIMP
Set up Trello board and Discord channel

Day 2:
Set tile map dimensions
Set up Github for Unity
Started with 2D art
Discussed areas, sorted game mechanics and UI
Basic movement and control

Day 3:
More art
Discussed inventory system, story development and pet design

Day 4:
Finally got Github working
New area art and theme

Day 5:
Research into similar games
Items list and NPC development
Work on inventory system

Day 6:
Major re-shading

Day 7:
Repaint my rocks

This is the product of the first 7 days of art work. I think it’s pretty decent for someone who is not an artist and doing pixel art for the first time.

From 0 to first production app

Oh Corona-chan, what have you done. This site was supposed to be about my life and travels in Japan. Instead, I’ve been stuck in Singapore since April, missing out on Hanami season, when the sakuras are in bloom. Our real estate agent in Oita even sent us a letter asking if the water meter is broken because it read 0. Hopefully, we will be able to get back soon before someone discards my bicycle (the communal bicycle parking is full of bikes and people have to jostle for space).

So with all this time being stuck indoors due to a nationwide lockdown, a.k.a. “circuit breaker” in Singapore, I decided to finish my project which I started in January before the madness with the virus started. Initially I wanted to create an app to keep a record of my travels. But after realizing that making an app is unlike making a PowerPoint presentation, I had to scale down my expectations.

tl;dr: My app is called PlanBuyCook and it is available on the Google Play Store. This is the result of 6 months of learning code.

This post is dedicated to those who decide life is not hard enough and want to learn to program from scratch. If you want my code, you can ask me, I won’t put it on Github yet.

So first off, I had to start over before I even got anywhere. My 3-year-old Macbook was too slow and I had to buy a new laptop. I’m quite happy to get one from Japan because it has a Japanese keyboard:)

The real complaint I have is that apps on iOS and Android work differently and I had started learning Swift (iOS) but now had to change to Java (Android). It’s like switching from Spanish to French, there are a lot of similarities, but ultimately they are both read differently. Also, their programs are different (Android Studio for Android, XCode for iOS).

I didn’t want to spend money paying for online courses teaching code, so I took the cheapskate way of watching Youtube videos and reading responses on StackOverflow. Personally, I would recommend this over paying for lessons. You actually learn faster because you are made to do everything yourself. A word of warning for those who are really impatient like me: The computer doesn’t care if you spent the last 5 days troubleshooting and debugging, it will mercilessly reject your code if you make a small mistake.

But don’t fret! Most coding programs now come with intelligent environments that help you with menial things like spelling errors and code highlighting/completion. Just don’t expect it to be smart enough to know what you want especially the UI/UX aspects.

After I finished the coding part of the app, I realized it looked horribly unappealing, the visual equivalent of drinking room temperature tepid water. So I engaged the artistic services of a close acquaintance, known only as Munchy-san (you can find her on Instagram @munchysan_). Every drawing and illustration in the app was created by her.

And so it was done. The only amount I ever paid was USD$25, to start a developer account on Google. I’m pretty happy with it since it actually helps me when I finally get back to Japan and start shopping for groceries and planning meals. I have also recently completed a course in Artificial Intelligence and Machine Learning so my next project will probably have that, but I’m still looking for ideas.