What serverless relational database for follow system?

App Development
1

Hi everyone!

Flo and I have been making a lot of progress with our app, but things are getting somewhat… challenging
And I would LOVE some guidance and wisdom from the community :pray:t2:

TL;DR: What serverless relational database would you choose for an iOS app featuring a following system? And how could I learn it :thinking:

Our app has both a “solo” use and a “social” use. While Firebase is PERFECT for all the solo features of the app, I’ve realized that it simply won’t do for all the social features :disappointed:

Specifically, we want to build a follow system (a user can follow another user and see their “posts” on a timeline feed)

I’ve found this 10-part article of a guy trying to build a follow model with FireStore. Even though he succeeds, he is clearly against using Firestore for a relational database. Apparently it was just painful to make. So even if I did succeed like him, I’m scared that with Firebase the app won’t scale well and we suddenly end up with a 30,000$ bill because each user requires a gazillion reads and writes for a minute of browsing.

So I’ve looked for a different solution and ended up learning about SQL versus NoSQL databases.

From what I understand, NoSQL basically models data as objects (like JSON and FireStore), whereas SQL models data as matrices (like Excel) which allows for more powerful queries.

For example, I’ve read this type of query is hard in NoSQL but easy in SQL:

SELECT * posts WHERE users/followers == $userId ORDER BY createdAt DESC

Down the road, I also want to decide the order of the posts with an algorithm of my making and not simply by chronological order like the code above, so that’s another important level of complexity to anticipate.

Some possible solutions I found on the internet. Would love to get feedback:

  • RedisGraph: the guy who tried to build the follow model on Firebase wrote this article where he apparently managed to combine Firebase and Redis to create the following model. I also found this medium article (slightly outdated, 2013) of a Pinterest engineer who talks about how the built a relational database on Redis.
  • SQL Lite: apparently it’s a little old, but it’s well known ?
  • How bout Amazon Relational Database Service ?
  • Hopefully, you have some suggestions to make :smiley:

As if things weren’t tricky enough, I do have some important constraints:

  • It’s gotta be serverless like Firebase
  • It’s gotta be compliant with GDPR & SOC
  • It’s gotta be secure, yadda yadda yadda

Ideally I would love to hear from someone who’s tried to build a following model, but I’d be grateful to get input from anyone!

I just need to figure out the right technology for our back-end, and how to learn to use it :smiley:

Thanks a ton :heart:

2 Likes
2

Interesting question! My friends and I were just discussing databases this week… see here to get a great quick overview of your options:

I’d recommend FaunaDB. It’s serverless, and behind the scenes takes advantage of the best features from Graph, Relational, and Document databases. Plus, you define everything with GraphQL.

GraphQL is extremely powerful. Let’s say you want a list of the posts published in America. You would normally execute a query that returns all of the posts, so let’s say this returns 40,000 results. However, you only cared about the posts published in America, so you need your app to filter through those 40,000 results (megabytes of data), then show only the 5,000 from America. With GraphQL instead, you can say in your query, just give me back the results for the posts published in America, and it will return only those. Although, I haven’t used GraphQL myself with an iOS app yet. Bottom line is that the data still returns as JSON, but you can create more powerful and accurate queries.

My friend said your use case might be related to a streaming database (also feature of FaunaDB), but we don’t know about it enough to say more. I shot an email with your use case to this same group of friends, so will update as I get more (one was an Oracle database administrator for years. Oracle is the Ferrari of databases, super-pricey and often packed with features you don’t need).

You’d likely want to use a relational/ schema based database over NoSQL. The schemas fit easily to classes/ models in Swift. You define the schema, then you know that that data/ fields have to exist in your model. From there, you can directly map it to a struct in Swift.

In contrast, I’ve heard bad things about NoSQL. At first, it’s great, but later problems pop up. It’s like the Wild West: anyone can create any type of JSON document. That means sometimes the data you want exists there, while sometimes it doesn’t, because there’s no schema to enforce the rule that it must be there. So over time teams’ codebases grow larger and larger as they have to check if the data exists in a particular field. Versus with traditional SQL, you know the data must exist in that column. See CockroachDB for a serverless SQL database. Interesting post about migrating from NoSQL’s MongoDB to SQL’s PostgreSQL here: Why We Moved From NoSQL MongoDB to PostgreSQL - DZone

Also, I wouldn’t consider SQLite as one of your database choices. This database is good for development/ testing purposes, but not for production. At least that’s how we view it in the Python world.

Cheers,
Andrew

1 Like
3

SQLite is quite good on iOS and is even used as the default backend for Apple’s own Core Data object persistence framework. I wouldn’t have any hesitation using it for an iOS app. And it’s built-in to the OS, so that’s good.

You can interface with SQLite using C APIs or one of several available Swift wrappers, such as SQLite.swift, FMDB or GRDB.

Of course, it may not meet your needs for other reasons.

1 Like
4

Thank you so much for your responses @RedFox1 and @roosterboy !

Fireship’s video and the PostgreSQL article were super informative.

Although I am extremely confused and have so many questions… Forgive me, CWC iOS Foundations and Databases were my first serious coding experience.

I haven’t found any FaunaDB tutorial for an iOS app so I guess I’ll need to figure out the implementation on my own. I have a feeling that setting up a serverless database for my app like I did with Firebase is going to take a hella lot more brainpower for a SQL database…

:sweat_smile: :sweat_smile: :sweat_smile: What is a framework? What does it mean that Apple uses SQLite as its framework for CoreData? (I think I remember hearing that CoreData uses SQL “under the hood” but I didn’t really get what it meant) What is GraphQL and why do I need it “on top of” my database? Why SQLite over PostgreSQL (or vice versa)? What is a schema? :sweat_smile: :sweat_smile: :sweat_smile:

I googled all these things and ended up even more confused, so I know those questions are way too complex to be simply answered on a forum.

Bottom line, it seems that I must first wrap my head around all that foundational stuff. So in the mean time I subscribed to Codecademy Pro for the back-end engineer path and started studying. I can’t stand feeling clueless :joy:

How did you first learn all this? I’d be extremely thankful for any advice you can give me :slight_smile:

5

And thank you @RedFox1 for shooting an email to your group! You’ve got pretty cool friends and it’s really generous of you to try asking them about my use case :slight_smile: I would be extremely curious to hear what database they would recommend

6

I’ve never heard of this database. And yes any custom solution is going to take a lot of work

Frameworks are basically a bunch of code someone has written so you don’t need to reinvent the wheel.

CoreData is a framework for interacting with a database. The database CoreData uses is SQLite. That means it uses SQLite “under the hood.”
Also note CoreData is for saving data to the phone only!! Not really to sync to a cloud database.

GraphQL is a way to query a database. It depends on what database you use, some use GraphQL to query it, some use SQL, some use their own API.

It depends on your needs, these are both SQL databases (relational databases)

A schema is basically HOW you layout the data. How it’s organized. However you define columns in a table (sql database). Or how you define the JSON (in NoSQL dbs)

DM me on Instagram to Twitter and I can have a call with you about all this because what you’re trying to build is very in depth and seems beyond your current knowledge at the moment. (Not saying that in a bad way, but you do need to be familiar with backend, before you can make your own)

1 Like
7

Oh my thank you for your help Mikaela!! Indeed this is all wayyy beyond my current knowledge haha
I’d really appreciate discussing this with you, I’m DMing you on insta my name’s Raphael
Thank you so much!!!

1 Like
8

@CalStark you’re welcome, and glad those articles helped!

I reached out to a person at work that sets up databases for multiple teams as part of our cloud team supporting thousands of people throughout the US. He said your use case most closely resembles the “Pub Sub Pattern.” Sure enough, on one of the first articles in Google’s results it shows a similar use case like yours (although, I still recommend looking at other articles/ videos for an understanding of this topic):

“When a user sends a message to a chat room, her chat app instance publishes the message on that chat room’s topic. Subscribers of the topic receive the message.” This could be similar to your own, where those followers receive other users’ posts in their news feed.

One of the most popular and fast databases that fit this pub/ sub pattern is Redis with its messaging-based database. This database is in-memory only, which means it’s super fast. However, it also means that it’s not the best for storing long-term data. So you might have to combine this with another database. Also, I’m not sure how it works for serverless/ if serverless is even possible with Redis, because the database resides in memory (serverless normally starts computations for a short period of time, then closes the serverless function). Also, how does it work as a distributed database (with nodes around the world)? Does it mean each node transmits data to the other nodes as soon as they receive it, or if clients interacting in a certain node only access data in that node, and thus miss the data in other nodes around the world?

If you chose something like FaunaDB, then they would provide you with a GraphQL API to hit that returns your normal, expected JSON. I came across a connection driver for FaunaDB. The only thing that gives me concern is that FaunaDB no longer supports it as a company (unlike they do for languages such as Python, Javascript etc), rather it is now fully community supported. This wouldn’t normally be a concern, but it seems like the community version is not on the latest/ greatest Swift. Still, if it works, then that’s all that matters, and it seems like they simply are not adding more features to it.

Here’s their driver repo. They even include a sample iOS app (although, on a very outdated version of iOS). Still, the code seems simple enough:

import FaunaDB

struct Post {
    let title: String
    let body: String?
}

extension Post: FaunaDB.Encodable {
    func encode() -> Expr {
        return Obj(
            "title" => title,
            "body" => body
        )
    }
}

extension Post: FaunaDB.Decodable {
    init?(value: Value) throws {
        try self.init(
            title: value.get("title") ?? "Untitled",
            body: value.get("body")
        )
    }
}

let client = FaunaDB.Client(secret: "your-key-secret-here")

I’ve heard great things about FaunaDB, but haven’t used it for my own projects yet. Rather, I’ve used PostgreSQL (CockroachDB for the globally-distributed nature) and MongoDB databases in my work. FaunaDB is an up and coming database system. It was created by the founders of Twitter. Here’s an article by a company that used it with their iOS apps, and thankfully also reference that communty driver above:

You’re on a good path, but this stuff is definitely overwhelming. For me, I took a database class in college, and only setup databases as needed for different projects. Mostly I learned this stuff, when my app grew to the point that it needed a database. I learned by doing this work with Python web frameworks like Flask and FastAPI (super simple to setup and get started), and a JavaScript framework called RedwoodJS (full serverless, exciting, JAMStack framework).

From there, you basically setup your database structure, and if things don’t change too much, then you don’t even think about your database too often, because your code can use it fine. As long as your app can connect to the database, create/ update/ delete (CRUD) records, then you don’t even think about the database setup any more. You can stay focused on the true crown: new features and a better experience for your app.

Let us know what you end up going with, and feel free to message me more for more details! Again, I will just leave one warning: think about data consistency as your app grows, and maybe not adopt NoSQL for this reason.

1 Like
9

Hey @RedFox1,

I’ve looked into FaunaDB and I see Fireship talks about it a lot on YouTube and has a course on it - probably a good sign.

I’ve realized I’m super out of my league right now in terms of database knowledge, so for the prototype I’m gonna stick to Firebase and build an imperfect relational model in NoSQL. It definitely won’t scale but at least we can start testing the app with family and friends this month.

In due time I will implement a complementary SQL database under the hood to handle all the relational stuff that NoSQL Firebase can’t handle at scale. For now FaunaDB seems like a really good recommendation! I like that they said “when we founded twitter databases were bad, when we left they still were”. Considering Twitter is super relational it makes me feel hopeful about FaunaDB.

I’ve got a few months ahead of me before public launch so I’m taking the time to complete the back-end engineer path on Codecademy. Apparently it’s meant for people who want to become back-end devs for a living so hopefully it’ll provide enough foundational knowledge to make all this stuff seem less overwhelming. So far the JavaScript modules are pretty good! (but way too easy compared to the CWC challenges haha)

I’ll give updates in a couple months on our Journal with our progress regarding everything

Anyways. For real, heartfelt thanks :slight_smile:

2 Likes